No- in fact you can let them go for quite some time without ammonia to no ill effect. Eve if one removes 1/4 of their bacteria from a cycled tank, the remaining ones will reproduce in the presence of excess ammonia pretty fast. Consider under optimal conditions the ammonia bacs can double in about 8 hours and nitrite ones are more like 10 or 12. So that would mean that 25% of the ammonia bacs are returned to the system in a matter of a few hours.
More important here is that you do a large water change (to reduce nitrates etc, and to put back things the tank has consumed) and then insure the tank temp is at the desired level before you add the fish.