Welcome to the forum Dusty.
The bacteria that you are building up live as a coating on the filter, gravel, tank glass, decor and lots of other things. The film that they end up forming sticks to things fairly well. That means that unless you get very aggressive in cleaning the filter media, you will still have most of the bacteria after you rinse out the media with tank water. People that use brushes or use chlorinated tap water can indeed disturb the bacterial colony and take a step backward in establishing an effective biological filter. It is a good question to ask and you are thinking about it correctly. The only thing that makes the cleaning OK is that we do not scrub media, we just rinse it in used tank water. You really only clean your filter media when it starts to become plugged up, not based on some time schedule. This is true during any cycle and for me it is also true on an established tank.