Completely agree with Skins. The reasons are twofold: Not only is the main reason what he stated (the vast majority of the beneficial bacteria we want live inside the filter (because that's where they get the high flow of ammonia and fresh oxygenated water)) but the other reason is that bacteria live inside bacterial biofilms (of their own making) which allow them to adhere quite tightly to their surface of choice (feel the portion of your teeth near the gums before and after brushing your teeth to feel what biofilm texture can be like) and in our freshwater case, the films are like brown stains on the substrate gravel or on the sponges or other biomedia inside the filter.
While it's true that anytime you remove debris (either via gravel cleaning or filter maintenance) you are going to lose some bacteria that is riding on that debris, it is just not a significant factor except in cases where you are still cycling the tank or other unusual circumstances. Most of the time you would not be able to detect an effect.
~~waterdrop~~