There are two very different factors here, the substrate and the filter are very different "operators" in an aquarium.
The filter serves one or two purposes. The first, common to all filters, is the mechanical removal of particulate matter from the water to keep it clear. A second purpose is biological filtration, involving nitrifying bacteria that assimilate the ammonia, producing nitrite which other nitrifying bacteria use producing nitrate. In planted tanks this biological filtration is totally unnecessary, and should never be "encouraged," because it is competing with the plants for the ammonia. Plants actually take up ammonia much faster than the nitrifying bacteria, so the plants "win" the race, but this only illustrates why biological filtration is less than necessary. It will still occur regardless, but at a minimal level. Always assuming the plants are healthy and capable of dealing with things.
Obviously the filter media must be kept clean of all that "gunk," or it will not be able to do a good job of the mechanical filtration. And all those organics sitting in the filter only risk nitrates, though again in balanced planted tanks this is or should be less of an issue because the plants are hopefully out-competing the nitrifying bacteria. Still, it cannot hurt to rinse the filter media at every weekly water change. I always have in 20 or more years at least for internal filters like sponge; canisters are a bit different.
Now, the substrate is a very different issue. This is the foundation of an aquarium. The many species of bacteria that live in a healthy substrate far surpass those in the filter with respect to filtration other than nitrifying. The substrate should not clog up with detritus...that is a sign of too many fish, or too large fish, or too much food being fed. My Corydoras tank ran for years and I never touched the sand, and when I disturbed it to plant or something, there was remarkably little mess stirred up. The cories poking into it continually, plus rooted plants, plus Malaysian Livebearing Snails, all worked to keep the substrate "sweet and fresh" as one book put it [can't remember the book, it was some years back now]. What goes on in the substrate, again assuming the tank is balanced biologically, is crucial to having a healthy aquarium. You can shut off the filter, but you cannot do without a suitable substrate.
Now, I did have some tanks in which I had to clean into the substrate at each water change, the open areas anyway (never move wood or rock to clean under it, this is a serious risk). Each aquarium can be quite different; the same natural laws of biology and chemistry will apply, but they may play out differently.