It depends you how heavily your tank is and how good your filter is, also whether it is planted.
The reason for vacuuming besides what's mentioned in the above posts is also to let some oxygen go down in the gravel. If you let a lot of gunk build up in it, it blocks oxygen from going in it which causes the wrong type of the bacteria to grow and in turn suffocates the good one and then ammonia/nitrite spikes occur. People normally notice cloudy water when this happens.
If your filtration is good enough to flow the water all around the tank, not too much will settle in the gravel but may go into the filter sponges instead. So washing the sponges in tank water to remove the big pieces will do the trick.
And of course, high ammounts of waiste leads to high ammounts of nitrAtes so if you don't do enough water changes they'll go beyond control and although they are not poisonous to fish, they still need to be kept below a certain level(20-30ppm)
Also, high ammounts of ammonia processed, even if your filter is managing to process them into nitrAtes on time without any spikes to hurt the fish, can lead to a sudden Ph crash because this is an acidifying process.
And additionally, agae blooms can happen with the excessive ammounts of nitrAtes and other nutritiens.
The exclusion is in a planted aquarium that also has plants with good root system. Then in this case it is better not to vacuum and disturb the gravel around plants. The plant roots naturally oxygenate the gravel and the gunk is beneficial to the plants as it contains a lot of nutritients.