With the debris in the gravel, it is already in the water column. As it decays it pollutes whether its stirred or not.
Vacuum the gravel, stir the gravel, whatever it takes to get it out. Vacuum is less stressful if the fish are still in, if they are out then go for stirring, just get the debris...