If you do not disturb it, you will not have problems with poisoning fish. If on the contrary you regularly disturb it significantly, you also will not (should not) have problems. Whether the fish do the disturbing or the aquarist doesn't really matter. The following excerpt from an article I wrote on bacteria some years ago may be instructive.
Substrate Bacteria
The greatest population of bacteria in a healthy balanced aquarium occurs in the substrate, not the filter. The floc or humic compost that collects in the substrate is the host for the biofilms; this is why the substrate in planted tanks should never be disturbed significantly, and many aquarists apply this to non-planted tanks as well.
In very general terms, aerobic nitrification takes place in the top 1-2 inches of the substrate; anaerobic de-nitrification takes place approximately 2-4 inches down, and anaerobic bacteria producing hydrogen sulfide occurs in substrates deeper than 3-4 inches. In all three cases, it will be deeper in coarse substrates (like pea gravel) and more shallow in finer substrates such as sand. These generalities will also vary with the presence of live plant roots and substrate “diggers” such as snails and worms, since these factors result in more oxygen being made available in the substrate, reducing anaerobic bacteria activity. An oxygen level in the substrate of as little as 1 ppm promotes nitrogen reduction rather than sulfur reduction (hydrogen sulfide). [6]
Maintaining a substrate of fine gravel or sand no deeper than 4 inches, having live plants rooted in the substrate, and keeping Malaysian Livebearing snails are the best and safest methods of providing a healthy biological system for aerobic and denitrifying anaerobic bacteria.