I'd say no to the ants, but yes to the earthworms, provided you chop them into tiny pieces for your fish to swallow. Then remove any uneaten pieces or they'll foul the water.
If your garden uses any pesticides, herbicides, or artificial fertilizers, I'd avoid putting things from said garden into the fish tank. It can't be worth the risk.
If you had your own worm farm and could ensure that no chemicals where in farm then worms would be relished by most fish (my cory's love earthworms), the bonus of having your own worm farm is that your not adding to landfill with household scraps and you can choose suitable sized worms for feeding to your fish. I have given my fish ant eggs and pupae which they love (its an older feeding method), the main trick is to once again try to make sure that no pesticides etc have been used in the area and don't use huge eggs/ pupae unless you have fish with bigger mouths. I used to collect the eggs and pupae and sift off the dirt and try to remove as many angry ants as possible (fish don't like getting bitten on the lip any more than you would by an enraged ant ), one of my fighters loved the ant egg feast so much he used to swim into the sifter that I had partly submerged to allow the eggs to float out without dumping left over dirt into the tank and start scoffing before the other fish got a look in.
In summer we used to catch Christmas Beetles and feed them to our oscars and they also readily ate grasshoppers.
The only other tip I would add is make sure that any live foods such as insects that you feed your fish are a suitable size for the fish to eat and not choke on, and that the insects that your hoping to feed to fish are not toxic.