I wouldn't worry too much unless there's a colony of worms, some worms are a natural part of an ecosystem and arent harmful.
thanks for not using bleach or any "cleaner" a lot of folks kill their whole aquarium that way. and did they look like detritus worms? they eat fish waste and plant matter...