As long as you have these guys:
http://www.reefs.org/hhfaq/worms/faq_bristle.jpg/variant/medium
which are often in the Eurythoe genus and related genenera, then everything should be fine. There is one similar-looking exception, Hermodice carunculata, but those worms are rather rare as hitchhikers.
Eurythoe worms are usually harmless as long as you don't touch them. I can't say always harmless, but when not allowed to become dominant in the system, they are beneficial scavengers. The situations under which they create problems are when the tank is massively overfed with nothing to keep the worms in check (predators include some crabs, coral banded shrimp, and some wrasses), so you can get a huge population boom of bristleworms and also very large individual worms. When that happens, broken bristles can start to accumulate in the sand and attempts to starve out the worms can cause them to mob other things in the tank out of desperation. That said, they are not something to be afraid of. It would be
very hard to find a marine tank without a few hiding in the rocks somewhere - they really are that common. As a precaution, some people use tongs to remove any large worms that become bold enough to come out during the day (usually a sign that they are hungrier than normal).
Do make sure that's the type of worm you are seeing though. Two other types of hichhiker worms that are often called "bristleworms" are Eunicid species (5 tentacles on the head, some are ok and others grow scarey big) and Oenone fulgida (bad, a snail predator). However, these two are far less common than the typical Eurythoe sort of worm.