I could never bring myself to kill a creature on live rock, be it whelk, mantis, or medusa worm. If you are truley afraid of this animal, use Tiggs's suggestion to remove it, and then give it to someone who might want it.
Mithrax crabs are very effective cleanup critters, they may eat LPS however. Although this is a very rare occurence, so watch him for a while and see.