Pin Cushion and Fan worms

Hi all, I have a nice yellow pin cushion which has been thriving in my tank for the last 3 months. However in the last 6 weeks I am having more and more fan worms coming up everywhere. They are growing on 1 or 2 rocks specifically and have now covered the base rock of my pin cushion. I have left it; but on inspection last night I noticed that the pin cushion's flesh was a little eroded where the fan worms has grown. I scrubbed them off last night (pin cushion obviously not happy about that) but I am sure they are just going to grow back.
Is this serious? Can they cause damage to the pin cushion should I keep scrubbing them off or just leave them.

Any help will be greatly appreciated.
THanks,

 

Nope, won't harm the pin cushion. Have had similar situation.

Fan worms are good at growing anywhere.

I have them growing in my skimmer, carbon reactor, PO4 reactor and on my macro algae.