It's never unsafe to pull over (where there's space), and any speed demon would sigh in relief at the sign of a right turn signal (or left if you're in one of those wrong-side countries ; ) with slowing down to pull off. If someone else wants to go much faster than you, please pull off for them. Its much safer for them and for you, and in lots of places its actually the law.
As far as social harmony is concerned, intentionally going slower is the worst option. It makes you the Ahole. Refusing to pull over also doesn't score you any social points. So if you take longer than something like 4 or 5 minutes to recognize you're blocking traffic and pull over, you're either being unaware of yourself (which isn't good in any situation, social or not) or you're simply being rude. That said, you shouldn't feel forced to go faster than you're comfortable with. But its not acceptable to have a drivers license and yet still feel unconfortable pulling over for someone. They won't hit you if they want to get where they're going.