Considering all of the points in your question, I'd say that you have little reason to sign the card and should feel perfectly fine with simply skipping it.
It'd be one thing if you'd worked with them or had to interact with them at some point but, considering that you'd never worked with them or really met them, you have little reason to want to sign the card. Additionally, if this person doesn't know you, they would be unlikely to appreciate your note and may be confused by a name in the card that they don't recognize.
For Admin's Day our office sent around cards for the four administrative assistants who work with us and I only signed the cards for the two I actually interact with, which left space for people who wanted to write a thanks to the other two.
If someone asks you to sign the card and you want to know how to respond, you have a couple options:
- You can say that you're busy right then and hope that they forget to come back.
- If they offer to leave it with you and ask you to pass it on to someone else, this is simple, just pass it on without signing it.
- If they press you on it, be honest. Say that you don't really know them and you didn't work with them and you'd rather leave space for others.
Alternately, you can do what I did when someone I barely knew was leaving - the others around the office bought a poster for them with a meme on it (I don't remember which one) and I wrote something funny that related to the poster and had to do with him leaving.
Regardless, I would hope that none of your coworkers would be so pushy as to require you to sign a card.