You can't without angering people.
These people know they aren't following the rules. They also know they can get away with it. They know people will let them in because they don't want to be rude, and they are taking advantage of that. People who do this often get aggressive when they're refused, because they know this will often get the person refusing them to back down.
I know this because I used to work at a returns desk at a large retail store. I saw the exact same thing every day: people trying to return stuff against policy, often things they broke or stole. They knew most people would do it if they made enough of a fuss, so that's what they did. Why would they follow the rules if they can get away with breaking them and get what they want?
To deal with the immediate problem, really all you can do is politely but firmly state you cannot let them in, and they need to use an access card to enter instead. They will argue with you, say other people let them in, that it causes no harm, why do you even care when it doesn't affect you, etc. etc. Don't give them any justifications, as that will just give them something to argue against; you are not going to convince them. Simply say you cannot let them in, then shut the door. Many people will consider that rude, but it's more rude of them to attempt to take advantage of your politeness, so try not to feel bad about it.
In the long term, you need to get management and/or security on your side. Tell them the issue you are having, and explain how you cannot possibly enforce this rule when everyone else is setting the expectation that it's okay to break it. Ask them what to do if somebody tries to get in without a card or if you witness someone else allowing an unauthorized person in. Should you say something but not push it if they argue? Should you call security? If possible, email them a summary of your conversation to help cover yourself if something happens later.
Ultimately management may not care enough to take any action until it becomes a problem. That puts you in a tough spot, and you'll have to decide whether you want to risk getting in trouble when a problem inevitably occurs or if you'd rather deal with being the only person who tries to enforce the rules.