I had a manager once who had either no or very little hearing in one ear. I learned of the problem when we were both in a noisy conference room (pre-meeting) trying to have a conversation. At one point he cupped his hand behind one ear to try to cut out background noise, and soon after he asked if we could step outside. When we did, he told me about the problem and that background noise made it even worse. I paid attention to which side of him I was on after that.
Once I was "in the know" I noticed the other people who also were, who did the same things. If a room was getting too noisy, chances were good that any one of us would try to quiet it down. (I hear fine but hate noise, so I was already onboard with this.) Sometimes he'd say something like "sorry, I didn't catch that -- could you repeat that?", the person would, and things would carry on. By saying "sorry" even if it's not really his fault or something he can control, he deflected any negative feelings that people would have felt otherwise. (I've had similar experiences with "sorry, I didn't see you" when someone was trying to get my attention. People seem to understand.)
What I saw from him aligned pretty well with how I handle my own visual disability in group settings where it matters. Either of us might sometimes ask "could I have that seat? It'll be easier for me to {see, hear} the presentation from there", or ask "can you repeat that / could you zoom that?". Most of the time it's not necessary to actually say "I have a disability", and therefore there's no natural opportunity for people to believe or disbelieve. Focus on the thing you need -- for something to be repeated, to adjust the speaker's mic, to cut down the side conversations, a better seat -- and not exactly why you need it. Save the explanation for cases where it's needed or for people you work with a lot (so sharing it can fend off future problems).
There's not a lot you can do about people you never heard (or in my case saw) who don't get your attention and just sulk away offended instead. You can't address what you can't perceive. For me and my former manager, a combination of effectively dealing with problems when they come up and getting to know people (so more people know about the limitation) has been enough to alleviate any concerns over perceptions of rudeness.