Would it be rude if I just handed him the doctor's note or emailed him?
As an employee, I've done similar things only to bosses (once, small company, 2 partners) who I couldn't deal with anymore (because of their really bad attitudes), and did it just before I quit (as all but one in a team of seven did too). We all did it on purpose, with no explanation, but we were fed up, and yes, we wanted to be rude. We handed our notice without a word, and didn't answer any questions. We were out, period.
So I'd definitely recommend not to do that. It would indeed be rude.
Now, as a business owner, I'd appreciate if you would tell me as soon as possible. Just be nice. :)
You: Hi boss, I have something to tell you.
Me: Hi Alice. How can I help?
You: I'm 22 weeks pregnant.
From there, let's talk about you for a few minutes (be nice and "pleasantries" on both sides), then switch back to business, so that I can deal with the upcoming events and organize my schedules, team, check with you what you'll need, when and how you'll leave, and so on.
You don't have to be scared, unless your boss is really mean and a [ you name it ]. Be nice and professional, let him know, that's it. But it should be a 1-on-1 meeting. Not by mail or just handing them a doctor's note.