Easier said than done because I feel that if I do this I will make it a bigger deal than it was
It's your body, which makes it as big a deal as it is to you. If it concerns you enough to raise the issue, raise it.
and that it will make things awkward for all of us
It's already awkward for you from the moment he touched you.
I think him being so comfortable with me was a bit too much. Was what he did normal? Was it sexual? I know it all depends on how it made me feel but even if hypothetically hadn't bothered me, was it appropriate?
There is no normal. Your body, your rules. Same thing regarding what he meant, it's your body. Apparently it bothered you, which means that it's inappropriate.
Moreover, it happened not just once but twice.
You don't want to make the things awkward based on you not knowing what he meant by it and that he may even not make it that big a deal.
He did it twice and did it on purpose, which means that he knew what he was doing and probably watched your reaction. If you don't react, it will certainly continue or escalate.
Your husband's friend assumed it was appropriate for him to touch your thighs; just tell him that it isn't - don't even wait until it happens again. As your husband suggested, you should bring up the next time that man comes to your place:
Bob, since you are here, I'd like you to know that I didn't like the way you touched me the last two evenings and don't want you to do it again.
Be assertive, straight to the point. If he continue his behaviour, he is no good to be around. We haven't much information, but maybe neither do you. Maybe his divorce was related to abuse, maybe he suddenly feels lonely and can't resist your sight. Maybe he is just flirting or playing around.
Either way, you don't care; that's none of your concerns.
The fact is that he did something to your body that you didn't like and that therefore he has to stop doing it. Not bringing it up will only help him. It will not help your husband, and it will definitely not help you.
In the light of your comment to Bradley Wilson's answer, I would suggest you and your partner have a serious chat with Bob, because either he doesn't speak your language or he doesn't care at all. Let him know that your husband and his friend knows about it and is on your side. That is, assuming that the both of you still want him around after him showing this kind of disgusting behaviour.
Something that I forgot to point out because I thought it would be obvious, something I'm not so sure after giving it more thoughts :
Him touching you discreetly means that he intends as few people to be aware of it as possible: only you and him. That is because he doesn't want your husband or anybody else to witness it. Your knowledge of it is unavoidable, but he is certainly counting on you not daring to confront him - which is by the way the reason you asked the question.
I feel quite stupid for not addressing that point, since that is a very common pattern regarding sexual harassment/abuse. In this situation, Bob is only seeing you as an object but is aware that there is a risk for him to be in trouble. That's why he began with something that can be overlooked - he is testing the fence.
Not confronting him will only confirm him that he can do it without you addressing the issue, whether it is because you like it or are too afraid, shy or ashamed to raise your concern. In addition to that, he can still pretend you are making things up if you confront him. If he gets away with it, it may give you the impression that you are powerless and that confronting him is useless. That's something this kind of abusers not only use to their benefit, but count on since the beginning.
I don't know what to think about your spouse's reaction. He seems to believe you but it doesn't look like he understood the problem. Maybe it's because you were yourself unsure of what was happening but still. I hope you will edit your question if you confront Bob regarding how it went.
Given the uncertainty of your situation, this is a cause of worry and I hope that it will be resolved soon and in a safe way.
In addition, reading about both your state of mind when you reported the situation and what happened when you confronted Bob, women in the same case landing on this page could have first hand testimony that reporting these arseholes does work. There are way too many women in your situation who don't speak about it because they think that nobody will believe them or even care at all, or fear that they would be shamed for it instead of the abuser.