The first reason that springs to my mind for not paying you back is that your friend CANNOT pay you back, though $20 seems like a smallish sum in the USA, but that’s still a possibility.
Another reason that happened to me personally recently is that he never agreed that he had to pay you back, and hence resists what feels (maybe unconsciously) to him like an exortion. In my case, I actually was the not-willing-to-pay person, and what decided me to do it nonetheless is that though we didn’t agree beforehand, the deed had been done and I had objectively benefited from it, so it was fairer to the both of us that I paid for something that we didn’t agree on but had been done, rather than the contrary.
However, in both case, your message suggestion doesn’t seem rude to me: you’re stating a fact without accusing, you seem genuinely concerned but not angry, and straightforward enough so that he can’t avoid replying without a good reason, so I’d say go for it (then again, I’m just a stranger on the Internet...). :-)
Also, have you tried getting in touch with the other people you saw him with at the event? What did they say?