The first reason that springs to my mind for not paying you back is that your friend CANNOT pay you back, though $20 seems like a smallish sum in the USA, but that’s still a possibility.
Another reason that happened to me personnally recently is that he never agreed that he had to pay you back, and hence resists what feels (maybe unconsciously) to him like an exortion.
However, in both case, your message suggestion doesn’t seem rude to me : you’re stating a fact without accusing, you seem genuinely concerned but not angry, and straightforward enough so that he can’t avoid replying without a good reason, so I’d say go for it (then again, I’m just a stranger on the Internet...). :-)
Also, have you tried getting in touch with the other people you saw him with at the event ? What did they say ?