What's the harm... ?
Imagine that you see some people in the distance and somehow you know they are talking about you, but you can't hear what they are saying. I think most people will feel uncomfortable with that. You would want to know what they are discussing. If someone is pointing at you, that suggests that you are the subject of something the pointer is planning to do or that you are the subject of conversation. If you don't know what it is about, this can be annoying. So the fact that the one who is being pointed at does not know why he or she is being pointed at is the main reason it is impolite. At least that is how I have always interpreted it.