Since I travel by public transport a lot, yes, this is annoying.

I usually try using non-verbal communication first, since people using public transport sometimes explode really easily. And, on a full bus/train, sometimes people don't really have a choice, they can look at my phone, their feet or another person's ass. 

Generally, my first move upon noticing is:

 - turn my body a little away from them, if possible.
 - turn my screen a little
 - start shielding the phone screen with my hand or if you have one of those book-like covers, the flap of that. 

Don't do these things too subtle. Make it obvious you are shielding your screen, make it obvious that you noted their interest in your screen and are reacting to it. Usually, this works fine. Most people know it's impolite to look at someone else's screen and once they know you noticed they'll look away hurriedly. 

Of course, there's always the incidental nosey person that will only become more curious about your activities when you try to keep them secret. For these situations, I always go very carefully. 

Try to get a good feeling for your surroundings. Are there any people there that can help you if this person reacts badly to you asking to respect your privacy? Does the curious person gives you a creepy or dangerous vibe, or look like they are in a bad mood? Then you might want to just turn off your screen like you are doing now. 

You might want to explain that you'd like to keep your privacy, but that is most likely to be rebuffed by 'then don't do privacy sensitive things in such a public place', especially if it's really busy. So, I generally avoid the word privacy when it comes to asking people to look away. 

If you really want to verbally say something, I'd go with something that explains that you *think* they are looking at your screen, and that *you* feel uncomfortable about it. Make it about you, and make sure to the mention the 'think' part to avoid sounding too accusatory:

> Excuse me, Sir/Madam, I get the impression that you're very interested in what's on my screen, but it's making me rather uncomfortable. Would you mind looking the other way? 

If they refuse, promptly turn off your device and start staring out the window. Don't try to have a last word about them being rude, just use your body language. That will prevent things from escalating further.