I used to be a recruiter, and while the goal was different, the methods to approach people I think are applicable to everyone in any situation. In your case, I would take a second to get an idea of the reader's non-verbal communication first. There are some obvious cues, like maybe he or she has headphones in while reading. This ought to be a clear sign that maybe he or she is trying to eliminate distractions, or maybe even avoid human interaction altogether.
I would also look to see how they respond to other stimulus, for example, do they look up to see who is getting on the subway? Do they acknowledge and smile when people walk by? Or do they just tuck themselves in closer and not make eye contact with anyone? Unless they are at least already willing to make eye contact (even if just for a second when they pass by) then they may either be intentionally trying to avoid contact, or they may be so absorbed that even if they aren't trying to avoid contact they may be startled if you interrupt them.
In any case, if they show some kind of engagement outside of their book, (such as smiling at passers by or occasionally looking out the window, etc) then I would consider that as a go-ahead opportunity to engage. To do that, I would follow some of the guidance already mentioned, and especially the Tom Au's guidance about making it worth their while, and also the general comments about using an introductory and polite phrase. And be genuine! Say what really drives you to ask! I might say:
"Excuse me, hi! Would you mind sharing what book you are reading? I really like the cover artwork!"
And smile/show enthusiasm while you say it! Or maybe:
"Sorry to bother you, but would you mind telling me the author of that book? I am fascinated by Chernobyl!"
Whatever it is, it doesn't matter, the general format is "(say something to get attention and portray that you acknowledge you are interrupting them), (ask your question in a way that really asks for permission for them to answer you)? (give them the psycho-social reciprocity, tell them why you want to know what only they can tell you and how much happier it will make you)!"
Following this methodology will:
- help you rule out readers that may not welcome such a question
- improve your odds of a positive and friendly response and make sure your question is fairly well received at least
- has a good chance of leading to an interesting conversation!
Note: I advise against making something up as to why you want to know. If you wind up in situation 3 but you weren't really that excited about Chernobyl, then you may find yourself in an awkward conversation... Also, in regards to challenging books, don't worry about it! Maybe if you know it is more advanced, just dismiss or downplay it.
Say something like:
"Oh, I'm just trying to learn more about Astrophysics, its really interesting!"
Then, if you see them kind of recoil a bit, maybe distance yourself from the book, so something like:
"I think I might have bit off more than I can chew!"
Or
"I think I dove head first into the deep end!"
At any rate, after these first few exchanges, the possibilities are
endless, just go out and try different things and let us know how it
works out!