I think the reason why a taxi driver wants to chat with someone is when the passenger looks like they have nothing to do (read: bored) while in the middle of the trip.
Perhaps it's not everybody's hobby, but I usually take and play with my smartphone after telling the address. While that's what I usually do to kill time in public (in a sense that I don't explicitly tell people not to talk with me, because I am actually okay with it), drivers also tend to avoid initiating a conversation when they see it, unless it's important like, choosing the route/avoiding tollway, etc.
However, when they still try to initiate a conversation, I have to reply that I'm busy, like,
Sorry, I'm a bit busy now.
Lying or not, it's true that I'm currently busy with the thing on the smartphone.
Reasonable people tend to avoid initiating conversation when the opponent looks busy, especially if they're stranger. However, this is my anecdote in Asian culture where individualism is quiet strong (in fact, I have never had a chat with any taxi drivers...).