First of all, you've stated you asked 'where do you want to go' and that if he answers 'Wherever', you have asked if there are restrictions/preferences on his side. I don't know how you asked, but sometimes, what I call 'the toddler approach' might work best: Offer them two choices, and have them pick what they want. So:
Where do you want to go today? I'm hungry for McDonald's or Pizza Hut, where would you like to go?
Well, if it doesn't matter to you, do you want a cheap meal or one that leaves you full?
This narrows down their choices a bit. It might be that your co-worker is just struggling when there are too many things to choose from. I've noticed the same thing happening before.
If that doesn't work, here's a bit of a mean solution, but since you're not specifically asking for a nice one... I've been here before when I went to Disneyland with my brothers. They had the annoying habit of not picking a ride to go on, but when I said 'well, then we'll go there', they suddenly had better ideas.
After 2 days of these situations, I got pretty fed up with them. I asked them where they wanted to go (no reply), if they wanted to go on the rollercoaster or visit a show, and they said they were fine with either. So I picked the show, and suddenly they had to go on a completely other ride.
What I did was just say:
No. I've picked the show now. I'm going to visit the show. Feel free to come with me, but I'm done with me picking things and then being told by you to go completely elsewhere.
And I followed through and visited the show. Nice thing was, initially my brothers were very mad at me for behaving like that, but after I had visited the show and they were left to their own devices for a while when we met up again they recognized that they had been a little wrong as well.
After we had all apologized (me for overreacting, they for annoying me and not immediately making clear their preferences) the next 3 days went very smooth: If I offered 2 choices, and they were thinking about a third, they would say so immediately, and we managed to visit all the rides and have a awesome time!