How to honestly answer questions from a girlfriend like “How did you find this place” without giving the impression I'm always talking about my exes?
You don't need to avoid mentioning them. Just talk about them the same way as you would talk about any other (male) friends that you might have done things with.
Much like your description of your background, I grew up with a lot of female friends, many of whom were among my closest friends. When I finally started dating, I remained friends with the majority of my exes after breaking up and one of them remains perhaps my closest friend some 25ish years later. On top of that, ballroom and tango dancing were my primary social activities for many years. So, yeah, girls/women were present and involved in a lot of the things I've done in my life, but I have never had a bad reaction when talking to a current girlfriend about places I've gone or things I've done with other women.
From the description in the question of how these conversations go, I think I can see why you might have problems: In your attempts to avoid mentioning who you were with, you come off as being conspicuously evasive. When people notice that you're trying to avoid telling them something, they naturally become suspicious and will frequently assume the worst because, if it wasn't bad, why would you be trying to keep it a secret?
This is why I talk about my female friends the same way as I talk about my male friends, and suggest that you should try to do the same. If I talk about Mary the same way (and as easily as) I talk about John, then it's generally assumed that I interact with both of them the same way as well. If, on the other hand, I talked freely about John, but became evasive whenever Mary came up in conversation, then that can be read as implying that I have a different relationship with Mary than with John - perhaps even that I'm secretly in love with her - which is just what you're trying to avoid.