Some people are just looking for a debate - while that's fine, it sounds like all you really want to do is avoid starting a debate about particular topics. And if that's the case, then you should just be upfront about it.
You would say something like:
Well, you might have a point, but I'd rather not get into a debate about X right now.
Mind that a wary listener might notice "you might have a point" as very weak agreement and take offense - so you may want to leave it off and cut straight to "I'd rather not talk about/get into a debate about X right now".
Which acknowledges the validity of their point (not that you agree with it, but that they have one - which is usually what the speaker is looking for anyway) and indicates that you'd rather not continue the discussion. This works even if you do agree with the person, but would rather not talk about the subject at the moment.
If you'd rather not acknowledge that the point they're making is valid, you can side-step it by simply cutting to "I'd rather not get into a debate about X" - and if the speaker insists on making their point, reiterate that you don't like to talk about X - this makes it so that you aren't directly shutting their point down, but just trying to avoid talking about something that makes you uncomfortable.
Now this doesn't completely absolve you of any rudeness - not wanting to talk about something can upset certain people if they're very passionate about it - but it does allow you an avenue to avoid certain topics you'd rather not discuss at the moment.