You can not fight intolerance with more intolerance
This little pearl of wisdom comes from a friend of mine. Sorry for the story, but I think it will help make some sense of this.
My wife and I were being married, I ask my friend to be best man. He is black, my wife and I are white. I asked him because of all my friends (I don't have many) he is the one who I felt I could ask the most "husband" questions to. While other friends might judge or make small comments, I felt that he could be a great role model and source of advice for my marriage.
One of the things we had to talk about very early on when planning the wedding was my now brother in law. My wife's, half sister's husband (to be exact). He is a racist dirt bag. One of those people that should at least know to keep their mouth shut but never do. The rest of my wife's family that I had met were fine, but this one guy... While my best man wasn't going to be the only non-white person there, all I could think of is the hate filled racist making a scene. Even my wife's family was in fear of this guy showing out. Every one from great-grand parents down warned him and more importantly my sister in law not to make a scene or say anything.
As the day grew closer my now father in law, my best man, and a few other family members where all having a dinner together, just guys, and my father in law says something along the lines of "Look, every family has problems, and our biggest one is going to be this racist little **** that married my daughter. I just want to apologize in advance, and if he gives you a hard time let us know and we will "take care of him""
My friend, without missing a beat says "Thanks, but you can't fight intolerance by being intolerant. One of the great things about this country is freedom. And that means that he has to be free to be a racist, even if every one is thinking it is wrong. That said if he makes me or any of the other guests uncomfortable I will just ask him to leave, and if he doesn't then we can get everyone involved in getting him out of there."
Later that night I asked him, "You know you could just say that the brother in law can't come. No one would fight you on it. In fact that's basically what they wanted." He told me, that would do no good. How is someone ever going to learn, if never given the opportunity. If he never gets to see a black man in a good light, how would he ever learn. My friend went on to explain that there are two views that he could present.
- Black people don't like you so don't come around them. They have just as much hate as you do.
- Black people can't be all that bad. There was one at my sister in laws wedding.
Of those two the later was the best. That people often make the mistake of trying to fight intolerance and hate with intolerance and hate. Instead people should lead by example.
To give closure to the story, the brother in law decided not to come. But my wife's biological father (not a big part of her life, but we invited him anyway) ended up being a bit racist too. He (bio-dad) nearly shit a brick when he found out that we had instructed every one to give money and gifts to the best man, because he had keys to our house and was going to drop them off. Honestly it was one of the funniest moments of the wedding.
It several years later now, and while the brother in law continues to be a minor sore in the side, the bio-dad isn't. He (bio-dad) my still be racist, but he keeps it to him self, and that's all you can really ask for.
At the same time, my wife and I have become foster parents and we do care for children that are not our own race. To protect the children we don't allow racist comments or conversation in the house. That doesn't mean race isn't an allowed topic, but we try to teach that judging based on race, or hating based on race isn't useful.
What it does mean is that we don't let the problematic brother in law around. The kids have enough to deal with, without having to deal with that at home. An arrangement that he hasn't had a problem with yet. So there is a line.
Story over, back to your question.
What does all that mean. Well you have to decide. You are entitled to a work place that you feel comfortable in. But you are not entitled to make this person feel or believe differently. There is a line, where you become so uncomfortable that it effects your work. If he crosses that line, then document it with HR and asked that he be removed to another team.
However, you may have done exactly the right thing. You mead it clear that you were not comfortable with the conversation, and he moved on. It may be upsetting, but your not going to change his opinion.