Recently, I've been talking to someone that is cross-eyed, and his face has a pretty visible scar as well. These talks are mainly at lunchtime when there's more than 1 person present. Whenever he's talking, I try to maintain eye contact or look at his face, as I normally would when having a conversation including multiple people.
I'm generally not awkward at making eye contact, I don't stare or make people uncomfortable. But this person seems to feel uncomfortable with eye contact nonetheless. Sometimes they seem to suddenly be aware that someone is looking at their face, resulting in them shortly making eye contact before looking down and stuttering a bit. After a while, the conversation continues as usual, but usually with them keeping their head down and not looking at whoever they are conversating with.
I'm not sure how to deal with this. I don't want to make them uncomfortable. Looking away when he looks at me seems like a bad option that would really send a message of 'oops, I got caught staring at your remarkable face' which I'm really not, I'm just making the regular eye contact I make with everyone. Not looking at their face at all seems to me like it would seem that I'm treating them differently because of their look, which also seems wrong to me.
How do I properly handle eye contact with someone with distinctive facial features, that seems uncomfortable from time to time with this eye contact?