1. Befriend the wife.

I'm a single man who is friend with married women.  In my case I don't want either of us getting feelings for each other.  I know it can happen if you're not careful.  So I intentionally befriend the husbands.  It doesn't take long for it to be geniuine friendship.  (It moves from being intentional to being natural.)  I usually end up closer to the husband than the wife.  As a byproduct, I've never had an issue with husbands trusting me.

I suspect if you befriended the wives, things would work out better for you. 

2. Set boundaries for yourself

Here are the ones I use:

I don't call/hang out with/etc. a married woman more than once week.

When I message or text a married woman I do group texts between her and her husband.

Pick ones that you think strengthen their marriage.  That should be your desire.

3. Be willing to let go.

If you love someone else, you will support them.  Even if they don't want to be with you anymore.  If you were truly friends with these husbands, you'd value their marriages to the point that you'd sacrifice your friendship for the sake of their marriage (and therefore them).

The closer you are to this, the less likely they'll want to leave.  However it's not something you can fake.  It has to be genuine.

I'm not sure where you are with this.  But I thought it was worth mentioning.