In addition to writing the excellent earlier answer, @Lio Elbammalf made this very pertinent comment to OP:
Try not to consider the two as linked, there are all sorts of reasons they might not come to yours that could have nothing to do with the fact it is your party.
That is exactly correct. I am not very sociable myself and in my hometown in India our family is usually invited not for routine birthday parties but only to relatively formal events like weddings; moving into a new house; 1st birthday of a relative's child; or ritual 'celebration' of an old person's life either at their 60th, 70th or 80th birthday or a specified number of days after their death.
I may or may not attend and usually keep away because I am very introverted and get quickly exhausted by social interactions -- but what is never a reason for going or not going is the consideration whether they accepted our previous invitation.
Feel under no personal or social obligation either to attend the party or to keep away. Go if you want to and you think you will enjoy the event. you need not go if you don't want to!
Once you are invited you can legitimately attend any function even if they didn't attend yours. Equally you must be relaxed and not care if somebody misunderstands your not going.
You can learn not to feel social guilt for failing to attend someone's party, nor take social revenge by deliberately keeping away in such situations. It is also easy to follow typical etiquette for not attending such an event. In our region it is customary to contact the person over the phone beforehand, explain why we won't be coming (even if it's clearly a made-up reason), and wish them well.
However, your attending their birthday party may help strengthen family bonds with those relatives and remind them how they have neglected your parties, and hopefully they will attend next time!