Get a t-shirt that says "This is my Human Costume"
Neither boring nor are you forced to wear a costume. They already know your objections, and this way, you've participated ironically without having to bow to a dress code. You'll still have to plan your outfit, but it won't be about a costume. Anyone asks, say "I'm not big into fancy dress, and I'd said yes before I knew it was a fancy dress party, so this is me." or "I don't like dressing up, but I do like parties, so this is me." Or "I'm not into fancy dress, but I do like NAMES OF FRIENDS HERE and wanted to come out tonight."
This one says "I'm really a worm," but there are a bunch of others that say "I'm really a giraffe" and so on...the most that might happen is that you might have to answer questions about really being a worm or a giraffe. (Answer "I'm not a giraffe tonight for sure--I'm super human, look at me doing human things! I'm eating chips! It's a really good costume. Giraffes don't eat chips.")
There are other options out there "Sorry I left my costume in the TARDIS" for Doctor Who fans, t-shirts that say "This is my Lazy Millennial Costume" and so on...
Basically, this is the lowest of low effort while still fitting in and being fun.
Just keep in mind that really, no one will probably care. Other people will be far more worried about their own stuff to give it much of a thought unless YOU make a big deal of it. Because it doesn't need to be, and your t-shirt can basically say that. I've been to parties in the U.S. where folks have worn these, and it's never been a big deal.
Congrats, you have now made it politely clear without saying a word, even though they already know! Once you've bought one of these, you'll now have a standard go-to outfit for any fancy dress party! Do wear pants.
As for not going to the party at all--you've really already done your due diligence on that basically saying "I'm not doing a costume and I won't be the only one there not in costume." They are being the rude ones for not empathizing with you..You've pretty much said all that you can say, and they are being pushy and making you anxious. Just stay firm but kind if you opt not to go.
With my solution, you neither have to wear a costume, nor are you excluded from the fun. And, I've seen this on folks who don't like costumes and people who just didn't feel like dressing up.