I'm an intermittent faster as well. I'm autistic and hardly ever feel hungry, so I skip breakfasts to make it a bit easier for me to feel the hunger. However, I have other dietary restrictions due to autism that sometimes make it difficult to eat outside with other people who don't have the same restrictions.
My coworkers know about my restrictions so it is easier for them to accept that I won't eat with them but still would love to join. When they first invited me to join on their lunch break, I casually told them beforehand that I wouldn't be able to actually share a meal with them but that I'd be happy to have a drink or a side dish I could eat alongside. Maybe you could opt for a drink too? I'm sure you're aware that the rule with intermittent fasting, in order not to break the fast, is to have drinks that don't have calories in it or sweeteners that may be understand by the brain as sugar. So maybe you could have black coffee, tea, or other drinks with sweeteners that don't break the fast? You just need to make sure you'll have something to drink or share with them when joining.
Although it is much easier because I told them why I couldn't share a meal with them, if you haven't told you coworkers and are not willing to disclose it to them, the key is not to make a big deal about it. You could say something along the lines of
Thanks for the invitation, I'd love to join you for lunch. I won't be able to share a meal but I'd be happy to share some time off with you and banter.
Now they will likely have questions about why is it you can't have lunch. The simplest way to handle this would be to be honest about your journey and why you're doing this. If you don't want to tell them, you could say that you're restricting your food intake to a big meal per day. Surely they will be intrigued, but fasting is much more common these days and telling them would dismiss most of the potential questions they may have. The key is to show them you enjoy these times together, no matter what's in your plate - or whether you actually have one in front of you.
Another option would be to propose other ways of spending time together. My spouse recently started organizing board games during lunch break. Their coworkers loved the idea and they're now having a great time together almost every day, after they've eaten (I don't think any of them do fast, but some of them have restrictions like veganism or IBS that make it difficult for them to join if the others go to a "regular" restaurant). Maybe this could be a way for you to spend time to get to know your coworkers while not being uncomfortable by the fact you can't eat with them.