Option 1: Try to back out of it
One option which is a possibility but I would not recommend is to make up something in your schedule that you "forgot to consider" when you agreed. That being said, if your coworker finds out about this, it may ruin your relationship with him, as well as possibly with other coworkers in your workplace.
If you absolutely refuse to compromise and go with your agreement, the simplest method is to just confront him and say that you really dislike camping (try to give a legitimate reason why you dislike camping) and invite him to another activity. Firstly, by directly confronting him rather than beating around the bush, etc, you show that you aren't trying to avoid him, you simply have a desire that differs from your original agreement. Secondly, by giving him a legitimate reason as to why you dislike camping, you show that it's not him you dislike, it's the activity of camping. Finally, by inviting him to another activity, you further reinforce the fact that you still like and want to interact with him, you just don't want to go camping. Hopefully your coworker will understand.
That being said, backing out of your agreements often, even with reasons, will still deteriorate your trustworthiness, which in a workplace is something you should avoid.
Option 2: Make a compromise
The last time I went camping, we spent some of the time camping (it was in a cabin), some time hiking, but the majority of the time at a nearby museum (this was with my family). Camping doesn't have to be restricted to being in a tent/cabin on a campsite, if that's the thing you dislike. Try to make a compromise or find some other activities less related to camping that you can both enjoy, so that you can go camping as he likes and do another activity as you like. Firstly, you aren't contradicting your previous agreement. This option is less likely to deteriorate your credit/trustworthiness. Secondly, by taking some initiative in planning what you are going to do there, it demonstrates your initiative and engagement in the activity your coworker invited you to. Depending on the type of person he is, this could even make your interactions/relationship with him better. Finally, this, while not avoiding going camping, makes the activity more enjoyable for you (and maybe for him too!) which makes it both a compromise and a win-win situation.