A good question. There is a difference between dealing with charity collectors and other street vendors for two reasons I can see. Firstly, charity collectors are governed by laws* as to how they invite donations, so they ought to be less aggressive in their approach than the typical sales pitch of vendors. Secondly, it somehow feels worse saying no to a charity, especially if it seems a 'good cause'.
In this financial climate you shouldn't feel obligated to donate to charity, and if you can afford it, most people have a 'preferred charity' that they donate to rather than spread their cash around thinly.
Some brief things you could say, if you feel obliged to say anything at all, are:
If saying no to a charity is really bothering you on a conscience level, do some reading about charity transparency. For the UK try Charity Clarity or for anywhere else check out GiveWell. I personally don't feel bad turning down charities that have a dozen executives all paid more than the British Prime Minister, and there are several of those on the high street! Also worth educating yourself on this in the event that you get drawn into a debate.
But despite all of the above info, if you don't want to give, or cannot give, just walk on by, don't feel guilty.
*I am in the UK. For information on the law governing charity fundraising in the UK you can go here.