Personally, my policy is two-fold:
If the reason for rejection is likely to upset the person, just generically and politely indicate that you're not interested. "Sorry, I'm not interested" works in 99% of the cases, and only rarely would the person pursue "why?".
In your specific example (you rejected them based on looks), this may be a good avenue. If they aren't conventionally attractive, they already know that from experience, so hearing that from Yet Another Person will just be more painful. If they are conventionally attractive, they may just view your "different" opinion as insult.
If the reason for rejection is less likely to upset the person, you may want to honestly state the reason. Frankly, that's rarely useful, as most people wouldn't see much benefit in such details. But it probably wouldn't hurt if you say "sorry I already am in a relationship"/"sorry I'm a Jedi prohibited to have relationships".
However, having answered your direct question, I must agree with other commenters - in this specific case, it may be worth giving the person a chance to get to know you. They either are already greatly attracted to you, and/or are courageous, or both - both those factors are worthy to recommend in a partner.
One thing I would NOT recommend is "ghosting" the person (just not calling the number/emailing). If someone took the courage to approach you, at least be courteous enough to not leave them guessing and hanging. You want to encourage that in people; not discourage.