He seems friendly [...] but when he takes a loan, he just plays hide and seek to them.
This looks like someone only motivated by money and interested in getting money. As you said, you got a hard time getting your money back, asking multiple times.
Ask yourself how you would feel and (re)act if you borrow a book/movie from a friend, then damage it beyond repair or lose it ? Would you excuse yourself, and buy it back, or hide and seek ?
I want to avoid giving him any loans.
Just say : NO. No need to argue, no need to explain, no need to be rude. It's your money, it's your decision. Just make sure you pick the appropriate moment to tell him, if you can (delay the answer for a couple of days, if needed). Don't put yourself in a situation that makes you more uncomfortable after (try and get your money back) than before (say no, and don't lend the money).
Nevertheless, if you still wish to explain why you don't want, I would just say I need the money for myself (this is actually true, as you work to make a living), and can't afford to share/loan.
I would not go into stuff like: but I had to fight to get my money back last time or but you didn't give the money back to X last time, and just play hide and seek with her/him now. Endless arguing...
As my ol'man used to say : You lend money, you lose a friend. If your really want to help him out, give what you want/can, but don't ask it back, and don't expect it back....