This is big.
I'm in the exact same condition of low self-esteeem as your dear friend, so I apologize if I might sound biased, I apologize if this all might sound harsh and I apologize even more if you think this answer will be utterly unhelpful, nevertheless I feel compelled to answer as sincerely as the top of my heart can.
I'll now analyze your post (both premises and questions) by breaking it down in points and answering point by point.
Compliments just make her feel that it couldn't possibly be true
Because, actually, they might be not true.
"You have two beautiful eyes" is subjective, what is beautiful to you may not be to your friend.
"You're soooo good at computers" is also subjective: you might have a lower understanding of computer science than your friend's one so you possibly can't assess whether your friend is actually good at it or maybe just an amateur.
Please, try to accept that you might actually be wrong, and therefore your compliment can be wrong likewise.
You're in your own shoes, not your friend's ones.
Therefore what you see as true your friend could perceive as false.
By the way you generically used the word "compliments" I assume you already tried this step a number of times and it didn't work, so you may want to try to give this one up. You two are likely on very different perspectives/points of views/opinions that cannot be reconciled.
And by keeping on grasping on your own views which your friends disagrees from will only put additional strain on the relationship.
We dated last year
This is devastatingly bigger than anything else in this post.
The end of your dating period is extremely likely to just represent one more reason for which the self esteem is low and the perceived desirability from men is low.
You are one who went on an unsuccesful dating period with her and at the same time you want to be the one giving her confidence about desirability from men?
Please, imagine this logical mental process of hers: "Hey, how can YOU tell me I am desirable by men if YOU, a man yourself, desire me no longer?"
Let men other than you talk to her about this subject, because you represent that part of men that considers her undesirable, you'd be lacking coherence severely here.
I merely want to build our friendship because I value her immensely as a person.
As per my own experience and that of some friends of mine, a friendship after a dating period is not always possible.
She might not want what you want, regardless how much you value her.
Therefore, try to not take for granted that you can actually be a good friend.
Which leads us, eventually, to your questions:
How can I can be a good friend who doesn't take her kindness for granted without triggering her low self esteem?
How can I express affection/fondness for her without upsetting her?
How can I compliment her intelligence/character/ability without making her feel worthless?
Please notice that she may not want it, and she may communicate it not explicitly. It could also be inconscious to her that she doesn't want to you to be her friend.
So, before asking yourself "How can I...", try asking yourself "Can I...", because you may not be the one who can help her, as bitter as it can sound.
My suggestion (talking as the one who relates to her feelings): give her space and time.
Regardless whether you are acting on good heart and true good feelings toward her, help from you is not working as she keeps on rejecting it, so all you can do is taking a break and giving her the same break.
Take a distance and keep it for a while.
She will be in the position of not being able to rely on your "help", which will lead to two possible outcomes:
- she will be fine with that distance and won't come back: it will be the sign that, regardless of your good heart, you where more of a weight (on her already strained self esteem) than a helping hand. If you do love her and value her, as bad as you can feel about it, let her be. Dalai Lama says "[...] if you can't help them, at least don't hurt them."
- she will come back: that will be the sign that, regardless of her on-the-spot reactions, she enjoys how you compliment her, hence you can come back to assist her as naturally as you feel, because she did decide to deliberately come back to that assistance.
A final consideration: personally, I prefer facts than words.
Facts are stubborn; facts are strong and, conversely, words/opinions are weak.
If she's any like-minded to me, only things that will heal her self-esteem will be facts, e.g. a suitable man actually approaching her and actually successfully building a love story with her, thus actually proving her she is worth.
Wishing all the best for both her and you. :)