You shouldn't offer them any cash at the end of the ride.
I've frequently given rides to co-workers in a variety of situations. I can't promise everyone will feel the same way I do, but this is my perspective.
First off, there are a lot of ways that giving someone a ride is inconvenient for me, even if I don't have to drive out of my way at all:
I feel obligated to hold a conversation, which takes a lot of effort for me, especially at the end of a day of work
I'm concerned I'll be judged on how dirty/old/small/whatever my car is
I have to think more about my driving because I want my passenger to feel safe
I have to consider the comfort of my passengers when adjusting the temperature or radio
I have to coordinate with a person about when we're going to leave and can't change my plans last minute
So you can see, even if I don't have to drive out of my way at all, it can make my commute a lot more stressful. Your $5 may have more than covered any financial burden you imposed on your coworker, but I highly doubt he'd feel it's worth it to give a ride for that amount. I'd personally feel really weird accepting something like that. So what should you do instead? I'd start by reducing the inconvenience and making them feel appreciated:
Give people an easy out when you ask for a ride. It's not your co-workers responsibility to get you home, and they shouldn't feel like it is. Make it clear you're okay with a "no", and you'll only get rides from people who are truly okay with giving them.
Don't force them into a conversation. Maybe you could ask if they'd prefer to talk or listen to the radio. This is the biggest reason I dread giving people rides: they invariably jabber on constantly and ask me personal questions.
If you are comfortable with their driving, say so, especially if it's obviously a stressful situation (lots of traffic, missed a turn, unfamiliar area, etc).
Sincerely thank them at both the beginning and end of the ride. Acknowledge the inconvenience to them and the benefit to you, even if they didn't drive much out of their way.
Don't make them look for you or wait on you when it's time to leave.
If you want to show your appreciation, get them a gift card to their favorite coffee place or a six pack of their favorite beer or something like that afterwards. People value food and drink much higher than an equivalent amount of cash.
If you're going to pitch in for commuting costs (which you should do if this is a regular thing), say so while requesting the ride, and explicitly say it's to help pay for the cost of the commute. I'd probably offer about fifty cents a mile (USD), with a few bucks added on if they have to pay for parking at work. (I picked that amount because the IRS mileage rate is 58 cents per mile, which you should pay at least half of, plus a bit extra since they are driving and the rate is probably not all that generous.) Or offer to pay the entire amount of what the gas for the trip costs them, or something like that.
Don't just hand them a few bucks at the end of the ride and say thanks, because it'll feel like you value their effort and inconvenience at only a few dollars, which will be disappointing. I'd phrase this as, "hey, can I get a ride? I'll pay ten bucks to help cover commuting costs." Or if you want to pay for gas, just replace the second sentence with "I'll pay for the gas for the trip." Then once they agree, ask them how much that is for their car. I'd also pay them at the beginning of the ride, not at the end, so the debt is settled and doesn't cause any stress during the ride.
One last note: don't worry too much about the person who doesn't seem keen on giving you another ride. He may just prefer driving by himself, and doesn't feel like whatever you'd be willing to pay would make it worth giving a ride. Unless he offers rides to others and refuses you, it's probably not personal.
