The answer is actually surprisingly simple: just say "no" with a gentle smile.
Anything else you add to that "no" will only open up an argument. What you're trying to do here is be really clear about your boundary: you don't let anyone else drive your car.
The only exception is if you have a strong reason that they can't argue with in the first place. For a car this could be that your insurance only allows you and maybe your wife to drive your car.
Sorry, my insurance doesn't allow anyone but me and my wife to drive my car. So I really can't let anyone else drive it.
Aside from such strong reasons anything else you add to the "no" wont work.
For example, if you add "I don't want you to scratch it", then they'll just start explaining that they'll be extra careful this time and they wont stop asking for it.
The same goes if they try to ask you for reasons or just won't accept the no right away. Don't argue with them, don't provide reasons that they can try to refute. "No" is a full answer by itself.
If they call you a miser just because you say no, then that's because they didn't get what they want and are trying to manipulate you into changing your answer. Don't fall for this! You did nothing wrong with setting your boundary.
If they do keep pushing you and try to start an argument you just repeat firmly, still without giving a reason:
I already said no, (aside from my wife) I don't let anyone else drive my car. Can we please talk about something else now?
Being an introvert isn't an excuse to give everyone else what they want just because you want to avoid an argument. If anything it's a really good reason to firmly say no right then and there. Think about what would happen if you don't clearly set your boundary right there.
They'll just try to convince you in any way possible to give you your keys. This will either lead to endless discussions in the future trying to convince you, or if you cave in will let you worry about your car, not only until you have your keys back but every time you'll visit them because they might ask your keys again.
Both of those things are exactly what you really want to avoid as an introvert because in those situations you can't stop it anymore. So be firm right from the start. Don't allow any discussion in the first place and your life after those really difficult 5 minutes will be a lot easier.
Addendum on your comment:
I have noticed my spouse giving them car keys to do XYZ or park it around which I do not like so with my new car I wanna set boundaries but being introvert I have a problem discussing such matter.
This is actually a question on it's own but my last paragraph applies here as well. What do you think is the most difficult overall:
1) Trying to discuss with your spouse how hard it is for you to see someone else drive your car (and thus that they shouldn't give your car keys to anyone else)
2) Seeing your spouse give your keys to their siblings and worry they'll do it every time you visit their family.
I'd say I would do anything to avoid (2), no matter how hard it is to explain this to my spouse.