It depends what you mean by "without it escalating". Of course you don't want it to escalate into a fight, verbal or otherwise. And you don't want to make your situation worse. You need him on board to put it right (begrudgingly or cheerfully, it doesn't matter which, so long as he fixes it). But if you got bad service from, for example, a store assistant, you would "escalate" your complaint to a manager. Escalating a complaint is normal, and can achieve what you want.
As a contractor, I assume he is a self-employed tradesman and has no superior as such to escalate your complaint to. But in most countries, he will be subject to certain building regulations. Tradesmen in some countries must have various forms of registration which require they comply with such regulations. You may have statutory consumer rights. So there are people and places that you may be able to escalate these things to.
Threatening to take someone to court or report them to a regulatory body isn't the first interpersonal solution I would recommend, but knowing your rights is very important before you get into any conversation like this. The old proverb is, forewarned, forearmed. Here on IPS we can put words in your mouth but you never know what is going to come back. Having all the facts, knowing what you can threaten if it comes to that is very important.
From the examples you gave, he sounds like he will try and blag his way out of anything you put to him. But people who try this are less able if they can see you know what you are talking about.
Do your research into your consumer rights. Perhaps talk to some other contractors about the way they would do things (just call a couple as if you are interested in having the work done, and ask the questions). Decide what you are prepared to go through with (ie small claims court, or report him to a regulatory body if there is one). Then contact him and perhaps say something like:
I am not satisfied with the work you have carried out. It isn't working as it ought to. I have given you the opportunity to put it right, and you have not done that.
That is the grounds for your complaint. If you do your research (and I'm saying this because laws and the language used by the law varies between countries) you should pick up standard phrases such as "does not meet agreed expectations", or "does not work as specified". Modify what you say to include the phrases you find. This will show him you have familiarised yourself with your rights.
Then state your expectations. Either ask him to fix it once and for all, refund your money, put it back, or tell him you are not prepared to pay him (if you are sure your rights support this).
If he claims that he does not have to, or will not do something you know as a consumer you can expect of him, then you could choose to say something like:
I was hoping that you would be reasonable and put this right. If I must, I will take this further [and specify what you believe is the correct course of action and what you are prepared to do]
If you want to give him one last chance to do the right thing, you could just add:
Are you sure that this is worth your reputation?