If your car isn't going further than the boundary of the footpath (as pictured in your diagram) then the manoeuvre does not concern the owner of the driveway as footpaths are public land. However, it is illegal to drive on the pavement. Any pedestrians nearby may find your manoeuvre rude.
I used to have a neighbour who got annoyed with people turning around on their drive halfway down a single-track dead-end road, but in this case people would often go right into the drive and do the entire 3-point turn manoeuvre on their property. I think this drive was particularly popular as it was done up nicely, and was flat and open.
I think it is a question of pride, and indignation of the drive owner, who has possibly spent a lot of money on their drive so that they could benefit from it, not so that random strangers could benefit from it, and also protectiveness over the vehicles, and other property they have stored on their drive, as they did not expect their private property to be put at risk by a random driver, regardless of how careful the random driver is.
Others may take the less-offended opinion that it's not harming anyone as long as you're careful, but it is still a bit rude to turn around in someone's driveway. It's also trespassing, however brief.
My advice would be to find a driveway that looks less-cared-for to turn around in, this probably means the owners are more casual about it.