Whether this is rude (or even just impolite/unfriendly) depends on where you live.
Where I grew up in a smaller town, it was definitely considered normal to say thank you to the bus driver when getting off. Note that these were buses with only one door at the front, so you had to pass the driver to get off.
Living in London now, I'd say it's more normal not to speak to the driver at all. Note that London buses almost always have two doors (one at the front and one in the middle), and you exit from the middle door, which is nowhere near the driver. Saying thank you would mean shouting across the bus!
I've noticed that some people say thank you to the driver in London regardless, and assume this is people brought up in smaller towns like me, but otherwise I think it's perfectly fine not to thank the driver unless they've done something in particular for you (e.g. waited while you ran for the bus).