Breaking (and creating) rapport
I studied NLP to master's level. (see edit)
My guess is that you are naturally good at creating rapport but don't have skill in breaking it. If you master that then you can do it subtly without causing offence.
I have successfully used the technique many times based on my studies of NLP (Neuro-Linguistic-Programming). You can create and break rapport multiple times in the same conversation in order to take charge of its direction and length. If you do it right the other person won't even notice.
You can search online for "creating and breaking rapport". The first result I found looks ideal for your case. Here it is:
There are times when breaking rapport is crucial. Some people get so
good at creating rapport with others that conversation flows too
fluently and they forget how to bring it to a close. So they end up
listening to story after story, or hearing more detailed information
than they ever needed to know. Either they are afraid to offend by
disengaging or they don’t know how to finish an interaction
gracefully. Time gets wasted. Appointments or meetings over-run. Work
doesn’t get done. Irritation and boredom step in. Then people risk
having to break off quite abruptly, which causes bad feelings.
I didn't vet the article for correctness and I suggest you read several to get a wider view. The point is that you can do it at many levels of subtlety. Perhaps the most obvious is to stifle a yawn every time the person strays off track. More subtle methods include rate of breathing and rate of speaking.
EDIT in response to thought-provoking comment by @Johanna
Although this does not address the question directly, I think it is valid for me to follow up on a comment that throws doubt on my answer. As someone who has an MSc in mathematics and computer science from a UK university, I applaud rational scepticism. I gave up religion as a child, because I tried praying and didn't get the results I hoped for. When I tried some simple NLP techniques, I did get the results so I decided to pursue it further. I use it only occasionally now - usually to simplify dealing with conflicts. I am not using it here.
NLP has never as far as I know made any claim to be a science in the sense of hypothesis testing. It is for me a purely pragmatic system which I personally have found to give useful results. I could provide many personal examples but this is not the place. For anyone who is sceptical there is a simple remedy - try it and see. When I referred to "master's level" I was careful to use lowercase to avoid confusion with "Master's level" which implies university study or its equivalent.