Given an international (work-) environment, I have plenty of coworkers1 not speaking German. It happened already many times to me, that I was having a English conversation with someone not speaking German. Almost always, when a coworker approached me to ask me about something or to join that conversation they start talking to me in German. Except for the fact that really almost everyone does it, and it seems to be a German cultural thing, I personally find it very rude. Especially since they not rarely just interrupt the conversation I am having with others.
What I tried so far was, I explained quickly in German that I am having a conversation in English cause the other person doesn't understand German. But that statement in German followed by the other person responding to it with an excuse or even justifying why they talked in German to me, usually takes enough time that the person I was speaking to before in English, just left us or engaged a different topic in that time with someone else, so me not being able to finishing a conversation I really would have liked to continue.
Yesterday I went one step further, as I was on a tech convention and it happened to me multiple times. So that at one point I explained what I just had written above, in English (rather than in German) to someone approaching me in German. I was hoping the result might be that the person I had the initial conversation with will this way be aware, that I intend to continue the conversation and that person also can feel invited to join the conversation we just got interrupted by. Sadly this just made it a little bit awkward for everyone. The person heading towards me, responded (in German again) That he is "Not getting along so good with $SpecificEnglishTerm you used" and after I told him then, that in that case he might please wait till I finished my other conversation, he just said "Is nicht so wichtig" what would be like saying "Nevermind then" before he left. The result was me feeling awkward, as it felt to me like I was kinda rude, my conversation partner made to me the impression he was feeling awkward cause I just sent someone away due to the language barrier and the person approaching me, despite I couldn't take note of it, very likely was feeling for some reasons awkward, too.
So my question is, given the above scenario:
How can I prevent awkward situations and keep track of an English conversation, when someone interrupts me in that conversation talking to me in German?
1 I would like to keep this question more general and not workplace related, tho.