How to switch conversation from language to language in peers?
In seperate conversations, with seperate groups of people I agree with @PaulaHasstenteufel's answer. However, OP said he has no problems speaking in seperate sessions, in his situation it seems as though this is one single group conversation, so my answer is: Don't, pick one base language that best suits the group as a whole and in all conversations directed to the whole group, stick to it.
They are all guests in your house, and I guess that you invited them with the intention of these friends from mixed cultures interacting with each other. If they are aware of this and have accepted your invitation, then it would be considered rude to speak to this group in a language that benefits only a select few people while considerably handicapping everyone else. Also, switching back and forth between languages within a single conversation too much is confusing and difficult for such a mixture to follow.
Admittedly, no matter which language you choose it will upset some people, making contributing to the group conversation more difficult for them. Unfortunately, as you have found this is not one of those times that you will be able to please everyone. Instead it is up to you the host, to choose the language that will result in the smallest amount of collective difficulty for the group as a whole.
If your Chinese friends refuse to interact when (in this context) you speak to them in Japanese, they are the ones being rude. You should hold your ground, using the reasons you arrived at the decision to choose Japanese as the base language for the group conversation in an effort to explain.
We are all eating dinner speaking together, Joe does not know a lick of Chinese and we all know how horrendous Aito and Daichi are at English. So Japanese is our only real choice in order to include everyone.
When you split off into seperate conversations, people will re-assess the best group language to use and this is more than fine, just keep in mind how seperate the conversations are... if Daichi took one sip of beer and suddenly everyone is speaking in english, it serves as a barrier for him to re-join the conversation.
