I have a mixed group of friends from different countries (JP, CH, US.) I found it difficult to keep the conversation when inviting them to my house (I live in the U.S.) Certain people do not like me to speak English with them and they will not respond. Other people do not like me to speak my native language for obvious reason. I try to hang out with them in different sessions but can't avoid some occasions. 

What do you feel & what is the proper way to handle this situation if you are an English speaker and non-English speaker? How to switch multiple languages in one conversation casually?