You are right. It is an inappropriate question. Similar to the "When will you start on kids?" question.
You can talk to the family members and friends asking the question one on one (without your partner) and tell them that it makes you and your partner uncomfortable, and that you are just not interested in making that step right now. You dislike the pressure it puts on your relationship and would like them to stop asking the question. If they object or see it as a chance to bring it up again, you repeat that you are not interested or thinking about it right now.
When someone asks you the question when you are in company, you can curtly say you (both) are not interested in marriage right now and change the subject to something else that occupies your time and attention at the moment. Take a minute right now and come up with some positive subjects (a project at work, an upcoming vacation, a new hobby/interest, ...). Try to find something that not includes your partner to take the spotlight off of you both as a couple even more. It's important that you have something to say after you tell them off, so as not to make things awkward. So again, prepare something to say. Bonus about this tactic, you don't need to discuss it with your partner and talk about "the question" again.