Let's take a few steps back...
The result doesn't especially matter to me, but the uncertainty does.
You also say that you're annoyed and have been trying to make note of which bathroom they use and the like.
These are somewhat incongruent. It doesn't matter or it does. There's nothing necessarily wrong with having preferences, but be honest with yourself about your preferences. You may be bisexual and enjoy dating both men and women, but it sounds like you may not be pansexual, as in, you may be less comfortable dating someone who may not be comfortable in either of those binary categories.
Taking another step back, you need to know what you mean by "man or a woman". Are you talking about:
- (the anatomy of an individual's reproductive system, and secondary sex characteristics)
- (typically used with reference to social and cultural differences rather than biological ones)
- (how someone outwardly expresses their gender through dress and/or behavior)
It sounds like this person's presentation is androgynous, that obviously leaves you clueless about their gender, and sex. Their presentation may be telling you that they'd rather not be identified, or labeled, as either male or female... Or it may not....
The thing is, you don't know without asking, and asking will be an awkward conversation no matter what the case may be. Whether they strongly identify as a particular gender, and you haven't picked up on it, or if they'd rather not identify as a particular gender... this is likely to be an uncomfortable conversation.
There's no sure way to get around it. Many people don't use the restrooms, clothing, or other constructs that correspond to their assigned sex/gender, so trying to use these as sign posts may not give you the information you want.
If you really want to know, you're going to have to ask. Just be sure you know what you're asking, why you're asking, and what you intend to do with the information once you have it. Does it really matter to you, and if it does, why does it?
As far as the conversation goes, be honest, but be gentle. Try to be self aware and aware that these are questions that may be intensely uncomfortable for the person you're asking. If the answers really don't matter, consider letting them disclose in their own time and in their own way.
As an aside...
A few answers here mentioned checking state issued ID. That isn't exactly a great suggestion for a few reasons.
Many countries allow folks to change their names, and the sex/gender listed on their state issued identification, birth certificates etc. without requiring surgical transition. And some places allow nonbinary gender markers on IDs.
- This means that the sex listed on their ID may not match their plumbing. (So, if that's what you really want to know, ID won't necessarily tell you that.)
Some people who are in the process of transitioning and have yet to do the legal paper work shuffle will not be likely to show you their ID. If the ID card lists a name and sex that they no longer identify with, this may be an intensely uncomfortable/delicate question and asking may be seen as a pretty huge invasion of their privacy. The same goes for some agender, intersex, and gender-fluid folks.
Basically the information on their ID card may lead you to make inaccurate assumptions, and asking to see their ID may be seen as incredibly invasive.