So, this male was talking to Nina first, and pointing at Mary. This sounds very much like the male might have asked Nina if it was okay to make a move on Mary. Which is (in my opinion) a good thing, he's a polite man and didn't want to offend Mary/ do anything rude. Nina brought him over, and at the end of the ride, he offers Mary a card and asks her to 'call him'. I'm assuming that's it, that he didn't wait or push Mary for a reply.
I, as a female and ex-retail worker, feel a certain amount of respect for this guy. He was polite, subtle and at the same time had the balls to ask Mary in a respectable way 'would you like to get to know me better? I certainly would like to get to know you better', gave the choice completely to Mary and left it at that. Trust me, I've been through worse when working retail (Hey honey, wanna have sex tonight?)
But, as MichaelKarnerfors rightly pointed out in the comments, no matter what I think of the guy's behavior, this doesn't mean that Mary had no right to be upset. Mary might well have found her boundaries violated by such behavior or found the behavior completely unacceptable for a workplace. The two paragraphs above are just from personal experience and to share my thought on the subject, and to provide with some insight into how I'm thinking and some background to this answer.
How could Mary have politely communicated to Nina that what Nina had done was inappropriate and unprofessional when asked "Did you get upset?" and ask that she doesn't do that again?
If Mary got really upset she probably showed it on her face. So, Nina might have seen this and approached Mary to ask if she was okay. This means that even Nina might not have been sure that what she did was okay to do, that she misjudged Mary as somebody that might appreciate such a move. This was probably the best moment for Mary to have just stated the facts:
Remember that male you asked me to process his purchases? He asked me to call him. I'm freaked out right now, I'm not feeling very safe anymore.
If you want to find out if Nina knew of the man's intent, it's better to ask this than to outright accuse her of conspiracy with the customer. Maybe Nina didn't know the man was going to make that move. Ask her if she did, follow up the question by stating:
I saw you talking with the customer, were you aware he was going to do this? If you were, could you please tell them I'm not available next time? I really don't like this happening to me, it freaks me out.
This isn't accusing Nina, and also offers Nina a solution that she can use the next time such a situation arises. Don't go around throwing terms like 'unprofessional' and 'inappropriate' directly at Nina. Keep those for when she makes the same mistake multiple times. For now, be gentle in communicating that you didn't like this 'surprise' at all.
In response to your edit:
If, as you suspect, the customer didn't tell Nina about his plans, Nina didn't have much to prevent him going to Mary's register. After all, if a customer points out Mary because he knows her from a previous visit, as long as Nina doesn't have proof that he's 'misbehaving', she can't politely decline his request. Your first goal here is to find out what the conversation between the customer and Nina was about.
Have Mary tell to Nina what happened. But don't accuse Nina of 'you should have had more tact'. Just state that Mary was upset by the incident, and ask Nina what happened between her and the customer. Let Nina know of what happened and ask her to work with Mary to figure this out. And if anything more happens (read: if the customer starts to stalk Mary), keep Nina up to date. I don't know how much power a store-manager has over there, but I've once seen my boss deny entrance to a customer that was being very creepy (almost stalking) a colleague of mine.
By providing Nina with the facts, Nina might be able to prevent the customer from visiting Mary's register again (or at least give him a very stern warning not to bring the 'call me' up again --> I've seen my boss do that and it worked). The only things Mary and Nina can do here are retroactive and aimed at that specific customer since he already crossed the line. Nina will never be able to protect Mary from customers doing this again (or worse) if she has no clue they are going to do so. But once they do, she can (and must) protect Mary from further harm.