As far as bullying attempts are concerned, in my experience, what works best is to escalate several levels above what the bully expects.
Verbal threats usually do not work. Anyone can say anything without it carrying any weight. However, actions matter.
Here's an example:
My parents own an appartment in a collective ownership building. An inspection resulted in a list of mandatory upgrades we had to do. The property management company sent a 4000€ quote. After a bit of investigation, I found out that only €1000 out of that was part of the mandatory upgrades, the rest most likely not mandatory, but simply extortion (ie, €3000 for 3 electric radiators). However, it was impossible to make the property manager say what was mandatory and what was not, most likely because she would get a cut on the extortion, so it was in her interest for me to believe the non-mandatory stuff was actually mandatory.
On an old man with cancer who can't leave his home, this legally constitutes "abuse of weakness", which is punished by 3 years jail time, a hefty fine, and other goodies.
Knowing the manager was trying to extort, therefore an enemy, there was no point in further discussion. So I simply escalated over what they expected. I had the lawyer print out the legislation, write a nice summary ending in "you have been warned" and send a registered mail copy to the property manager. Then I sent a copy to everyone, and called everyone in the building to check what kind of stuff they were trying to force them to buy at inflated prices, thereby saving a couple thousand bucks to quite a lot of people who were being extorted too... and killing the manager's business.
They simply dropped the ball and pretended nothing had happened, problem solved.
So, concerning your "friend", here's what I'd do if I were you:
If someone doesn't understand on their own that posting pics of someone's mom on a cum tribute site is not something you do, then there's no point in having a polite discussion to convince them of this fact. Especially since you already tried. Most likely, this guy thinks it's funny.
It's always like that. There's nothing wrong in being civilized, but that only works on civilized people.
So, you should first archive everything: reddit posts, incriminating answers, any mail, text, or voice where he confessed doing it, etc. Several copies, at least two in the cloud, for example MEGA and google drive, on separate storage accounts from your mains.
If your mother is getting anonymous phone calls, then it means someone shared her phone number. That goes even further past the red line than publishing the photos. If you don't want the cops to get involved, IMHO you're being way too merciful.
The anonymous phone calls could be explained in different ways. First, EXIF tags with GPS coordinates added by the phone to photos. Facebook is supposed to strip those. Second, someone using a reverse image search on the photo to find your mother's public profile, then getting the name. Third, your "friend" actually sharing the name, location, or phone number.
It is quite important you figure out why she's getting those phone calls, because if the address or name was leaked, someone could show up uninvited.
So you can make him confess that he did it via text message. If he already confessed via text, this is ideal. But you need evidence to link the anonymous reddit account and the guy in question. Back up everything and make several copies.
Then you can simply show him that you have all the evidence. Basically, don't make threats, instead simply make statements, like "You posted the photo, I have all the evidence you did, and now my mother receives anonymous phone calls, what do you plan to do about it before we call the cops." Simple as that. Wait for a reaction and act accordingly.
It would be better to do this by text, or he could try to take the phone and delete the messages.
If he doesn't get it, involve your mother, who will most likely send the evidence to his parents. At this point, something should happen that will make him understand.
