"I'm sorry for your loss" is definitely appropriate when offering condolences:
While this phrase has become a cliché, it is also a simple and succinct way of communicating your empathy. If you are at a loss for words, telling a person “I’m sorry for your loss” can let the person know that you care. 1
It is indeed a standard fall back for when you don't really know what to say, but want to let the person know that you care about them. Also, the sorry in this case acknowledges their grieving, and extends sympathy:
Feeling distress, especially through sympathy with someone else's misfortune. 2
The chances will be very small you'll be seen as the one wanting reassurance. Instead, you're extending sympathy for their misfortune, letting the person know you care about them.
I think you need to work on your timing though. You say: 'there they are, conversing with me, and suddenly here I am reminding them of their recent bereavement'. Condolences and sympathies are usually offered at set times:
- During a visitation. This is after death but before burial.
- While greeting the mourners briefly before a funeral or memorial service.
- At a gathering following the burial or memorial service. 3
Since you're talking about friends and extended family, if you can't make any of these/aren't invited to any of these, a written condolence would be preferable. In The Netherlands, it is proper to go buy and send a card with your support the same day/day after the news of the death reaches you.
When you haven't seen someone for a while, and want to express your condolences upon meeting them for the first time since their loved one died, you do so upon their arrival. Over here, that's appropriate for e.g. coworkers that return to work after bereavement leave. Timing is always still a bit awkward, but usually you wait until someone has walked into the door, said good morning and has taken off their coat. The same could be applied to friends visiting you. Do NOT randomly put your condolences into the middle of a casual conversation. It ruins their value and makes it much harder to take them seriously for the bereaved. Of course, if the bereaved brings it up themselves, it's appropriate to acknowledge their loss and grief.
As for the alternatives to saying 'I'm sorry for your loss', there are plenty out there on the internet.. Find one that works better for you personally, and you can use it in the same way most people use 'I'm sorry for your loss'. You can combine another sentence with "I'm sorry for your loss", or keep a small mental list and pick one or two that seem most appropriate for the situation.
Your example of "I'm sorry to hear that" may miss the part about 'your loss', it misses acknowledging the other person. To be safe, I personally stick with condolences that explicitly acknowledge the other person and their bereavement. "I'm sorry to hear about your loss" feels like a better reply than "I'm sorry to hear that", even though most people will interpret it just the same.
When someone offers their condolences to you, you accept them gracefully and genuinely. 4 A simple 'Thank you' works. If you're e.g. glad to be back at work and don't want to talk about it much, usually 'I'm glad to be back' is enough of a hint. Since you're the person grieving, you can basically do everything from telling people how much you still miss the deceased to sharing fond memories. Don't worry too much about it, things may get a little awkward but this is one of the few times in life where that shouldn't matter much.
And sadly, as always... practice makes things easier. The feelings never get easier, but after having given and received condolences a few times, the rituals do.