"give me back my phone or else I will tell the police about this"...
What you describe is indeed blackmail.
"If you do/don't do , I will " is the blackmail formula.
The proper way to handle the bus issue in your example you gave, would be to ask for the phone back, and if not returned, go to the police with the facts, without making the situation worse.
"that the thief can go to the cops first and report you for blackmail and you will be locked up for it
Hah, so this is the kind of country you live in ? Rewarding thiefs by throwing victims in jail ?
Keep blackmailing.

My personal opinion is that you could use some "how to behave in society for dummies" book.