For extra privacy you can also delete your username alltogether so only those who know your phone number or that are in the same group as you can contact you first.
that will give him less privacy not more.
with telegram, you would much rather people know your username than your phone number.
if your phone number gets leaked by one of your contacts, that would be much more annoying to deal with as you can't change your phone number that easily as changing your username.