That's annoying. Anyway, some privacy settings that might help in such cases:

- Phone Number: Nobody
- Calls: My Contacts
- Voice Messages: My Contacts (option available only with Telegram Premium)
- Messages: My Contacts or Premium users (option available only with Telegram Premium)

You also did the right thing by changing your Telegram username to ghost them, but if you have these settings, even if they find your handle again somehow, they will probably not be able to text or call you.