That's happening, I even have a lot of group now in telegram that I did not remember I joined.
I don't know how to configure my telegram so people will ask permission first before adding me, I not a heavy user of telegram so I don't explore it a lot.