Funnily enough around the time this incident first happened, there were rumours swirling around my extended friend group that I had a substantial number of BTC. It was all untrue and I made sure to shut down the rumours ASAP since there were politician's kids in the circle and I was afraid some might take notice.
Glad I took those steps because who wants to get roughed up just for them to find out the rumours held no water. I never found out who started the rumours either.
It would have definitely been a person close to you people just can't tolerate others' success. They get jealous soon and some spread rumours just to annoy us.
Good that the rumours stopped because if the same fate would have happened to you then your life would have been at risk.
You know that you don't hold a big number of BTC but how would you prove that ? There is no way to prove that.
They could suspect you of holding those substantial amount of BTC in some unknown wallet.