Teaching people about bitcoin is great, but i will not let people into my home for that purpose, that is too risky for a bitcoin user like me, even if they are neighbors, i may just be exposing myself to attacks in the future, people think every bitcoin user is rich you know.

A change of venue may be necessary, a group on any social media app may be even better, anything to maintain your privacy and anonymity and to protect yourself.
This is not a serious issue, no one is saying that people who use bitcoins or invest in bitcoins are wealthy, that's not true.
If you stay in the mansion and have a super car, they can also attack you to steal your money, you don't need to own bitcoins. Many people get rich without owning bitcoins. You are giving knowledge and teaching about bitcoin, you are not calling them to your house to show how many bitcoins you have, so there will be no attack here.