Always trade in a public place with plenty of CCTV.
For the "somebody comes around the house" scenario, I like the idea of using the Shamir secret sharing for any serious amounts. Make it something 4 out of 12. But instead of putting them in locations that you control, hand the parts to other people who do not know each other (also think lawyers, notaries etc.). It will be impossible for you to spend the coins without making the rounds and see these people. Not entirely risk free of course and you still might get shot.
So, yeah, the fewer people know, the better.