I think yes you should be scared of hackers if you have a lot of BTC because why are hackers going to hack someone with 1 btc instead of someone with 1000 btc that seems sensible enough for me that they will hack the one with 1000 btc.