Do not worry too much, it will make you tortured.
This may happen to beginners but as long as you secure your wallet no one will be able to steal your assets.
Many options of wallet that can be used.
yes your right be comfortable if you think your key they dont know, but always be secure always.