yes you are right, the best option is not to use the wallet extension. better use a mobile wallet like trustwallet, safepal, tokenpoket or metamask app. This method is safer than using a browser. I've done it, and till date I've never had a problem with hacking.
Your suggestion is obviously very useful, but Metamask also provides a wallet extension that is very safe to use even though it is located in the browser, but so far I have never had a problem with Metamask because I also use double security to enter my wallet.