I recommend you to use Bisq. Exchange, Decentralized.
It is the only fully decentralized, non-custodial, no-KYC exchange I know of and it's been delivering great service for years so far. It has a built-in wallet - fully stored on your PC; nobody else has access to your funds.

If you want to be safer than that, consider looking into Open-Source Hardware Wallets and AirGapped Hardware Wallets. There are also various threads in the Hardware Wallet section where people asked which one to buy and contain lots of recommendations and explanations. Just have a look around.

I'm personally mostly using the Passport Batch 2. Definitely recommend against just using a USB flash drive. And whatever you do, create a few seed phrase backups. A cheap and long-term secure way to do that is using steel washers.