Both the private keys and the seed phrase must be kept secret and safe from theft. Without them, you can't prove ownership of the funds in your wallet  and you won't have control over them. There are other use case of the private keys, that is for example. If you lose your device you can use the private keys to restore your wallet on a new device and regain access to your funds. Similarly, if you switch to a different wallet provider you'll need the private keys to import your wallet and use the new service.

You just have to remember that your funds are not stored in the wallet itself but are recorded on the Blockchain. The wallet only displays the amount of funds associated with that address.