I realized another drawback: the receiver needs their private key to know they've received a payment. That makes a watch-only wallet impossible,
I think you are right. Generally to track all incoming payments, you need a master public key from which all other public addresses are deterministically derived. To (re)calculate a particular address, you need two pieces of information, namely your xpub (ypub, zpub) and derivation path to the address. Anyone who has both can reconstruct the address and see all associated transactions. In some cases, one even doesn't need to know a derivation path. In silent payments, you also need only two pieces of information to calculate the address, one of which is a public key of the sender and the other is a private key of either sender or receiver. This requirement (having to have a private key) makes watch-only wallets impossible, let alone the fact that each particular sender creates unique private-public keypair for the receiver.
and even if you use Bitcoin Core, private keys are stored encrypted until you enter your password. Keeping the wallet unlocked adds a security risk.
Hmm. You're not obligated to do a continual real-time scanning to make use of silent payments. Just scan the blockchain once a month, unlocking your wallet for a short time.
Alternatively, we can separate the silent public key into the "scanning key" and "spending key" to mitigate the security risk.
We can extend the silent payment address with a scanning key, which allows for separation of detecting and spending payments. We redefine the silent payment address as the concatenation of X_scan, X_spend, and derivation becomes X' = hash(i*X_scan)*G + X_spend. This allows your internet-connected node to hold the private key of X_scan to detect incoming payments, while your hardware wallet controls X_spend to make payments. If X_scan is compromised, privacy is lost, but your funds are not.