disclaimer: I'm just learning about silent payments today, from this thread.  So my understanding is incomplete.

I read some comments indicating that with silent payments, only one recipient address would be needed, for privacy purposes.  And we could get rid of key derivation, gap-limit, etc.

I don't think that is adequate for all situations.

case in point:  Monero started with a single address and added stealth addresses which ensured unique addresses visible on the blockchain, much like silent payments.

Over time it became apparent this was insufficient and they eventually added sub-addresses so that the recipient could give out a unique address per tx.

why?

Let's say Alice has ETH and is buying BTC with some kind of "anon" exchange, like shapeshift.io was originally.  She does this once a month for 5 months.

Each time she provides the exchange her single, unique BTC address.   The exchange uses silent payments and on the blockchain a unique address is seen for each tx.

However the exchange can easily correlate that these 5 tx, spread over 5 months, all used the same address and thus went to the same recipient.

So privacy is increased on the blockchain, but not between sender and receiver.

The exchange is now able to sell this correlated data to blockchain analytics companies, etc, who can perform further correlations between exchanges, perhaps with KYC data, and now Alice's privacy is completely gone.

By contrast, if Alice provides a unique address to the exchange each time, there is no way to correlate those, and no privacy problem.