It's not entirely accurate to say "the lower the fee the slower the confirmation time". This does generally hold true during periods when the mempool is more full, and there are several megabytes worth of transactions with low fees which are being overlooked by miners in favor of transaction with higher fees. In these case, then sure, the lower the fee you set, the longer it will take for your transaction to confirm.

However, in times when the mempool is pretty much empty (as has largely been the case for the past several weeks) then 1 sat/byte transactions are frequently making it in to the next block. In these cases, setting a higher fee won't get you confirmed any faster, and setting a lower fee won't take any longer. Block #588717 earlier today contained a transaction with a fee of 1000 sat/vByte, as well as several with a fee of 1 sat/vByte. The person who paid 1000 sat/vByte massively overpaid for literally zero increase in speed of confirmation.