I think fees are only cheaper when you do a SegWit to SegWit transaction.
That's incorrect. Using Segwit to send gives a discount on the weight (virtual size) of the input scripts of the transaction so it will be smaller regardless of whether the output addresses are legacy or Segwit. There is also a very small additional saving if using a bech32 address as it takes a few bites less but that is a minor saving.