The downside to keeping hodl stash/cold storage coins in legacy addresses is that you'll probably have higher transaction fees when you finally move them.
The upside is that you'd avoid any potential segwit based attacks on bitcoin, like if miners decide to take all the segwit coins.

So, I guess you have to balance the estimated future fees for non-segwit transactions with the estimated potential for miner theft.