I have an idea that I want to see implement in Monero because it has the same problem but on a smaller percentage scale (10-15% since May 2018)

Try a weighted average instead of a normal average
in the example of 720 blocks,
take the first 50 blocks and give them a weight of 50%
take block 51-100 and give them a weight of 25%
take block 101-360 and give them a weight of 12.5%
take block 361-720 and give them a weight of 12.5%

720 is not necessary though, cut it down to a smaller sample and keep the same ratio, i.e. you could do 360 blocks instead of 720, or even 180 blocks