Since the chain trust is only based on the length of the chain and not cumulated work, wouldn't it be quite easy to perform a double spend with much less than 51% hashrate?

On the original chain, send e.g. 10M XVG to an exchange.

While you wait for confirmations, start mining an alternative chain where you step down the difficulty by fakeing the timestamps. This should still be possible even though the maximum drift is changed to 10 minutes, it will just be somewhat slower. In this chain you include a transaction which invalidates your original transaction. Even if you only have e.g. 5% hashpower, your chain should eventually get longer than the original chain as it has much lower difficulty.

After you have converted the XVG to BTC on the exchange, you submit your privately mined chain to the network, and as it is longer than the original chain, it will cause the nodes to reorganize to your chain. Am I missing anything or would this be completly possible?
This would also make it possible to perform the attack with extremly limited ressources as you don't really have to compete with the honest miners.