The way I see it is that the wallet is just released and has little data to base retarget on and the span for target data as I posted above is huge
That still doesn't make any sense. When the difficulty only gets retargetted every 2016 blocks, there is no way this can go well. When the diff is low, it means the block time will be super short when hashes get added. When the diff is high, it'll take forever to get to the next block when the hashrate goes lower.
The whole point of difficulty is to keep the block time reasonably constant. When it only retargets every 2016 blocks, there's no way that can work.