As far as I read here, it syncs fast when you start from scratch and download a fresh (mini-)blockchain. If you want to sync from a previous state, the wallet may download all blocks between your last "local block" and the current block height, which can take a while if you connected to the network long ago. However, it only should store the last mini-blockchain permanently.

Some pages ago I meant I had read a fix or workaround for this "problem" (which isn't really a problem because you help validate while you sync). A rough workaround is obviously to delete the old blockchain data and sync from scratch again.