Most VPS's are overcrowded with other customers anyway, so it'll have poor performance. You may have a fast connection, but must of the time is spent in verifying the blocks (CPU and HDD time), therefore it's normal.