Oh man, thanks for that. If you hadn't said that, I would have given up on it a while ago.
Now that from what I can tell, I have to restart the miner up to 5-6 times until the diff seems high right away from the start and then it will typically accept about ~12 shares, reject ~10 shares and start behaving normally after that. YMMV
Or - you can do the coin a favour and mine with another pool and let vardiff "just work" for you.