could be yes.
But from a pool operator point of view it sill makes sense to run vardiff. Simply to lower or even out server / network load.
With my knowledge so far i don't see any security issue with vardiff on ethereum.  But the big pools out there like ethermine / nanopool and dwarfpool don't use vardiff. So i'm curious to know if there is a reason behind it.