The caveat being that the amount that you're depositing has to be more than your block reward at least for the attack to the worthwhile.
Of course. And a lot of such centralized services require 3 or more confirmations before crediting your deposit or allowing you to request a withdrawal, which renders the attack useless.