I just realized though.. removing all that hashing from the pool wouldn't reduce the diff.rate (immediately) so your personal odds of mining a block don't actually change.. I think you'd have to effectively remove a shit-ton of hashing power for an extended period of time and get the diff.rate to go back down for this to be an effective attack vector..

Math isn't necessarily my strong suit. Someone smarter than me please comment.