If you don't move very big fund, 3 confirmations are enough and waiting times for 2 confirmations can be smaller than 20 minutes, like within 2 or 3 minutes. Sometimes two blocks are found quickly.
If you're moving more than a million dollars, I think you should wait as much as it is financially discouraged to have your transaction reversed. For instance, if you're moving $10M, then 6 confirmations aren't enough as 6 blocks are valuated at about 37.5 BTC ~= $1.1M. To be confident there is no such incentive, you should wait for x blocks, where x satisfies: x*block_reward > transaction_value.
It doesn't completely remove the incentive one might have to rob you that way; it depends on what you're exchanging.