Guys, thanks for your responses.
It means that my suggestion wouldn't fly, especially due to following scenario:
1) Attacker gets the private key, and starts a transaction;
2) I am notified and cancel the attaker's transaction;
3) Then I try to start my transaction to move the bitcoins to safer place;
4) However the attacker cancels my transaction as well;
5) Then all repeats...
I realized the delaying of the transaction doesn't really help. We need something else. Something which is more like a "vault" then just a "wallet".
I think I have another idea, but I'll start another thread on that.
Thanks!