I had this problem earlier selling 5BTC.

Actually I found the problem.
If I sell for 0.00000001 less then the amount of BTC in my account it works.

If I add a sell order for all of my BTC it duplicates it with one saying "active" and the other saying "insufficient funds", but neither actually works.
This is probably some rounding error at MtGox.