If I block the internet from bitcoin-qt, changing the settings works, but there's still ~5secs of unresponsiveness. So then I checked out what it wrote in debug.log after it got up again:
2014-02-09 19:21:57 No valid UPnP IGDs found
2014-02-09 19:21:57 upnp thread exit
2014-02-09 19:21:57 upnp thread start
So I disabled upnp and it now works like normal.
Fixed I guess...