It doesn't have to be a problem per sé... As long as you realise what you're doing...

dumpprivkey "myaddress"
or
dumpwallet "filename"

executed from the console should export either the private key belonging to the funded address, or all private keys in your wallet...
These key(s) can be imported into many wallets (like jackg already said: electrum works really well)