The system was connected to from one of our other boxes which was accessed through a virtual console. The wallet box had all public ports blocked but was able to be connected to from a few of the other boxes.
Thanks for confirming. This is why I prefer no incoming connections allowed on the secure box. If you must have occasional ssh, you can have it enabled on boot and then login to disable it. That way you can reboot first if you must login.
How do you solve getting to the secluded bitcoind to command it to sent bitcoins out?
You don't. You have a process on the 'wallet server' that checks an external source and base it off of that.
In a 100% ideal security scenario you don't have ANY incoming connections. That isn't 100% possible because bitcoind needs to get blocks so that has to be at least port 8333 open. Also other ports were
probably open as well for convenience, just firewalled to allow certain machines access.
EDIT: Or what the other 100 people above me said (teach me to reply without reading the whole thread).