The arduino generates a random passphrase, say 256 bits of random characters. You encrypt the private keys using this passphrase, and split the passphrase into 2 (or more) parts. You put one part on each usb key, and you put the encrypted file wherever you want. You might as well just put it on those usb drives, because if you lose one of them you're through anyway. When you need the private keys, you put both usb drives into the device, it concatenates the random bits into the passphrase and decrypts the private keys.

You don't need anything pre-programmed, it can all be done with random streams.