Probably best to start with Damn Small Linux, and customise things as needed from there -- I'm assuming you'd want to update the whole CD every time you have a new Bitcoin-Qt version, otherwise you'd have to store it on the USB drive -- but ideally you would set the system up to not run any executables from anywhere except the cd. (Using LUKS with usb drives works well, so no issues with encrypting things there.) It's probably also worth having a wrapper program which can launch Bitcoin-Qt (or Armory, which I'd include if I was doing this myself, at least once the stability issues are solved) and deal with the mounting/unmounting of the external drive.