If you're interested in having such a device designed, I'd be interested in contracting.

Primary plan of attack would be a PIC32 doing USB HID+MSD with an EEPROM. Nothing revolutionary.

The PIC32 toolchain isn't entirely open, but the compiler is open and all required tools are cross platform. The PicKit3 programmer is available for $45. Clones exist if you want to save money. (IMHO they're not worth the trouble, but it's your prerogative.)

The parts list would look something like this:
uC: http://www.digikey.com/product-detail/en/PIC32MX230F064B-I%2FML/PIC32MX230F064B-I%2FML-ND/3046648
Display: http://www.digikey.com/product-detail/en/NHD-12232KZ-NSW-BBW-P/NHD-12232KZ-NSW-BBW-P-ND/1701265
Serial EEPROM (multiple for redundancy?)
Buttons: http://www.digikey.com/product-detail/en/MJTP1138BTR/679-2415-1-ND/2344168 (A shorter button would be used if you want keycaps)
USB connector: http://www.digikey.com/product-detail/en/48037-0001/WM17117-ND/857603

Estimated BOM price is about $25 with PCBs.

Potential issues:
Security. Are you looking for encryption and authentication? How about hardware security?

Size. The display is going to make a big difference here. What content do you want to display?

Case. I can 3d print cases, but they're neither as pretty nor as durable as injection molded cases.

Write only. A tamper proof write only setting would take some extra effort. How permanent does this need to be?

What's your desired capacity?

Software. USB software is generally painful to develop.

E: \/ John, that post made me search around for an upvote button. Looks like a great product!