Why not add a USB controller (controller IC plus socket available for not more than $4 and maybe 1 square inch of board space)?  USB host mode gives you a lot of flexibility to add any device you can find or write a driver for in the future.

You can get cheap LCD displays on eBay for around $4 for a 16x2 model.  Of course you'll need some headers to mount it, but that shouldn't add more than another $2 or $3 to the price tag.

Same thing goes for an SD card socket (or microSD if you want to reduce the board space required).  Around $3 is all it costs for the socket and you don't need any other support chips that you don't already have for other circuitry.

What other peripherals?  Oh, an ethernet port.  Microchip makes some suitable controller chips for about $4 (SPI interface).  Add a socket for under $2.

The SRAM SDRAM hurts a bit more.  For 32MB (32-bit wide), we're looking at maybe $15.

So altogether that's around $35 extra in parts to the price tag to be able to run embedded Linux and thus support full standalone operation.

It occurs to me that you could make two models.  One master model which is capable of standalone operation.  One slave model which can plug into a master model and extend it's mining capacity (and save you up to $35 in parts at the same time).  Since there's no need for a high speed bus between master and slave (they would operate as separate miners), you'd have a lot of flexibility in how you implemented that as well.