I like what you have on the site currently.

The forex method might be the standard for exchanges but these are used by traders all the time.

A standard rule for any user interface is "Don't make me think!"

Like most people on here, I'm an IT nerd so maths, rates and calculation make sense to me but I work with artists all day long and their math skills tend to be poor. My first thought would be how they would respond to the three options. When people change up money for holidays they tend to think of how many euros or dollars etc they're going to need but they also might want to think about it in terms of their home currency but I think they rarely think in terms of rates and the notion that they're either buying/seling euros etc doesn't even come in to it.

My preference would be to put in the number of GBP I want to sell and for the interface to calculate the resultant BTC at the current exchange rate and then to be able to adjust the rate and/or the BTC you're wlling to receive. IMO, That's a more natural way of thinking about it. I guess this is effectively all three methods at once - you can change any parameter and it will recalculate the other params based upon which one was changed. How that's stored in the DB is irrelevant to the user but I guess the forex format of storage is probably the best.

I guess the UI might look something like...

Qty1 * currency_seletor = rate * Qty 2 * currency_selector_2

If you update either quantity then the rate is recalculated. If you change the rate then Qty2 is recalculated. Put that with some nice Ajax currency graphics and you're on your way. It would also be good if it let you know if your order matches any in the orderbook.