Interesting...didn't read the whole thread, but here's my initial thought.

Ethically, I think it'd be nice if the Bitcoins were returned.

However, technically speaking, I'm not sure this is legally required.

The 'mistakenly' sent Bitcoins constitute one transaction.  Bitcoins are not yet a legal currency.  When the BTC was then converted to USD, this seems to constitute a 2nd and separate transaction.  Since there was obviously no mistake in the conversion of BTC to USD, there is nothing wrong with this transaction.  And, since the first transaction, the transfer of BTC, is a distinct and separate transaction not involving a legal form of currency, I'm not really sure if there can be any legal claim.  In other words, the first transaction consisted of a transfer of BTC in which the value of the BTC was NOT tied to USD.  However, in the 2nd transaction, the BTC was sold for and therefore valued in USD.

'Mistakenly' sent is also debatable.  If a bank mistakenly sends someone an extra $500, it means that someone accidentally typed in a wrong input into some computer -- a human error.  The BTC transfer however was described as being due to a 'bug.'  The BTC was then sent according to the parameters of whatever code was currently operating.  The sender, being the exchange owner, should be aware of these parameters.

Here's an analogy (albeit a little confusing) to consider.  Suppose the owner got drunk and there was a 'bug' in his thinking and this led to him 'mistakenly' transferring 511 BTC, then claimed he wanted them back.  Had he not been drunk, the parameters guiding his thinking would not allow for such an occurrence.  But, being drunk, the parameters were different and this accounted for the BTC transfer to occur.  Similarly, it seems as if in the case of described in the OP, the ignorance of the code's parameters is akin to the owner getting drunk.   The code acted according to its current state and carried out the transaction.  This is NOT a matter of the sender accidentally inputting a wrong number, but rather the code carrying out the transaction in its current state.  The owner should have been responsible for knowing how this code would affect the transfer of BTC.  

In the case of a wrong-number input, the code remains the same, but the USER messes up in that he/she mistakenly input a wrong number.  However, in this case, if there is bug claimed to be in the code, then this bug in the code is a separate mistake.  The transaction was NOT a mistake.  That is, the mistake only relates to the code, and NOT the transfer since this transfer was simply a consequence of the original mistake.  Relating back to the analogy, the mistake was to get drunk.  Anything that occurs after this is a consequence of getting drunk.

Edit:  My apologies.  This is probably one of the most poorly-written posts I've ever made lol.  The last paragraph is the gist of what I'm getting at.