just put a set price of 0,001 btc per buy/sell order in addition to the normal transaction fee, suddenly the spam got very expensive and wont really affect anyone else.
This, although to be specific, it should be *PER API CALL*, not per order actually completed. Or you could batch, charge say 0.01 per 100 calls to the API. Make it expensive to use a bot that functions by hammering the market so hard that nobody has any idea what's going on - such as filling the space between actual bids with 100 bids of 0.01 btc.
But then again, I think the bigger problem is the lag. Hard to know what to do when you're watching activity from 30 seconds ago, much less even 5, 10, or 60 minutes ago.