Try http://api.bitcoincharts.com/v1/csv/ for bulk second-by-second data (no raw ticks, but currently this is the most raw data I've been able to find). They also have a more detailed API here which may be useful.

Difficult to tell, it seems good, you'll need to judge that for yourself though I think. I have only looked at the bitstamp data, I can't comment on the other exchanges, but for bitstamp the one oddity I found is that the trade times are given with second resolution, and so it's difficult to tell the order when multiple trades happened in the same time period. It's likely that they are given in the correct order, but I'm taking a VWAP of each second to avoid any potential problems with that.

Ahh, thank you for posting a working version. Yeah, I used Tkinter but it doesn't work well with multithreading, hence the numerous errors we've all encountered with this. It doesn't seem to work with Mavericks either. Unfortunately, I don't really have time to maintain this anymore, but hopefully it should continue to work with the dump I posted for now.

Thanks again for the new update !

That's fair enough. You should probably be careful about using strategies developed for MtGox with other exchanges. Simple strategies are probably fine, but you might find subtle differences. And these will probably be amplified for more complex strategies. You should be able to compensate for this with some minor tweaks though, so using MtGox data initially should still be fine.

Yeah, Kraken is quite new. Sometimes their 24h XBTEUR volume peaks above 1000 though. I think there have been a few introductory problems as they've grown in popularity, but I think they're getting through this now.

Yes, they are unix timestamps and that was their first trade (or, at least, the first one recorded by bitcoincharts).

Haha, hindsight is a wonderful thing

Hmm, well the last timestamp from the 2nd dump is consistent with the time I made it, so it looks like recent data is at least consistent with UTC. I suppose the only advice I could give is to do a dry run of any algorithm you develop first, just to see whether it makes the right choices in a period of, say, a week, before risking any real assets.

Personally I've been looking at other exchanges more recently, so you might want to consider other exchanges too, especially as you're worried about whether you can trust Mark. I'm quite interested in Kraken as it looks like it has a lot of potential, but I'm primarily using Bitstamp at the moment. There's some published CSVs for many exchanges here http://api.bitcoincharts.com/v1/csv/, although obviously then you have to trust bitcoincharts to have collected the data properly (I haven't analysed their data yet, but I assume it's fairly reliable; from a quick look the bitstamp one definitely seems to include the major price spike at roughly the right time).

Good night

Have you found an inconsistency?

I'm pretty sure they should be consistent with UTC, I take the timestamps directly from that returned by each one of the APIs, and I don't think there are any problems. If you think there are though, tell me which ones and maybe I can have a quick look. If you're just speculating though, then you should be fine. UTC is pretty easy to get right, his servers should just automatically sync with a reliable timeserver in order to get reliable UTC.

You can get a trading account here https://cryptotrader.org/plans.
Free accounts can develop and backtest algorithms, but to actually trade you need to subscribe.

A few other people seem to be having this problem (see the last couple pages of this thread). I'm not entirely sure why it's happening, but I think Google have changed their protocols subtly and it's broken the tool. As a result, I've created some full dumps, including data all the way up to mid-December 2013 here.

Thanks, Merry Christmas to you too

Hi BNO,

Haha, no don't worry about wasting my time, that was really confusing! That makes a lot of sense now, although now I'm not sure how Excel managed to distinguish between the columns considering they use a comma for the column separator... RFC should really standardise the csv number format as this wasn't very obvious. Anyway, I'm glad you've fixed the problem now, and hopefully this will be useful for other people

I just found this resource and I thought it would be useful to some people -- http://api.bitcoincharts.com/v1/csv/

I can't believe I didn't find it before, anyway, it has regularly updated (15mins?) CSV files with {unix timestamp (1s resolution), price (float), volume (float)} fields for many different exchanges and currencies.

Hi BNO, Excel gets stranger and stranger! 

I just did a test CSV export myself and the CSV is definitely fine, I'm not sure how or why excel is mangling the data like this. Dividing one column by 1e8 should not affect any other column, especially as inconsistently as the price column seems to be being affected :S What is the exact procedure by which you imported the data into excel?

Hi BNO,

I don't think so -- if you calculated the date+time as just the concatenation of the two, it should be fine as I created the columns directly from the original unix timestamp (in dump #1) using a decent conversion function. How did you compute the column? I guess considering the weirdness from excel before, it is possible the excel function you used might not work well - hopefully it will be consistent though.

Yeah, anytime an exchange stops or starts, even for a few hours, the data either side will probably be a bit skewed -- both for emotional reasons (like being scared), but also mainly just as the price readjusts through arbitrage to the market price elsewhere (or at least, this is what I would expect).

Cya

Hmm, some really strange stuff happening... Firstly, showing up as 1.07 GB - I just realised that the tool will create a duplicate index, I should have thought about that. It's not really a problem except for taking a few minutes the first time you load it up and increasing the file size more than necessary, otherwise though it shouldn't affect your usage.

Yeah sorry about that - the format I was asked to support only went down to second resolution, so the microsecond resolution isn't present. In fact, microsecond resolution isn't even available for the first 218868 ticks. Unfortunately there's so many different possible formats I could export to, so I picked a few and stuck with them (although obviously the dump contains all the raw data unfiltered). If there's one you really want then I could release a new version of the tool with that supported, but if you want to manipulate the data into more formats or more than you can do in Excel consider playing around with Python and seeing what you can do with it

Excel is being very weird - if you notice, it's taking the minute and seconds and converting them into a new millisecond value for some very strange reason, such as 17:48:56 -> 17:48:56:4856. There's no imperative for it to do this.

Yes, the first data source is the Google BigQuery database, the second is the MtGox HTTP API, and the third is the MtGox Socket API -- basically the socket API is used to just collect the last few trades in real time. If you're going to cut them out, then you can cut out the last few minutes, half an hour to be safe, of the data, and just the one day on May 23rd (although really there shouldn't be any discrepancy, the data should be exact).

Yes, the large jump is because MtGox changed recording format -- Money_Trade__, otherwise known as the TID/Trade ID, used to be a (mostly) sequential integer, then it became a microsecond timestamp afterwards (coinciding with that closure, I believe). It doesn't make a difference to the data though, all ticks should still be present, it's just a curiosity. Of course, this discontinuity will probably have some effect on the prices around that time, so you might want to exclude a few days ± around that point for that reason as it might mess up your backtesting.

The data should be an exact representation of what happened in real time, or at least a microsecond-resolution approximation. By that I mean that it it is possible multiple trades were executed in the same tick, but for all intents and purposes this shouldn't affect your use of the data for strategy development.

There are a couple oddities with the Money_Trade__ values, but I don't think this will be particularly relevant to you. Otherwise, I think the data is relatively accurate.

It is also possible that there are some oddities around May 23rd 2013 and around today December 17th, purely because I collected the data from 3 sources, and those were the boundaries - I'm fairly sure that there shouldn't be a problem, but if you wan't to be really safe then you can avoid those two days.

Lastly, the Primary column -- some trades are duplicated into other orderbooks, and their duplicates will be marked with their Primary field as false. It should be easy to exclude this data from your exports though (and it should be excluded by default). Alternatively, you might want to include it as I think these non-primary trades can influence other currencies, but I'm not too sure about this -- if you're not sure, just exclude it.

Hi, yep, you should be able to just load it in and export

That's good (although I was sure that the dump I uploaded would be incompatible...) Anyway, in case it still is, I've just finished another dump, up to date for 2013-12-17 15:48:26.308886, which should be fully compatible (the difference with the first one is that the date column is now in ISO format rather than unix timestamp, and the primary column is [true, false] instead of [1, 0]).

Here are the two dumps:
1) https://docs.google.com/file/d/0B3hexlKVFpMpYmdoUUhXckRrT2s (2013-12-12 22:13:40.483107; 665.1 MB)
2) https://docs.google.com/file/d/0B3hexlKVFpMpdVhxd1NvQ0Fqanc (2013-12-17 15:48:26.308886; 899.2 MB) <--- fully compatible with my tool for export (don't try to update it though)

The size discrepancy is because the ISO date format (2) is stored as text while the unix timestamp (1) can be stored more compactly as an integer.

I haven't looked at his source code yet, but looking at the contents of history.dat it looks to be in a binary format, which means you'll need a script to convert it. Perhaps poikkeus could add conversion to CSV to his tool? I'm sure if not someone else could quickly hack up a conversion script, assuming the format is simple enough (it looks like it's a java object dump).

Hi BNO,

The 2 tables relate to a discussion I had with MagicalTux (CEO of MtGox). My tool only really works if the bigquery data is presorted, but that can't be guaranteed if data is repeatedly being inserted into the table due to the way BQ works. I asked MagicalTux if he could implement a system of two tables, where one is updated, and the other is a sorted version of the updated table, hence the two tables. Since no updates have occurred since May though, it doesn't make a difference currently. Long story short, there is no difference between the tables .

If you download the data using my tool, then there are export options which can convert into a few different formats, CSV included (you can also choose which data to use and even do candles). Obviously though, the tool can only get data up to May (and has been a bit buggy lately). As far as I know, there aren't any other tools that can give you bulk csv data, although you might want to look at http://wizb.it which may at some point provide a working data service. I believe there are some charting websites out there though which can give you csv data, though with much lower resolution.

Please note that my tool isn't compatible with the full dump I just posted, and so this cannot be used to generate CSV. I might reformat it into a compatible version though seeing as there's quite a lot of demand for this data.

Sorry that your options seem a little sparse currently, if I had more time I'd love to develop a proper tool or set up a web service to make it easy to get data for any exchange, I know there's demand for it. I think I might make this a side project of mine, though it'll probably be quite slow.

Haha, I sense a petition coming Good to see you too, IMO Kraken has the potential to be the best exchange we've ever had