Quote from: 100x on July 16, 2013, 11:08:38 PM

So I am teaching myself how to use Postgresql (among other things), and I thought "what better way could there be than finally getting around to messing with the mtgox historical trade data?" (I had been wanting to analyze this data for sometime). After searching around for a while, I came upon this thread.

Thank you so much for this awesome tool! You have saved me much time/effort

It was very easy to start a dump, and the green bar is moving to the right currently so it looks like it is working! Just curious, the easiest way to fill in the missing data (as the Big Query db is not yet being updated regularly) would be to create a small script that uses the mtgox API to fetch trades with ID's after the ID of the last trade in the dump, yes?

Glad you've found it useful

Yes, that's right. Here's an example of one script that's already been created - http://cahier2.ww7.be/bitcoinmirror/phantomcircuit/. This is a different format though, and doesn't store all the data (I did however port this to an export format). You'll need to determine which field in the API maps to which field in the BQ database, and fill in some that isn't included in the API (Bid_User_Rest_App__ for example).

There are some discrepancies with the API data - the missing fields, data being split into individual currencies, currencies occasionally being silently added, gaps in the API data, etc. Whilst this shouldn't be too much of a problem for your actual data processing, it could cause problems if you ever want to update from BQ again. I suggest that you instead insert the data from the API into a new table (say, dump_live). You can then create a view from a union between dump and dump_live, delete the obsolete index on dump, and index the view (Money_Trade__ ASC,[Primary] DESC,Currency__ ASC). This will allow you to still access all the data fairly fast, but without corrupting the BQ dump. If you don't mind an extra 200mb, you can leave the index on dump. My tool will recreate this index if it doesn't exist, so it might be easier and more efficient to just leave it.

I've actually given thought to putting this in the official tool, hence the insights I had above (I'm not sure when I'll have enough time to do that though, it would mean changing quite a lot of the tool), so if you want any help with your script let me know, especially as I've also written documentation on the MtGox API v2, so I can help you map the API data to the BQ schema and get around the data jumps Tongue