Quote from: northcape on December 26, 2013, 08:52:05 AM

Im currently running the localDB branch on the side and it seems to record the data just fine from what I can tell.

The db file produced seems to be similar to a CSV, but I noticed that it isnt ordered by the first fieldname ("s").

Right now my file has entries with "s" values from 388 to 482 in a random order, then a line that says

Code:

{"$$indexCreated":{"fieldName":"s","unique":true,"sparse":false}}

and then entries 483 to 522 neatly ordered by their incremental "s" value.

Could you maybe give me a very short explanation of what the field names stand for?
I just wanted to make sure that this is intended behaviour.

What data is it storing?

I had three options to store historical trade data in a way that would let me access the data again:

- Store all trades
- Store candles at the size that is in the current config
- Store 1 minute candles.

The first option creates the biggest databases. I need to store the data in a way that works cross platform and I want to limit the memory footprint. On markets which lots of volumes this can mean a couple of MB in a busy hour. This is not really an acceptable size for me.
The second option means that if you want to change from 60 min to 30 min candles you cannot use previous history.

So I went for the first option: I know exactly how much data per day is required and I am able to calculate every candle (> 1 minute) on the fly based on those.

How is it stored?

I didn't write all the code for storing the local database myself, instead I used neDB. Databases are hard to do correctly so I'm happy I don't have to reinvent the wheel here.

The first line tells the database that there is an index on the `s` field, which means we can query specific `s` recods faster. The other lines are part of the neDB filesystem (stored in JSON). Each record is a 1 minute candle with the following properties:

- s: start minute since UTC midnight (we store each day in a different database, check it in the filename)
- o: open price of this minute
- l: lowest price of this minute
- h: highest price of this minute
- c: close price of this minute
- v: total trade volume of this minute

Can I re use other data?

I am going to write tools that can convert this database into a popular CSV format that I see a lot in the forums, also the other way around (but if the CSV candles are > 1m this means you loose accuracy in the data).