Quote from: fbueller on March 22, 2014, 01:42:31 PM

I want to propose something that would hopefully open up some doors to researchers, analysts, who may not be able to code at this time, but want to process the blockchain for stats, trends, and other info unavailable through stuff like blockchain.info. I'm not looking for another blockexplorer, I want something that people can query for big data.

Does anyone want to help out with this? I plan to import the blockchain into a MySQL database, storing basic block information (output from getblock), and the same for transactions. Also build up a list of addresses related to a transaction.

I expect this would be an unwieldy amount of data, so input from people would be key to this. Also help in supporting this (a dedicated machine/vps that could manage this would be required.. if people donated to keep it floating that would be great.

I have made a start on coding some things would make importing the blockchain a lot quicker - A function to decode a raw transaction instead of asking bitcoind has drastically sped up my parsing times. I need to add some more rules so it recognizes unusual transactions.. it isn't incredibly flexible yet but it recognizes pay-to-pubkey-hash and pay-to-scripthash outputs. I'm tempted to write something that would just parse the blkxxxxx.dat files..

I think having a public querying engine to ask things like 'how many P2SH addresses are being used today, or how has this changed over time?'. Or a visual thing showing how transaction consensus has changed over time. Or other creative questions that I wouldn't think to ask. I know there are a few blockchain forensics/analytics companies popping up.. a community lead project focused on gleaning insight from the block chain would be great though.

Thoughts?

There are quite few open source blockchain parser available on github: blockparser, bitcoin-abe, block-browser. These can dump data into sql. MySQL/PostgreSQL will be fine to handle it but about 60G+ of storage and half to several days to process.
Some of the stats about script types used are available at http://webbtc.com/stats.
Theres is also a up to date sql dump available http://dumps.webbtc.com/bitcoin/