Quote from: verymuchso on December 13, 2017, 04:21:36 PM

WE DID IT, 2000 transactions PER SECOND OVER THE NETWORK!!
_{proof will follow, test it for yourself from your browser}

Hi fellow HEATers, please allow me to share a short development status update.

Some interesting and major milestones have been achieved over the past week. To list a few:

Blockchain-Event-Cache

Big steps have been made towards finalizing the Blockchain-Event-Cache, the event cache gave us some real hard challenges to solve. An example being how to store to disk the tip or last N events of a never ending stream of events, without running out of disk space! And do it in such a way that no matter if the HEAT server crashes or shuts down, that at all times the persisted state of the blockchain matches exactly with all persisted events.

The significance of the event cache really can't be overstated. It's workings and design are unique and as far as I know exclusive to HEAT. The event cache should be considered as an ordered list of events where most events map to a whole or part of a blockchain transaction. The event cache has a cursor which points to the current latest event and this cursor can move up and down the list of events.

Observers of the events (microservices listening to transactions for instance) will be notified when an event first arrives. HEAT being a decentralized consensus mechanism must account for the inevitable slowness of reaching full network wide consensus simply because nodes need time to talk to each other, the network speed determines this. The event cache is your friend that watches blocks come and go, switches to better forks and reorganizations of transactions, all the while giving you a normalized view of the transaction stream as if you observed centralized service.

When building business applications on top of HEAT using the blockchain as a place to transport and store your data having this 'normal' view of the transaction stream is of vital importance for any kind of application you build.

High-speed-RPC-channel

What was probably the coolest part of the week was to fire up the new high speed RPC channel we added to HEAT core. From unit tests we could already see that HEAT has been very fast when it comes to processing transactions, we've seen numbers of well over 2000 transactions a second when feeding transactions directly from memory into HEAT.

What hasn't been achieved yet was to process such high numbers over a network, until now that is!!. Two major obstacles of feeding lots of transactions to a HEAT server over the network are:

Everything is encoded as JSON, parsing lots of JSON costs memory and downloading large blocks of JSON makes things slow
Every transaction broadcasted did so over its own unique and new HTTP connection

The data encoding issue has been something we've looked at for some time now. What we wanted is basically write out raw binary data which is as dense and optimized as possible, yet still be able to handle this new format somewhat easily and preferably portable so it could be used for instance with the https://www.npmjs.com/package/heat-sdk. We would be using such for:

Microservices user defined messaging protocols
P2P message encoding
To encode high speed RPC messages

We decided to go with https://avro.apache.org/ which is the data serialization stack from http://hadoop.apache.org/ [HADOOP=The Apache Hadoop® project develops open-source software for reliable, scalable, distributed computing.]. A major factor to choose AVRO is its support for JavaScript and the fact it supports versioning of schemas. When developing microservices those aspects are extremely important and allow service operators to easily declare their custom binary protocols which can be shared and used in a decentralized way where service users only ever need the public schema (hosted perhaps on the blockchain - since even the schemas can be efficiently encoded to binary data) in order to communicate with a microservice.

To solve the slowness of sending each transaction over a new connection we've added a new channel to the existing websocket handlers. You can now open a websocket connection and leave it open, not needing to connect again you can now send binary transactions over the already open connection. For now this channel supports only one operation, to broadcast a single transaction, but we expect this RPC mechanism to grow to support more methods in the future.

And the cool thing is you can use this super fast transport from everywhere where there is Websocket support, this includes everywhere the HEAT-SDK can be used (browser, mobile, nodeJS).

Playing around, we've already seen huge speeds. On my PC alone we could already reach over 2000 transactions a second for the duration of 500,000 transactions. This was done with 4 websocket connections and after pre-generating and signing the 500,000 transactions. That particular node was forging but with no connected peers, for that to be possible we need to update the P2P layer still.

Currently we are working on completing the RPC and AVRO binary data support to HEAT-SDK so that all this can be used simply by getting HEAT-SDK.
When that is ready we'll be hosting the custom HEAT server somewhere and we'll release some in-browser samples that people could use to fire transactions at the HEAT instance. Would be fun to see what numbers it can do, I know you guys like breaking things. Grin

Thanks for sticking around! And let's make HEAT great again.

When you say "over the network", what do you mean by that? Is it over the internet or over a local network or even with just 2 instances on 1 computer?