I noticed that many people here are totally lost about what has be told and done, told but not done, done but not told and neither told nor done regarding the third party testing of ButterflyLab BitForce Single product.
With all the stuff scattered over two lengthy threads and weeks of endless debate, it is geting so confusing that even Inaba seems to be lost about what he has said and done.

This is why I have written for you the little pocket guide of "ButterflyLabs 3rd party testing for Dummies".
All the quotations are taken from the "is it a scam" and "is it for real" threads (courtesy of BFL and Inaba).
The emphasis in bold and comments are from me.

I have tried to remain objective, but I am of course totally biaised Tongue

, so do not base your opinion solely on my comments.
Please base your opinion on the facts, quotations, and linked messages.

This is a recommanded reading for :
- People waiting for their pre-order to be delivered
- Inaba, to help him sort out what he has said already
- Everyone else who wonders what the heck is going on with that ButterflyLabs testing nonsense.

----------------------

Quote from: Inaba on November 25, 2011, 08:05:35 PM

Yes, that's all I've seen and yes I've reviewed it. I've not had a lot of time to go into minute detail and step through the code as it would potentially operate, but I see nothing nefarious in it. I informed them that whatever they sent me is what I would be compiling and using on my own hardware that they have not had access to prior to the test... so what I've reviewed is what I'll be using to conduct the test(s).

Friday 25 November, no test have been done yet.
Inaba starts defining the experimental protocol and commits to use his own hardware and compile himself the binaries.

Commitments about the experimentation protocol:
- miner run on own computer
- self built software

Quote from: Inaba on November 26, 2011, 03:59:18 AM

I just got home after meeting with Sonny in person from BFL and their engineer via Skype. We had a fairly long conversation and I live over 30 miles away, so that's why it's taken me this long.
We did some simulated testing, but no live testing tonight. We elected to put that off until tomorrow or possibly Sunday to allow for proper testing

Inaba announces the tentative date for a first test : Saturday 26 Nov or Sunday 27 Nov.
The basic demo is done, but was more like simulation testing. Let's call that preliminary test "the basic demo".
The "proper testing" will be on Saturday 26 Nov or Sunday 27 Nov.

Status of the experiment:
- "basic demo" test done
- "proper testing" test pending => planned for 26 or 27 Nov

Quote from: Inaba on November 26, 2011, 11:38:59 PM

As I said yesterday, we'd either be doing the demo Saturday or Sunday. Right now, we have it planned for Sunday evening here in CST, probably around 18:00. I've been pretty swamped today, but tomorrow is mostly open.

The test (as called earlier "proper testing") will be on Sunday 27 Nov.

Status of the experiment:
- "basic demo" test done
- "proper testing" test pending => planned for 27 Nov

Quote from: Inaba on November 27, 2011, 12:37:19 AM

The demo plan is that I will:
A) connect it to a non-routable development side of the pool, so that the box is unable to communicate with the internet. I will then let it submit shares to the pool and I will have one of the getwork servers in debug mode and I will see what is sent out and what's sent back. As I found no evidence of any wireless communications on the board, and since the computer it's connected to will not be on the internet, it won't have any way of falsifying the shares submitted.

Commitment to do the necessary to segregate the computer used for the test from the Internet.
Full awareness that any possibility of external communication would make the test invalid.

Commitments about the experimentation protocol:
- miner run on own computer
- self built software
- pool run on own server
- segregated network environment

Quote from: Inaba on November 28, 2011, 02:10:18 AM

There are some technical issues with the timing between the board and the software that they are working out, apparently. I understand this will probably bring all the conspiracy theorists to the fore, so have at it. For everyone else, I'm told that they will be ready in a couple days or so for a live demo.

Technical problem, actual (proper testing) test delayed.
Test rescheduled for in a couple days.
This (proper testing) test is called "live demo" here. This is an important term. Let's remember it.
So the "live demo" will take place in a couple days.

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing") test pending => posponed to "in a couple days or so"

Quote from: Inaba on November 29, 2011, 04:36:22 PM

they agreed to do the demo tomorrow afternoon.

The "live demo" will take place on 30 Nov afternoon.

Quote from: BFL on November 30, 2011, 03:31:05 AM

As Inaba mentioned, we'll be bringing a live running unit to his data center tomorrow for a quick look and performance demonstration. Since we're not at our final figures, he's agreed to simply confirm or deny if the unit is generally within the performance class expected. I think rather than fuzzy numbers, everyone just wants to know if this unicorn is real. Fair enough... we expect to clarify that once and for all tomorrow afternoon.. We'll follow up with a more formal demo once we're finished.

This is a BFL anouncement.
The test in question (so the "live demo") will take place 30 Nov afternoon CST.
This test involved running a "live unit" (in case it wasn't clear enough this is a live demo) in Inaba's data center.
According to BFL's own words, the test aims to no less than proving that the product is real.

Quote from: Inaba on November 30, 2011, 07:18:02 PM

We plan on meeting at 16:00

The test in question (the "live demo") will take place 30 Nov afternoon at 16:00 CST.
No correction to what BFL said, so it is implicitely assumed that we are talking about the "live demo" that will prove that the product is real.

Quote from: Inaba on December 01, 2011, 01:19:28 PM

At any rate, we did a small demo of the hardware last night
[...]
In any case, the first test ran at a hashrate quite a bit lower than projected, but returned the expected nonce
[...]
But in either case, I feel that the test we conducted showed a POC that adequately demonstrates that at least the hardware does what it's designed to do

The test in question (the "live demo") took place on 30 Nov afternoon CST.
Positive feed back : the hardware does what it was designed to do.
At this point, no mention if the experimental protocol that was agreed on as been followed.

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing") test done

Quote from: BFL on December 01, 2011, 01:57:29 PM

The purpose of the pre-release demo was simply to answer the question of 'real' vs 'fake'.I think most reasonable people would agree that has been resolved.

Inaba confirms after the fact that the purpose of the earlier "live demo", also called "pre-release demo", was to confirm if the product was real of fake.
Strong suggestion that the test was successfull in showing that the product is in fact real.
At this point, no mention if the experimental protocol that was agreed on as been followed, and therefore whether the test was reliable.

Commitments about the experimentation protocol:
- miner run on own computerassumed
- self built software assumed
- pool run on own server assumed
- segregated network environment assumed

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test done

Quote from: Inaba on December 02, 2011, 12:48:37 PM

I have reported exactly what I've seen, without bias and within the ground rules that were set forth PRIOR to the test.

Inaba confirms after the fact that the test (the "live demo") followed the experimental protocol that was agreed.
He also mentions that he reported exactly, and without bias.
This effectively clears any doubt about the reliability of the test, the applicablity of the agreed commitment, and therefore the correctness of the conclusions given earlier.

Commitments about the experimentation protocol:
- miner run on own computerapplicable
- self built software applicable
- pool run on own serverapplicable
- segregated network environmentapplicable

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test done

Quote from: bulanula on December 09, 2011, 01:19:29 AM

Strange as it may look it does appear like the screenshot BFL Labs posted was running on a Lenovo W520 laptop ( which Inaba also seems to own in the other pics earlier on ). Maybe they ran the demo on Inaba's laptop and BFL posted the screenshot here or they oddly have the exact same laptop model or they are the same person after all Tongue

One week later, the original "scam thread" has been closed and a new "is it for real" thread has been created.
On 9 Dec, Bulanula uncovers that the laptop used for the "basic demo" (which photo have been published) is the same as the one used by BFL for their screen shots.

Quote from: Inaba on December 09, 2011, 02:30:51 AM

That is BFL's laptop, not mine. The one I plan on using for the test is an HP that BFL has not seen/had access to yet.

Inaba confirms that the laptop that was used for the "basic demo" and the "live demo" is BFL's own laptop.
His own laptop was actually never seen or accessed to by BFL.

Earlier commitments about the experimentation protocol:
- miner run on own computer~~applicable~~ not applicable
- self built software applicable
- pool run on own serverapplicable
- segregated network environmentapplicable

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test done

[NR]
It is doubtful that the drivers / miner built by Inaba for his own computer / distribution would work straight away on BFL's own laptop under Ubuntu.
It is also rather unlikely that Inaba thought of transferring the drivers he built to some portable storage, but didn't think of bringing his own laptop as agreed.
Even assuming that Inaba really gave the binaries he built himself to BFL, the laptop being BFL's, the path could have been set so as to run their own version of the binary.
Under these conditions, the condition "self built software" cannot be ascertained.

Earlier commitments about the experimentation protocol:
- miner run on own computer~~applicable~~ not applicable
- self built software~~applicable~~ doubtful
- pool run on own serverapplicable
- segregated network environmentapplicable

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test done

Quote from: Inaba on December 09, 2011, 01:45:38 PM

Quote from: makomk on December 09, 2011, 10:49:36 AM

Quote from: Inaba on December 09, 2011, 02:30:51 AM

That is BFL's laptop, not mine. The one I plan on using for the test is an HP that BFL has not seen/had access to yet.

So let me get this straight... you wrote all that stuff about compiling the code from source and about how the board and test machine didn't have any kind of internet connection, and then you neglected to mention that you'd used a laptop set up and provided by BFL which could've had a hidden wireless connection to anywhere. Did you even make sure that the binaries you were running actually corresponded to the source code you received, or did you use binaries that BFL provided too?

Neglected to mention it for what? We haven't done the test yet... How can I not mention something that hasn't even happened?

Makomk question regarding the objectivity and reliability of the live test.
Inaba clarifies the situation as he understands / remember it : the live test did not take place.
[NR] Note for people who are not familiar with block chain reorganizations : the live test that took place on 30 Nov at 16:00 CST, did in fact not take place (i.e. the live test has been orphaned).

Earlier commitments about the experimentation protocol:
- miner run on own computer~~applicable~~ not applicable
- self built software~~applicable~~ doubtful
- pool run on own serverapplicable
- segregated network environmentapplicable

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test ~~done~~ not done

Quote from: Inaba on December 09, 2011, 01:45:38 PM

So my plan was published after the first meeting and we haven't done a second, live test yet. So I'm a little confused as to what you are taking issue with?

Later in the same post, after referring to quotations of the "basic demo" test and highlighting that this was not a live test, Inaba states that there hasn't been a second live test (which also means that there was a first live test).
[NR] Note for people who, like Inaba, would be confused at this point : the live test that took place on 30 Nov at 16:00 CST, and that subsequently did not take place, actually took place but it was the first (this is important).

Earlier commitments about the experimentation protocol:
- miner run on own computer~~applicable~~ not applicable
- self built software~~applicable~~ doubtful
- pool run on own serverapplicable
- segregated network environment applicable

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test ~~done~~ ~~not done~~ kind of done
- "second live demo" test pending

Quote from: Inaba on December 10, 2011, 12:34:37 AM

I know what wireless is available in the building we were in and I know the response times on those routers - they are way too congested to be of any value for that sort of thing. For 3G, I agree, a few hundred ms, so you're looking at .5 seconds on a good day, probably longer on a bad one, so lets go with .5 round trip. That leaves 3.5 seconds to hash ~4.2 billion nonces, which is (off the top of my head) about 1.3 (?) GH/s? That means either a custom build FPGA rig hidden somewhere (expensive) or a multi GPU rig hidden somewhere with custom mining software able to split the data into partitioned nonce ranges (less expensive, more complex) - and all this would have to be done in LESS than 3.5 seconds, meaning they'd need I'd say at least double the hashrate to overcome the latency issues, so 2.6 GH/s.

Inaba recognizes that after all, it is possible that BFL's laptop was able to communicate outside (albeit with a high latency) using 3G or WiFi routers that he knows of in the data center.
Under these conditions, the "segregated network environment" doesn't apply.
The fact he knows of the WiFi router in the building suggests that he could scan the WiFi, which suggests in turn that the data center building isn't isolated from WiFi signal.

Earlier commitments about the experimentation protocol:
- miner run on own computer~~applicable~~ not applicable
- self built software~~applicable~~ doubtful
- pool run on own serverapplicable
- segregated network environment~~applicable~~ not applicable

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test ~~done~~ ~~not done~~ kind of done
- "second live demo" test pending

Quote from: Inaba on December 10, 2011, 12:34:37 AM

Additionally they were ready and willing to go to the datacenter which is essentially one giant faraday cage and would have prevented any sort of reliable (if any) data connection over wireless (either Wifi or 3G).

Later in the same post, in fact the datacenter is a giant faraday cage that would have prevented WiFi and 3G to get through.
[NR] Note : the faraday cage there prevented BFL to connect from inside the building to the routers that Inaba detected in spite of the faraday cage by scanning from inside the building.

Earlier commitments about the experimentation protocol:
- miner run on own computer~~applicable~~ not applicable
- self built software~~applicable~~ doubtful
- pool run on own serverapplicable
- segregated network environment~~applicable~~ ~~not applicable~~ kind of applicable

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test ~~done~~ ~~not done~~ kind of done
- "second live demo" test pending

----------------------

Conclusion :

Meeeehh Huh