[patio11]

http://bitcoinstats.com/irc/bitcoin-dev/logs/2013/03/11 http://bitcoinstats.com/irc/bitcoin-dev/logs/2013/03/12

The incident begins at 22:11. If you don't trust my summary of it, and you consider yourself very, very well-briefed on what Bitcoin is doing under the hood and who the players are in the Bitcoin community, just read the next ~12 hours of logs. It's absolutely riveting.

My summary:

The Bitcoin protocol doesn't exist. The only protocol which matters is the actual behavior of the Bitcoin Core client -- the one originally coded by Satoshi, which forms a supermajority of the network. Bitcoin Core released version 0.8 on or about March 11, 2013. This differed from Bitcoin Core 0.7 in at least one respect, which was that 0.7 used Berkeley DB and 0.8 did not.

BDB has a configuration issue, specifying the maximum number of locks it can use at once. If you attempt to use more, it returns an error.

Here's one reason I say the Bitcoin Protocol doesn't exist: No sane person says "An important feature of the Bitcoin Protocol is that conforming clients MUST REJECT any Bitcoin transaction which would exhaust the default number of locks available to the Berkeley DB."

Someone submitted a Bitcoin transaction which did, in fact, exhaust the number of locks available to the Berkeley DB. It was conformant with all the rules that Bitcoiners believe transactions have to be conformant with except that bit about the lock limit in the Berkeley DB. This transaction was accepted by a miner running the 0.8 software.

Let's, for convenience, call the blockchain as it existed prior to that block being mined Blockchain B. The blockchain with that block in it is B'.

Bitcoin Core 0.7 rejected the authenticity of B'. Accordingly, when 0.8 nodes said "I have a new block to publish! It checks out and builds off of the-present-head-of-B'", 0.7 nodes said "I don't know what that nonsense you're spouting is, but it sure isn't Bitcoin." Bitcoin Core 0.7 nodes and miners continued talking amongst themselves and building B up.

Bitcoin Core 0.8 nodes accept the authenticity of B' and all blocks chained on top of it. If you presented them with a block chained off of the head of B, they would say "Oops, sorry, sucks to be you -- someone already has a longer chain. You've created a blockchain, but by the writ of Satoshi, we only do business with the longest compliant blockchain, which is B'."

This is called a network split. And it is, in laymen's terms, utterly cataclysmic.

Here's why: Suppose Mt. Gox runs on 0.7 and Coinbase runs on 0.8. I can create a transaction which spends some output(s) $COIN and have it accepted into a B' block. Perhaps that transaction deposits my $COIN into Coinbase. Since that transaction doesn't exist on B, I can then deposit the same $COIN into Mt. Gox. Both Mt. Gox and Coinbase believe themselves to be in possession of the same Bitcoin.

And both of them are right.

Which is why after that channel figures out what is happening that engineering gets real. After discussions between several core developers they ascertain that the Bitcoin mining cabal behind B' is small enough to get, well, both of them on Skype and convince them to throw away hours of history on B' (which is, again, built off the new-and-improved bug free version of Bitcoin) and instead start building off of B instead. Their history must die so that Bitcoin can live.

This plan is executed. It works.

Read the chat transcript if you don't understand this: Bitcoin was virtually unusable during the interim -- most of the merchants people cared about turned off transactions entirely because they were, sensibly, scared shitless. Several hours of transactional history got wiped out. One security researcher successfully executed a ~$10,000 double spend attack against a merchant -- he gave the money back afterwards.

Now, you tell me: how do you rate my credibility here versus your friend? My assessment of this event is "Bitcoin has an identifiable governance structure. You can fit them into an IRC channel +/- a few Skype sessions. They can independently decide to change the rules of the 'Bitcoin protocol' a) at will b) retroactively. I cannot reconcile this with claims that Bitcoin is 'does not require trust' or is 'decentralized.'"

[maaku]

> They can independently decide to change the rules of the 'Bitcoin protocol' a) at will b) retroactively. I cannot reconcile this with claims that Bitcoin is 'does not require trust' or is 'decentralized.'"

Sigh. Your post was a great up until the very end.

Bitcoin developers can't push any rules on anybody. They haven't the power.

A majority of bitcoin hashpower can enforce a strictly stronger set of validation rules, as indeed happened here. Is it a problem that a very small number of individuals represent policy for >50% of the bitcoin hash rate? Yes. Is this intrinsic to the nature of bitcoin? No. And it's something that people are working to fix.

Can a majority of hashpower arbitrarily rewrite history? Yes, but only with a very real opportunity cost to themselves. And that is the rules of bitcoin since the beginning -- although in reality people would probably choose to reject a long reorg. Some level of human intervention is a good thing.

[0xFEED]

> Someone submitted a Bitcoin transaction which did, in fact, exhaust the number of locks available to the Berkeley DB.

You really need to get up to speed with the event before commenting on it, there's a very good writeup in the form of BIP50.

https://github.com/bitcoin/bips/blob/master/bip-0050.mediawi...

The issue has nothing to do with a particular transaction or block, it was going to fail anyway and just happened to be triggered by a large block. The oversight meant that all 0.7 clients were inconsistent with one another depending on the history of the node and how long it had been operating.

> No sane person says "An important feature of the Bitcoin Protocol is that conforming clients MUST REJECT any Bitcoin transaction which would exhaust the default number of locks available to the Berkeley DB."

Nobody did say that, the failure was implicit not explicit.

> Several hours of transactional history got wiped out.

No it didn't, it was replaced with a slightly different history, or history remained the same, depending on which side and which client you happened to be running at the time.

[mpyne]

> No it didn't, it was replaced with a slightly different history, or history remained the same, depending on which side and which client you happened to be running at the time.

This is precisely what he just said. A valid blockchain being replaced by fiat (heh) is certainly a destruction of transaction history, whether that happened to everyone using Bitcoin or just a subset of people using Bitcoin.

[lechuga]

Would you have advised people to use the network during the split?

[jerguismi]

> I cannot reconcile this with claims that Bitcoin is 'does not require trust' or is 'decentralized.'"

Aren't these floating variables, not booleans? "Level of decentralization" vs. "how much does it require trust". I would argue that the level of decentralization and trustlestness is better on bitcoin than fiat payment systems.

[zaroth]

Very good summary! Just one clarification, I'm pretty sure it wasn't a specific transaction that exceeded the lock count, it was the block produced by a v0.8 miner, while being within the max block size, the number and/or types of transactions within that block caused the BDB maximum lock count to be exceeded.

[enjoy-your-stay]

Interesting read.

So what has happened subsequently ?

Was everybody encouraged now to upgrade to the 0.8 client across the board? Otherwise the same thing could occur again, deliberately or not, couldn't it?

[mpyne]

We're effectively always a bitcoin-core bug away from another instance of this. Any bug which could lead to a partition (i.e. where post-upgrade clients believe one chain is the main chain and pre-upgrade clients think a separate chain is the main chain) could lead to this happening again.

Should it happen again you could fix it by either forcing everyone to quickly upgrade (and losing the transactions that occurred on the old main chain post-fork) or by doing what happened here (holding back or downgrading the upgraded clients until the old main chain was the longest chain again, and losing the transactions that had occurred on the new chain post-fork).

What will be even more interesting is what happens if we actually ever get multiple compatible implementations of Bitcoin. As patio11 hinted at, if those implementations are not all "bug for bug" compatible then type of thing could recur over and over again.

So we're left with the choice of remaining with a single reference implementation to reduce (but not eliminate) the risk of introducing partitions due to slight incompatibilities between Bitcoin clients, or to encourage multiple implementations of Bitcoin to reduce the dependency of an entire "decentralized" economy on a centralized reference code base.

[moe]

Do we really need to repeat the exact same circle-bashing on every HN bitcoin post ever?

Isn't everyone sufficiently aware by now that there have been bad bugs and fraud?

Wouldn't it be more interesting to discuss what needs to be improved?

Or is your point that distributed currency can not possibly work? (that would also be a more interesting debate)

[drcross]

>Or is your point that distributed currency can not possibly work? (that would also be a more interesting debate)

There is no debate, distributed currency does work. I have been using it on and off for four years. Bitcoin works the same at 1 dollar as it does as 1000. Using the currency as a store of value, however, is another matter.

[oafitupa]

What if I told you I have been using it as a store of value all these years? Seems to work too.

[Andrew_Quentin]

So the governance structure you have identified was for two individuals in some position of "power" to voluntarily decide to take a certain action because they see it in their best interest overall?

They could have just as well, seeing as they had the longer chain, decided to continue working on that chain, but they voluntarily decided otherwise. In the process, validating the inbuilt incentives of bitcoin.

The decision therefore wasn't "independent", but reached by consensus and further supported by everyone else. Had there been some disagreement, for example if what the devs were proposing was not seen by the two "powerful individuals" as being in their best interest, then matters could have developed far differently.

So, what exactly requires trust in all of this?

[kang]

Lets say this indeed was cataclysmic.

But can you imagine a possibility of building a system where the db that each person holds is not berkeley db or whatever but just plaintext and then writing your own functions to traverse the same.

I'll admit I do not understand the bitcoin core at all because I never invested time yet to read it. But I completely understand the blockchain algorithm and can write a cryptocurrency without even caring about what bitcoin did.

All I am asking you is do you believe that such a system can be built? Or do you believe that the current developers are intelligent enough that if they couldn't do it you can't do it either?

[oafitupa]

See, it's not even clear where patio11 is going with his recurrent rants about Bitcoin. Looks a lot like irrational hate.

[kalleboo]

All software has bugs. Despite massive investment, space craft have bugs, passenger jets have bugs, fighter aircraft have bugs, nuclear power plant control systems have bugs.

If you're basing your system on "oh we'll just make the software bug-free!", you have already lost.

[shayanbahal]

This is one of the few problems that still could be counted as open problems in Bitcoin, called "Blockchain fork" or "Orphaned Blocks". there are a few implementations that could resolve this issue but none is perfect and none totally works. I liked the speech of Andreas M. Antonopoulos in Canadian Senate that asked to not to regulate bitcoin now, because there are things that quite don't work perfectly and now we are able to fix it with less loss than if it is adopted by many more (https://www.youtube.com/watch?v=xUNGFZDO8mM).

So it is a Protocol but still really young and immature in some senses, but it doesn't mean anything else, it is evolving almost everyday.

[DennisP]

Ethereum, a new cryptocurrency project, is attempting to do better at defining a protocol. They're launching with at least two official clients, developed by different teams in different languages. They have an extensive test suite based on the spec, and when they find different teams have interpreted the spec differently they revise it to clarify.

https://blog.ethereum.org/2015/04/02/implementing-vitaliks-v...

[aminok]

>My assessment of this event is "Bitcoin has an identifiable governance structure. You can fit them into an IRC channel +/- a few Skype sessions.

Right, but in an attack scenario, Bitcoin can continue and survive without that governing structure. It can fall back entirely on its trustless protocol.

Your analysis does not appreciate the significance of this.

[luma]

This was an attack scenario and the protocol failed in the face of it until the governing structure stepped in.

[aminok]

This was not an attack scenario. An attack scenario is one where a person or group of people is attacking the Bitcoin network. This was a bug, and one that would have been very difficult to discover without running a network on the scale and complexity of the real network, meaning one that would have been extremely difficult to discover ahead of time by an adversary and exploited by her.

Insofar as this or future versions of Bitcoin might still have critical bugs that require social consensus to fix, then yes, Bitcoin is not guaranteed to be totally independent of the type of ad hoc centralized governing structure that emerges in IRC channels and Skype chats. But a bug is a failure of implementation, not the concept. Bitcoin, the concept, is sound, and is a major breakthrough that is not appreciated by OP's analysis.

[necessity]

Isn't that why people wait N confirmations before accepting a transaction? The recommended MINIMUM is one hour.

[rmc]

That is to address a different problem, and would not have worked in this case.

[siffinleex]

Well, it can address this problem as well but you would have needed to wait until the problem was fixed, which is a time you wouldn't reasonably wait unless the transaction was extremely large value.

Maybe a take away point from all this is that even if you are doing just a 20K transaction you should still wait 24+ hours to make sure a bug of this nature isn't occurring.

[rmc]

That presumes that the problem will be noticed and fixed within 24 hours, which is likely, but still depends on the developers fixing things. Which isn't so decentralised.