[Snackchez]

All of this is anecdotal:

These past couple of days, whenever a coin soars a ridiculous amount, on any given exchange, that exchange seems to mysteriously go down. For example: Binance and Bitfinex were down for "maintenance" (or just plain down), while IOTA was experiencing a surge in price (from ~1.10 to 6$). Similarly this happens with Bitcoin today (it seems Quadriga had some issues and Coinbase; not sure about others).

Something is up, and it's weird.

My conspiracy theory: big money institutions are testing pump and dumps for when short contracts come into play. On the one hand, the cryptocurrency world wants to remain as unregulated as possible; on the other, a lot of individuals are going to get reamed by big whales.

[crazypyro]

Isn't it just the simple explanation that big swings in the market bring insane traffic spikes to market websites?

Add to the fact that people are also DDOS websites (doesn't take a "big money institution" to do that) and it seems the explanation is clear...

[superfrank]

Yea. I work as a software engineer at a company that sees huge swings in traffic on our product (by nature of the product). It's not uncommon for us to see 20x swings in traffic day-to-day. It's taken us a long time to be able to handle that and even then shit still goes wrong.

I'm much more inclined to believe the downtime is caused by scaling issues rather than some big conspiracy,

EDIT: Just to give some insight, one of the problems we run into is that usually only one issue is exposed during an outage. For example, lets say we have 10 functions (or services or whatever) in a row that all call each other. All the odd numbered ones have issues where they fail at scale. During the first downtime, function one will fail and start to throw errors, but this means the full traffic load isn't reaching the rest of the functions. By the time the problem is solved, traffic has usually died down, so function 1 is fixed, but 3,5,7,9 still have issues that haven't been exposed.

Additionally, at a quickly growing company like Coinbase, the code isn't going to stay the same for very long, so even once those functions are fixed, new issues are introduced.

It's easy to say they should be load testing at 10x their expected load to prevent these type of issues, but it's really hard to replicate production traffic. Users do weird and unexpected shit that can cause problems.

[dmix]

There's an article on the frontpage of HN from NYTimes that mentions Coinbase has struggled with the massive spike in traffic recently.

It's funny how quickly people jump to overly-complex or borderline conspiratorial explanations when the obvious is staring us in the face...

Coinbase might be a large well funded organization but it's still run by developers with giant backlogs of tickets and conflicting priorities to build new stuff vs maintain old stuff.

[omg_ketchup]

> but it's really hard to replicate production traffic

This times 1,000.

[odonnellryan]

Honestly, I don't think these exchanges can handle a lot of traffic. By a lot, I mean a few hundred trades a second. It is odd to me.

[Demiurge]

I have entertained using cryptocurrency for game betting at some point, while running some typical web stack. When I started looking into security and scaling, I realized there was a expensive and complicated overhead. When security is of very high concern, traditional performance seems to really suffer.

[mv4]

They are very well funded, and they can afford scalability. It's not that hard, given the number of transactions even at peak so far.

So yeah, I find these weird circuit breaker patterns suspicious. The worst part is "partial" degradation, when you can't execute trades, but someone else can. It's very easy to hide all sorts of irregularities behind these partial outages.

[dwaltrip]

> It's not that hard

I'm sure they would pay you a lot of money to solve their scaling issues, if you were so inclined.

[mv4]

and I would do it.

[SilasX]

Not really. While most exchanges went down during the Nov 29th rally, most others didn't during today's, nor did they during the four June rallies where Coinbase also went down.

[spelunker]

I think you can take off your tinfoil hat. Lots of price movement motivates more traffic to the exchanges, resulting in DDoS.

As the bitcoin market grows GDAX, Bitfinex, etc are probably going to have to borrow some ideas/talent from NASDAQ & other big electronic exchanges to stabilize things.

[Snackchez]

Tinfoil hat off!

Shouldn't an exchange be able to handle volume? Seems like a basic requirement.

Regardless, I still believe that with the introduction of shorts, were going to see bigger swings.

[joshuahedlund]

> Shouldn't an exchange be able to handle volume? Seems like a basic requirement.

Exchanges have incentive to handle as much volume as possible to make as much via fees as possible, but no site can handle infinite volume. The coinbase app has been climbing the iTunes charts into the top 20 the last couple days. Just before the site went down the price was running up by hundreds of dollars per minute, attracting tons of trader traffic trying to profit from it. Even if they have basic scaling measures in place for the basic infrastructure, they most likely experienced unprecedented exponentially increasing volume that was inevitably bound to break something

[user5994461]

Judging by the GDAX runbook, they barely have a couple of orders per second. They can't have performance problem at this scale.

I've seen exchanges running 10 or 100 times that amount with no problem, ran by only a couple of graduates out of school.

Either way, doesn't matter. Exchanges have no performance problem, it's a big queue, you just queue requests.

[eropple]

In my capacity as an infrastructure/scaling consultant I've spoken to multiple exchanges. Without disclosing anything I'm prohibited from, my strong expectation is that almost every exchange can't handle traffic spikes.

They're not all shitty me-too Rails apps built by junior engineers who figured "how hard can it be?". But most of the ones I've seen are.

[cortesoft]

Clearly it isn't a basic requirement because they exist without being able to handle volume.

[Sohcahtoa82]

> Shouldn't an exchange be able to handle volume? Seems like a basic requirement.

Exchanges are just websites like any other. They're build to handle a certain amount of traffic based on the average amount plus a little extra to handle spikes.

If they only get (And I'm pulling numbers out of my ass here just to make a point) an average of 10 trades per second with occasional spikes to 100 trades per second, how much infrastructure should they build? What capacity should they build to? 1,000 trades per second? 10,000?

There's a cost analysis involved. I think it's a bit unrealistic for them to have enough infrastructure to handle spikes an order of magnitude higher than their typical peak loads.

[spelunker]

I agree, they _should_ be able to, but lots of times performance is a 2nd priority behind getting shit out the door - for better or or for worse.

I'm sure as they grow they'll harden their systems and/or do a total rewrite. In fact I wouldn't be surprised if they're doing it already since the big boys are starting to show interest with futures.

[wereHamster]

> handle volume

But what levels of volume? 100 quadrillion trades per nanosecond? Sure they could scale their infrastructure to those levels, but why? You overprovision, get estimate your worst-case scenario and work towards that. But in the end, if there is a huge rush, like a 10000% increase or more, even the largest exchanges would break down.

[mortehu]

US stock exchanges don't run websites on which you can trade. However, during the flash crash in 2015 I was unable to place trades on either of the two brokerage sites I used at the time (USAA and Vanguard), probably due to other people rushing in to trade like myself.

[castis]

I imagine they've spent a lot of time already beefing up their infrastructure, you can anticipate high loads but until you know what that looks/feels like, you cant compensate for it. They'll get there eventually. It'll just take some time.

[tomp]

No, most stock exchanges aren't exposed to the public like bitcoin exchanges are.

[Florin_Andrei]

No need for conspiracy theories.

These services are like banks, but the traffic they handle is more like a news site, due to the extreme volatility of cryptocurrencies. Super-security combined with extremely large traffic fluctuations.

Whoever is on PagerDuty at Coinbase these days - I feel for ya.

[kough]

Coinbase runs on Ruby and their devops aren't good enough to make up for it.

[JumpCrisscross]

> big money institutions are testing pump and dumps for when short contracts come into play

Big-money institutions won't do this. Random individuals taking advantage of the fact that lots of value rides on relatively little volume? Much more likely. Combined with the fact that the exchanges, intentionally or unintentionally, lock liquidity when volatility spikes and it's 1907 all over again.

[w-ll]

24 Hour Volume is north of 17 Billion.

https://coinmarketcap.com/currencies/bitcoin/

[JumpCrisscross]

Given the incentives involved (and history of exchanges, everywhere, before regulation) a good fraction of that is wash trading. (A better measure for the risk at hand is the effect of small out-of-market bids and offers on the overall price, i.e. how much it costs to move the price $1.)

[1] https://www.investopedia.com/terms/w/washtrading.asp

Disclaimer: This is not securities advice. Do not buy or sell anything based on this Internet comment.

[wolco]

On the 10th bitcoin will be listed on the chicaco exchange. This is the reason for the price swing lately.

[misja111]

Can you explain? You could also argue that when futures are listed on the 10th, there will finally be an easy way to go short on Bitcoin. For instance when you want to hedge a long position, or simply because you want to speculate on Bitcoin going down.

So the listing of Bitcoin futures on the Chicago exchange might not be a good thing for the Bitcoin price at all.

[kwelstr]

The 10th is a Sunday

[kgwgk]

The exchange opens Sunday evening.

[Afforess]

No, No, No! Conspiracy theories attract attention because they make the world seem more legible, when it's not. In this case, exchanges are not being attacked by shadowy corporations for short contracts.

Attention from price surges increases visitor traffic, and exchanges are understaffed or unprepared for the 10-100x increase in traffic volume. There's no need for complicated stories, evil corporations, or any of that nonsense. Stick with straightforward cause and effect.

[jrs95]

No, No, No! Don't pay attention to that thing! Look over there!

Seriously though, it's worth considering that this kind of thing could be possible even if we don't know whether or not it has anything to do with these specific outages.

[Afforess]

No, it's not. There are an infinite number of conspiracy theories. There is no value in considering them.

[marzell]

You are coming off as very emotional/desperate in your attempt to convince people to act cool and logical. It's not very persuasive, and if I could downvote you I would.