[dekhn]

I've seen this happen in a wide range of production environments (both industrial and computing). Not this effect specifically, but "odd emergent behavior that occurs only at scale that is non-obvious and state-dependent". For example I work at a company that grows a lot of cells is massive reactors, and some folks who run the largest reactors commented that they saw slow changes in overall production that were not explainable by any observed variable (we speculated that slow genetic drift occurred in populations, but it may also have been seasonal, or due to unobserved variables). And when I worked at Google, there were definitely cluster-wide things that you'd only notice if you were very knowledgeable and attuned to their ongoing processes.

My guess is that this happens in nearly all large-scale production systems but goes mostly unobserved.

[gopher_space]

> My guess is that this happens in nearly all large-scale production systems but goes mostly unobserved.

Not unobserved. Unremarked maybe? It's expected behavior that leads us into personification of systems e.g. calling ships 'she' or talking about temperament between similar machines on a line.

[empathy_m]

I think the disappearing polymorph stories are also pretty spooky. These have real-life impacts, like with ritonavir.

[MarkusQ]

Every time I look into those I come away thinking that Occam's Razor would suggest a different explanation: the original characterization was, knowingly or not, incorrect. Patents so frequently fail to contain sufficient information to allow a practitioner skilled to in the appropriate arts to reproduce the claims that it seems more plausible that the disappearing polymorph stories should be reclassified as "someone was caught fibbing" stories. In the replication crisis, we don't assume that the problem is that something about the world has changed, we assume that the original was flawed, and we should do the same here.

It would be much more convincing if there were more cases that weren't economically significant. A strange property of chemistry that only comes up when money and lawyers are involved seems inherently suspicious.

[empathy_m]

I skimmed the literature on this and the ritonavir story seems legit.

There really is a peer-reviewed paper saying that there are five crystalline forms of the stuff. ("Elucidation of crystal form diversity of the HIV protease inhibitor ritonavir by high-throughput crystallization", Applied Physical Sciences, Feb 2003).

It really does seem that in 1998 the more stable Form II suddenly started coming out of the factory, with lower solubility and such bad oral bioavailability that the oral capsules were withdrawn from the market until Abbott figured out a new way to make the drug. (I think they were already moving from a capsule to a gelcap and the gelcap didn't have the same issue? Just reading … this is not such a good source perhaps but lovely bare HTML: https://www.natap.org/1998/norvirupdate.html )

[toast0]

From experience with large scale clusters, yeah. Weird stuff happens. But it's very hard to setup a test cluster that is actually representative, and you can only do so much on a live cluster. Occasionally, I have been able to find explanations for some of the weird behavior, but usually it's like here's a bug in Linux packet forwarding that was fixed in Linus's tree 15 years ago, but apparently has never been deployed to some router, so it's just going to keep aggregating input packets because large receive offload, and then drop them with needs frag because the aggregated packet is too big to forward. sigh (that's not exactly a cluster scale issue, but it's the most relatable example of an investigation that comes to mind)

You're pretty unlikely to get academic papers when the required setup involves having 100M+ clients geographically dispersed. And it's going to be very hard for peers to reproduce your findings.

[TeMPOraL]

In case of this "invisible electrostatic wall", there were likely significant amount of people in that company who were at least somewhat into Star Trek[0], so I'd expect more than mere "meh, this happens" from people who had just seem to have accidentally invented a force field. It's not merely a weird emergent behavior, it's a behavior closely resembling a sci-fi technology, and therefore likely to have similar applications - so quite obviously a potential money and fame printer.

-- [0] - Which was well-known around the time of that event, and at its peak of popularity when the report in the article was filed!

[dekhn]

When you work in production and have quotas to meet, you often ignore interesting side-effects. When I worked at google I worked at global cluster scale and frequently saw any number of events that in themselves would have been graduate-student-for-two-years projects that I had to force myself to ignore so I could get my main work (large scale protein design using 1-3 million cores in prod) to finish.

As a side note, always test any global-scale torrent system for package distribution carefully, as sometimes the code can have "accidentally n**2" network usage that only shows up when you have a worldwide grid of clusters.

[swayvil]

It reminds me of that experiment where they had an audience of 1000 focus their attention upon a chair on a stage.

[krackers]

What happened?

[swayvil]

The chair got chairier.

I don't recall what that means.

[egypturnash]

Do you recall anything that would make it possible to find any descriptions of this experiment? When I try to search for "a thousand people focus attention on a chair" I just get stuff about meditation and an "ADHD chair". Which is apparently a thing.

[xobs]

It sounds vaguely like the Invisible Gorilla Experiment: https://en.m.wikipedia.org/wiki/The_Invisible_Gorilla

[swayvil]

I'll see what I can find