[jongjong]

The framing that there is some unknown particle affecting the results is probably an understatement. Probably there are a LOT of unknown 'forces' affecting the results. Look at the size of the machine, the aging components, the number of components involved, the amount of software required, surely all of that is affecting the results.

Are humans even capable of building perfectly reliable systems?

[exmadscientist]

You would probably be very surprised about just how much work is put in to characterizing these things. My graduate work was on another experiment (though our postdoc worked on one of the earlier g−2 experiments mentioned here!), but we literally had man-years of work sunk into measuring one thing about the experiment, one number, as accurately as possible and with valid error bars.

That was not the only thing about the apparatus, just one of the most important, but the point is: we cared, we cared a lot, and we cared a lot about a lot of things.

[jongjong]

Sure, but you can only solve a problem that you are aware of. My concern here is about the unknown unknowns. The task at hand is exceedingly complex and requires extreme precision, with many unknown variables... And the people solving the problems have conflicting interests as they are paid hourly.

Even if there are bonuses, all the bonuses would be given on the basis of solving known problems. There is literally zero monetary incentive to solve unknown problems; only owners care about that stuff. Employees don't care about that. They might point out some whilst not mentioning others.

I've been a software engineer for 15 years and my experience is that unknown unknowns in tech are like a secret piggy bank for engineers to tap into in times of need. They are rarely brought up; only in the event that an engineer feels the need to justify their value to the company/project. The accumulation of secrets gives engineers differentiable value within the project/company and makes them hard to replace... So they tend to keep at least a couple to themselves... They ration it; like retirement planning.

The individual incentive to conceal issues until a time of personal need (when layoffs are looming) outweighs the collective incentive to identify new issues to generate more billable hours of work for the team.

If you're the only person who knows the issue which prevents a multi-million dollar project from coming to fruition, that is valuable information and gives you a lot of leverage in terms of job security... Which is the biggest reward you can get in this line of work.

And it's not malicious; the people who know about those issues probably also solve more issues and contribute more solutions than anyone else. The concealment may be happening subconsciously; they won't admit to themselves that a specific kind of problem exists.

Like when I worked in crypto, all the engineers there believed that the project could scale and also be decentralized... When it couldn't, given the chosen approach. The belief is still there years later; though all the rational engineers left, they were promptly replaced by believers.

[panda-giddiness]

Precision results are usually blinded, so it's not as if scientists can just rejigger the calculation if it's giving an implausible result; once the result has been unblinded, the calculation is finished. The physicists working on these experiments are therefore extremely motivated to quantify all potentially non-negligible sources of systemic error beforehand.

That said, the issue in this case is almost certainly systemic, not new physics. But I don't think you should consider the misfortune of the data-driven calculation typical; rather, it's notable specifically because of the discrepancies. Most calculations do not have such large disagreements. The lattice results, in contrast to those from the data-driven approach, all line up very nicely.

[exmadscientist]

To be fair, sometimes the blinding is a piece of duct tape over the display on a piece of fancy OTS industrial equipment. Though we needed to poke at it occasionally, so we needed a duct tape flap we could lift up every now and then rather than just a bare piece. I forget if I made ours or if C made it, but I remember our little blinding flap fondly.

(This is more common than you'd think.)

[teamonkey]

Bluntly, I think this just highlights why software engineering is not a serious engineering discipline. There are some exceptions, like safety-critical systems, but on the whole, software is not designed and developed with a rigor comparable to physical sciences.

[jongjong]

Especially true now with AI coding. But I would push back on the idea that sciences or other engineering disciplines are not affected. Several of the people I met who were especially guilty of this in software sector had a PhD in mathematics and used to work in academic institutions.

I think intellectual integrity has been on the decline overall.

[exmadscientist]

> the people solving the problems have conflicting interests as they are paid hourly

LOL. No, no they are not. There's a lot of reasons I'm out of that game, and this is one of them. $1400 a month (2010 dollars), unlimited working hours, unlimited international travel. Sound fun? No?

> bonuses

I think I got a $5 Starbucks gift card once. I guess you could count the free leftover pizza as a "bonus"?

To address the heart of your argument, Science is not structured like software engineering, and if you're thinking it is, you're making a lot of mistakes. Scientists often do well as software engineers, should they change careers [waves hand!]; economics aside, software engineers very rarely make good scientists. The mentality is simply different.

As regards complexity management and dishonesty; yes, straight-up dishonesty does occur sometimes. We are all human, after all. But science is done in the physical world, and unlike software or mathematics, the physical world does not admit unlimited complexity. So things must be divided up in a manageable way right from the very beginning. On a big experiment, if your little "box" of responsibility does not work, you can and will be replaced. I have seen it done. Even to or by me. (If involved, I have historically been the one doing the replacing; but I've been on the other end, too.) And you will be cross-checked by independent people, if it is critical: I have seen this too. You had better get it right.

[jcranmer]

My gut reaction is that the probable cause of the discrepancy is ultimately "someone calculated something wrong." It's not like particle physics experiments actually generate something directly measurable like the muon's magnetic moment; you instead get an indirect result that you have to calculate to the value you're trying to measure. And half of the constants in that calculation are themselves from reported results in other experiments, which similarly returned indirect results that are dependent on correct results from other experiments... and it's not hard to imagine that the result of "value X is A" depends on an assumption that "value X is B" somewhere in the calculation tree.