[borramakot]

The amount of effort and extra hardware required to maintain this reproducibility across different hardware (cpu generations, GPU hardware/drivers, etc) feels insane to me. For example, people seem extremely interested in performance, but insist on using the much slower modes in MKL to maintain the bottom few bits of their golden data.

Do you find this genuinely important? If not, have you had any success encouraging e.g. validating against golden data in a non-bitwise comparison?

[lokedhs]

I personally don't find it important. And I have seen that the demand of having numbers match exactly comes from a misunderstanding of how floating point calculations work.

Sometimes this is understood by the banks and the solution is to implement a tolerance in the comparisons. In other cases other solutions may have to be used, such as forcing certain reports to be computed on certain software/hardware.

[perl4ever]

>I personally don't find it important

You remind me of the alleged UK Post Office scandal:

https://www.bbc.com/news/business-56718036

[lokedhs]

That's not what I'm talking about. Floating point computations happen when doing risk calculations, for example computing risk numbers in a Black-Sholes calculation.

A difference in the 15'th decimal has no impact on the risk number.

[perl4ever]

I don't have a good counterexample off hand, but I've seen a web page literally display "NaN" where a number should be.

The truth is, I have faith in Murphy's law.