| > Please don't do that: they won't necessarily be the outliers in the dataset and your model will converge to the wrong thing. They are the outliers. These experiments are typically measuring something that is effectively constant with almost zero noise, rather than some complex physical phenomena. If you don't get an almost perfect fit, there is something going on that invalidates what you are doing (e.g. cache effects, clock effects, etc.). In fact, if there are any outliers, I would not trust the benchmark at all. So removing them seems like trying to fix a bad benchmark with statistics. > Statistics is hard. Like, really really hard. Stuff goes wrong all the time. Please leave it to the experts. That is unnecessary gatekeeping. Benchmarking in CS is hard not because the maths/stats that are needed are hard, but because setting up the right experiment is hard and most people don't know all the pitfalls. Therefore, if anything, you should leave benchmarks to CS/SE experts, rather than a statistician! |