Hacker News new | ask | show | jobs
by _bxg1 2648 days ago
What about a "reduce" technique? Average the numbers in equal-sized chunks, then average those averages. You could even chunk the chunk averages and repeat the process as many levels down as you want to, and chunks could be as small as 2 each.

I guess this still assumes that the largest number in the original list is less than or equal to the maximum floating point value, but otherwise you stay roughly in the same space of precision as the original data.

4 comments

I believe you have pairwise / cascade summation in mind:

https://en.wikipedia.org/wiki/Pairwise_summation

That produces an unavoidable special case when the number of elements in the list is a prime number.
Just pad the end of the list with 0s, but keep the original size of the list.
Unfortunately, by averaging the averages you skew the results. Average of averages does not produce the same result as averaging the whole list.
Average of equal size chunks' averages does. (mathematically)
Ah my bad. Thanks for that correction
It does if you weight the chunks correctly. Combine pairs of numbers and keep a weight of the remaining odd number. Either cut the weight in half of the odd number every time or use it in the average if an iteration has an even number of numbers.
Thanks for that, didn't know.
\left( \sum_{i=1}^m x_i/m + \sum_{i=m+1}^{2m} x_i/m \right) / 2 = \sum_{i=1}^{2m} x_i /(2m)
I think this works perfectly if your list has 2^n elements. Otherwise, you have to resort to multiplying by imprecise fractions.
You would only need one special case: if a list (top level or not) has 2n+1 numbers, weight the last one differently.