Hacker News new | ask | show | jobs
Ask HN: Why is it so difficult to get great insights from data?
2 points by cneumann81 3817 days ago
2 comments

1] You are not measuring the "important" phenomena

2] Measurement error obscures the signal

3] Missing data

4] Data coming from disparate parts of a system may not have been collected at the same unit of analysis

5] Domain experts may have really good mental models of how various factors relate to each other so when you do analysis of data it mostly confirms what the experts already know and is therefore "not very insightful"

6] Too many variables, not enough cases

7] Data sparsity

8] Necessity to combine the skills of a researcher with those of a domain expert (both of those elements can have at least moderately steep learning curves)

9] Time pressures - great insights that arrive too late for action are useless

10] Combinations of any or all of the above

...it's not?

Do you have an example?

Well, in business, it seems that most deciders have difficulties to find the real important insights in their data. Dashboard are nice, but they often don't reveal the really relevant stuff. They show you only what you know, but no interesting data patterns. I wonder why this is? Is it a tooling issue, a skill problem, ...?