[mumblemumble]

With all credit due to Google's excellent and under-appreciated paper Machine Learning: The High Interest Credit Card of Technical Debt [1], I submit that Big Data is the high interest home equity line of credit of business operations debt.

It's not that big data tools aren't useful. It's that, when you just start amassing huge piles of data without a clear up-front plan for how it will be used, and assume that a whole bunch of people who have never heard of sampling bias or multiple comparisons bias or Coase's Law [2] can figure out what to do with it later, you're setting yourself up for a Bad Time.

  1: https://research.google/pubs/pub43146/ 
  2: "If you torture the data long enough, it will confess."

[abirch]

I'd say that Big Data is the Collateralized Debt Obligations of business operations. It looks fabulous from afar but it can blow things up quickly if there's no understanding of the internals.

[systemvoltage]

Yet, we abide by data-oriented conclusions outside of software engineering all the time. From Academics papers to FDA to crime statistics.

[mumblemumble]

I won't say any of those are perfect. But there's at least a little more effort toward responsible data analysis in academia. The FDA brings an interesting example to mind. Take a look at how, on paper, drugs suddenly magically became less effective when the FDA started requiring clinical trial pre-registration in 2007.

It's also worth noting that, over the past few decades, most academic fields have been getting increasingly skeptical of the value of correlative research on pre-existing data sets. Even among people who have been extensively trained in how to do it properly. And yet, the vast majority of big data business plans I've seen in practice boil down to "collect a huge data set and then let people do correlative research on it."

[systemvoltage]

Agreed, I want more scrutiny than some entity flashing “Here is the data”. It can easily be exploited behind the veneer of data-based-credibility.

[htrp]

> I submit that Big Data is the high interest home equity line of credit of business operations debt.

I like this but it's kinda like the payday loan of business operations.