Hacker News new | ask | show | jobs
by thousandautumns 3081 days ago
Entirely depends. "Moving the numbers significantly" doesn't mean much if the numbers aren't moving in the correct direction. Errors and scarcity in data are real, significant problems. I'm a statistician working in data science, and I can't count the number of times people complain about "why can't you just work with what is available" while failing to understand that what is available is total garbage.

You can dress up bad data in any number of ways to get results that sound and look pretty. I see this all the time. Sometimes you get lucky and the model is ok regardless. Lots of times the model performance isn't great, and it is later assumed there are other outside issues to blame, or the project is redone for the umpteenth time. Ocassionally you will have colossal failures that do real damage.

Keep in mind that when a poorly designed machine fails and kills dozens, or the financial system of the world crumbles under the weight of terrible loans and convoluted financial instruments, or millions of people's personal info gets hacked due to terrible, antiquated security systems, and everyone starts asking "How could people be so stupid to let something like this happen?", the answer is almost always executives, management, or "humble engineers" sweeping the problems they don't like under the rug because they believe "it's 10x better to just work with whatever you have available and get shit done".