[sanjha7]

This describes my situation to the point. I have worked in big unicorns and have deployed many ml based models in production which had moved the numbers significantly while many a data scientists in our team just kept cribbing about errors in data or scarcity of it.

I have no DS background, am a humble engineer but believe it's 10x better to just work with whatever you have available and get sit done.

[thousandautumns]

Entirely depends. "Moving the numbers significantly" doesn't mean much if the numbers aren't moving in the correct direction. Errors and scarcity in data are real, significant problems. I'm a statistician working in data science, and I can't count the number of times people complain about "why can't you just work with what is available" while failing to understand that what is available is total garbage.

You can dress up bad data in any number of ways to get results that sound and look pretty. I see this all the time. Sometimes you get lucky and the model is ok regardless. Lots of times the model performance isn't great, and it is later assumed there are other outside issues to blame, or the project is redone for the umpteenth time. Ocassionally you will have colossal failures that do real damage.

Keep in mind that when a poorly designed machine fails and kills dozens, or the financial system of the world crumbles under the weight of terrible loans and convoluted financial instruments, or millions of people's personal info gets hacked due to terrible, antiquated security systems, and everyone starts asking "How could people be so stupid to let something like this happen?", the answer is almost always executives, management, or "humble engineers" sweeping the problems they don't like under the rug because they believe "it's 10x better to just work with whatever you have available and get shit done".

[pavelromashkin]

same here, I haven't explored all nooks and crannies of the "data science map", but all the ml/dl I deployed were a success. I still feel like a major imposter though.