[stevesycombacct]

A significant chunk of my work involves no use of algorithms, statistics, or math. For the most part, my days are filled with one thing: cleaning up data.

If more firms took an educated, standardized approach to their data, I would do my job significantly faster, and the company would have access to their data-related products in far, far less time. Firms I have worked for, overall, refuse to do that.

Instead, such notions are treated as toxic- and I am too, by the associative property. They think I'm making extra work ("You want us to stop copying Powerpoints into Excels and put data into a whole new Excel that stops me from pasting in whatever I want? But I already have it in my own Excel!"), or they think I'm automating someone's job away to be replaced by a robot. This makes getting support from both employees and leaders near impossible.

It could save firms millions a year in reduced labor costs alone to not have entire teams of people sitting in basements fixing problems with data. However, data scientists are sold on what math they can do and what algorithms they can write, not on what processes they can improve.

It's, frankly, hell.

[karishmakunder]

Absolutely agree. It reminds me of an article that I read, had the statistics of Data scientists spend 80% of their time cleaning data rather than creating insights.

Curious to know, did you in any ways accomplish to standardise the way data was collected/ stored?

[stevesycombacct]

It's more like 90%.

On existing processes, standardizing collection has been nearly impossible unless I promised that it would be easier, it will take less time, and I'll do all the setup. It always takes leadership backing, and if I don't get it from one leader, I go over their head to the next.

On new processes, if I jump forward and volunteer, I find I'm given leeway to do it my way- that is, a standardized way.

If this sounds ruthless, it is.

[karishmakunder]

It is, what it is :)