[devonkim]

I'm mostly convinced that most companies interested in Big Data stuff are not as interested in the scale of the problem but that they want to create "data lakes" to unite thousands of different forms of data that exist in their organization under a federated, centralized database of some sort. But most of us experienced in either enterprise companies or machine learning is that data quality is the primary problem that almost nobody actually can solve without brute force human eyeballs, which simply won't scale with the amount of data pouring in. So now there's interest in machine learning primarily to try to do that instead of people.

[edw]

At the risk of sounding cynical, there are also companies out there that want to _appear_ to be interested in all the things you just mentioned, so they'll hire a few people to do their [wave hands] data science, machine learning "thing" and those few people then go down the rabbit hole, untethered from reality. The C-suite people will then have their message they can deliver externally—and internally—about their commitment to [wave hands again] all that stuff. The tragedy when this happens is that sometimes that small team of people is not aware what their function at the company is—being the human props for the sales and marketing teams. The alternative, I suppose, is far more depressing—that the "data scientists" are fully aware of their role in the grand scheme of things.

[emmanuel_1234]

Woah, that was exactly my previous position.

Upside: I had one of the most paying position among the technical people. I also got to play with expensive stuff.

Downside: it was soul crushing, I was delivering no value whatsoever and had a really hard time looking at my colleagues in the eye, as they were making a third of my salary (at best).

I got out, joined a new company with that in mind and now have a very exciting job. They do have a dedicated R&D, Data Science team which is a shit show: absolutely brilliant people completely wasted as they can't build anything for lack of programming/technology experience, in an environment where their theoretical skills are mostly useless. I'm genuinely sad for them.

[edit: one of the company still has a Hadoop/Spark cluster for their whooping 500Mb of data]

[FractalNerve]

You really just described me. I've to do such a show-case project to complete my MSc Thesis for a minimum pay. And make all the proofs, so that the C-Suite guys can use during their presentations to sell what I made for a lots of money. (Even unaware if they'll charge their customers in the hundred thausands or millions range).

However, I'm really glad to find out about SnappyData.io, that's gonna save me a lot of time waiting. It would truly be my perfect dream, if they allowed running any programming language inside an environment like Jupyter.org or BeakerNotebook.com, but with Pandoc.org Markdown. So that I can essentially work fulltime programming, while I can also document it and also be able to export my documentation to a good looking latex thesis.

[sudsmenon]

SnappyData has a Zeppellin interpreter and the code is open source. So if adding a interpreter for jupyter is something that can be easily added, I am sure someone from the community would find it a interesting project to undertake. Agree that it would be useful

[wakkaflokka]

This is so true. For what it's worth, there are probably a lot of people who would give anything to be a human prop earning a salary significantly above $100k (even far outside of SF). For someone who's actually interested in data science, this would be miserable.

[dandermotj]

This is excruciatingly accurate.

[jy2947]

+1