Hacker News new | ask | show | jobs
by throwaway_pdp09 2194 days ago
> People always looked down on those that used Hadoop or somesuch for <1GB of data, but while it wasn't needed from a technology perspective it gave a structure to the project.

What 'structure'? Why is it so important that it makes it worthwhile firing up a large, complex framework? I'm beyond baffled.
1 comments
cbcoutinho 2194 days ago
The same 'structure' that makes it easy to onboard new co-workers because they've seen the same project 'structure' before in the past. In that sense, the bottleneck in an organization is getting people productive as fast as possible, even that means using a cleaver instead of a scalpel.
link
throwaway_pdp09 2194 days ago
If all they can use is a massive cleaver (big data tools), and have no experience with scalpels (small, sharp, cheap and fast data tools), IMO your company has a serious, fundamental and systemic problem (no, let's call it failure) towards employee experience, training and knowledge. Edit: and resource management.
link
_jal 2193 days ago
Seems to be a sort of inverse of the massive spreadsheets that run supply chains on accretions of spaghetti-macros.

But, a tree chipper can serve as a paper shredder, and I imagine a lot of shops in certain markets saw it as a sort of prestige asset around 5-8 years back, when a bunch of companies started hiring data scientists for no apparent rational reason.

(Not bashing data scientists or data companies. Just remembering the fad that went around Bay Area companies a while ago.)

link