Hacker News new | ask | show | jobs
by dekhn 1692 days ago
The tech lead of the Google MapReduce team (which no longer exists) just received their award for turning down mapreduce. IIRC it was officially done 5 years ago. However I believe the code to delete MR was never checked in and I'm not sure if there are still users.

MapReduce was used at Google for highly inappropriate things. For example, the machine learning system I worked on, Sibyl https://www.datanami.com/2014/07/17/inside-sibyl-googles-mas... was implemented using mapreduce but there was no real technical justification for that- it's just that there was no other system that could scale to the volumes required or handle the constant failures endemic to GOogle's internal systems. It ended up requiring all sorts of heroic work to make MR scale, for example map-side combiners (which "reduced" items with common keys in the map output before it gets flushed to the shuffle files). All of this got replaced with TensorFlow and only the good bits of Sibyl were extracted to TFX.
3 comments
sokoloff 1692 days ago
It seems to me “there is no other technical system in the company capable to perform the task” is a valid technical justification.
link
jpgvm 1692 days ago
i.e this is a crappy tool but it's best tool we have come up with for this problem thus far.

There are lots of such tools which are used begrudgingly by people that have an intuition for the fact it can be done better but not the concrete idea and/or time to implement it.

link
dekhn 1692 days ago
it wasn't a crappy tool (mapreduce was amazing) but it definitely was an impedence mismatch for this particular job. Later, we tried to get SIbyl to move to the underlying compute engine that Flume was built on top of it but it turned out to be more profitable to just let it die slowly.
link
mrep 1692 days ago
MapReduce was deprecated because flume [0] the successor is better but it does practically the same thing and flume is used massively. I believe dataflow is the public google cloud version.

[0]: https://research.google/pubs/pub35650/

link
oblio 1692 days ago
> handle the constant failures endemic to Google's internal systems

This sounds bad.

link
dekhn 1692 days ago
yeah it's crazy, I don't understand why Google's data center machines shit themselves so often. Probably cheap ram.
link