[more-coffee]

About 5 years ago I worked at a small, business oriented telco. The biggest ETL was processing external CDRs, applying call tariffs, and creating bills and reports. A previous developer had been adamant about using a Hadoop cluster to process this, storing all of it in Cassandra NoSQL.

The idea was interesting, but it didn't quite work out. At some point he left the company and we had to do something about the pipeline, as it was crashing most of the time. We did some calculations, and figured that with the right approach, a good old MySQL+PHP solution would do the trick. And it did.

Having switched jobs myself in the meantime, I'm happy knowing the system is still running, and finding people to maintain it is relatively easy.

[threeseed]

Maybe that developer knew something unique about that use case i.e. data needs were expected to grow or they had plans to use it for Data Science (very common in telcos). And you just weren't aware of it.

It's always easy to pass judgement at technology choices but in my experience they are often made with the best intentions based on requirements that not everyone is aware of.

[em500]

More likely he knew that the pay for a hadoop / bigdata specialist was a lot better than for a mysql specialist, especially 5 years ago.

[huffmsa]

And the guy's resume probably says

> Implemented big data / real-time Hadoop streaming ETL service processing billions of requests.

I've become very skeptical of anyone who puts a combination of buzzwords and pseudo-numbers in their resume.

[huffmsa]

Doesn't matter what he "might have known or intended to do", he didn't build a solution which met the specifications the business needed.

[threeseed]

You really didn't get my point did you.

We don't know about the specifications because we don't work there and since the OP said he wasn't there he might not know either. And the fact is that requirements regularly change over time.

The point is that it's really easy to judge when you aren't there and are privy to all the facts.

[huffmsa]

But we do know the OPs solution met the specifications, and is still in use, meaning that those specifications haven't changed too much.

And you're damn right I'm going to be judgemental of someone not only promised the moon, but abandoned the work when the rocket exploded on the pad and someone else had to clean up.

[more-coffee]

Yes, there are more use cases I didn't knew of at the time. For example, optimizing revenue by calculating the optimal routing strategy for outbound calls. But data analysis wasn't a prority. Nor was scaling storage or computations.

I know this is circumstantial, but on average there were 2 developers. The company needed something to just get the bills out. It also needed to be correct, stable, easily expandable for new datasets (requiring all kinds of conversions), testable from unit to e2e, integrate with other APIs to retrieve metadata, and .. I forgot a few, probably.