[faizshah]

There was a post earlier on using command line tools instead of Hadoop for quick data processing. This shows a non-trivial example of how you could implement a complex data pipeline and an overview of some of the commands you could learn if you’re interested.

[crmrc114]

Yeah, this is a pretty cool post I never considered doing something like this in the shell. It seems silly to me that I forget how powerful basic tooling in the native shell can be.. sometimes I have a jackhammer and I forget that a sledgehammer will do the job just fine.

[ahi]

I've been there. 100s of lines into a ruby program then "oh yeah, cut sort grep"

[e12e]

Just for anyone else, you could probably end up in a similar corner with perl - but in both cases it's likely a case of "holding it wrong" - ruby borrows heavily from perl which borrowed heavily from shell with sed, awk, grep, cut and friends.

So this kind of thing should be quite doable in a short ruby script - or a few short scripts - albeit written in "shell" style, with eg '-n or -p (wrap code in "while gets...end",-p with "puts _"), probably along with -a (automatically split lines).

Its in some senses an entirely different dialect of ruby, though.

Some examples here:

https://github.com/learnbyexample/Command-line-text-processi...