Hacker News new | ask | show | jobs
by kevingranade 3291 days ago
A detail you seem to be missing is that you are not limited to a shell script. The shell sets up the pipeline, but the members of the pipeline can be written in arbitrary languages and just have each stdout linked to the next processes stdin.

As a result you can process very large volumes of data and consume (not waste, consume) significant system resources to perform your processing.