[bmcahren]

I'm not sure I know the use case for all of this logical complexity. Is there a specific use-case you had in mind?

Split the array of elements to process. Set up a promise.all against a function that handles sync/async function calls against each chunk and accumulates an array of results. This isn't the end of the world for any of the practical use cases I can think of. The efficiency loss and memory consumption is miniscule even across hundreds of thousands of records. This is as concurrent as anything else - the event loop is still a mandatory part of the processing loop. You can even defer every 1000 records to the event loop. Javascript is syncronous so there is no concurrency without threading. I can't think of an IO or non-IO function that would create the issues described of not yielding. We used to have chunks of 1000000+ records that would freeze the event loop. We solved this with structured yields to the event loop to allow other processes to continue without dropping TCP packets etc.

If the results are objects, you're not even bloating memory too much. 800KB for 100,000 records seems fine for any practical use case.

I like the idea of an async generator to prevent over-pre-calculating the result set before it's needed but I couldn't justify pivoting to a set of new functions called pipe, map, etc that require a few too many logical leaps for your typical ECMAScript developers to understand.

What's the win here? It must be for some abuse of Javascript that should be ported to WASM. Or some other use-case that I've somehow not seen yet.

[tomeraberbach]

To clarify, the point of it isn't _just_ performance.

It's a combination for writing the iteration code in a functional style, which some people like/prefer, while retaining certain performance characteristics that you'd get from writing more imperative code.

For example, the "naive" approach to functional programming + concurrency results in some unintentional bottlenecks (see the concurrent iteration example on the home page).

[bmcahren]

Their first example is basically pretending basic promise concurrency on maps doesn't exist when it's been around before Promises were an official API (Search: bluebird promise map concurrency)

And the third examples are much easier to maintain with a simple function I drafted up for sake of argument `processArrayWithStages(array: T[], limit: number, stages: Array<(item: T) => Promise<T>>)`

In my experience, unless you're doing math in promises which I would recommend shifting to WASM, you're not going to feel the overhead of these intermediary arrays at scales that you shouldn't be in WASM already (meaning over several millions of entries).

The amazing people and teams working on V8 have done well to optimize these flows in the past four years because they are extremely common in popular applications. Strings aren't even copied in the sense you might think, you have a pointer to a chunk that can be extended and overwritten which incurs a penalty but saves tons of RAM.

processArrayWithStages: https://gist.github.com/DevBrent/0ee8d6bbd0517223ac1f95d952b...

[gavmor]

pipe and map are too complex for "ECMAScript devs?" give me a break!

> should be ported to WASM

valid.

[bmcahren]

Node.js pipe and map are standard implementations most developers understand. I do not know what these strange implementations are from this library. They do not make sense to me. That's what I mean. Why learn 3 different meanings of pipe? I already have to learn too many meanings of pipe across multiple environments and languages. I do not need more pipe and map meanings within javascript.

[gavmor]

Ohhh, my mistake I misread you and revealed my ignorance. Thanks for clarifying.

Edit: I'll further reveal my ignorance. What's the function piping API in node? Is it in the browser? Does work like the pipe operator |> in R, Elixir, and F#?