[conartist6]

As it happens i have an even better API than this article proposes!

They propose just using an async iterator of UInt8Array. I almost like this idea, but it's not quite all the way there.

They propose this:

  type Stream<T> = {
    next(): Promise<{ done, value: UInt8Array<T> }>
  }

I propose this, which I call a stream iterator!

  type Stream<T> = {
    next(): { done, value: T } | Promise<{ done, value: T }>
  }

Obviously I'm gonna be biased, but I'm pretty sure my version is also objectively superior:

- I can easily make mine from theirs

- In theirs the conceptual "stream" is defined by an iterator of iterators, meaning you need a for loop of for loops to step through it. In mine it's just one iterator and it can be consumed with one for loop.

- I'm not limited to having only streams of integers, they are

- My way, if I define a sync transform over a sync input, the whole iteration can be sync making it possible to get and use the result in sync functions. This is huge as otherwise you have to write all the code twice: once with sync iterator and for loops and once with async iterators and for await loops.

- The problem with thrashing Promises when splitting input up into words goes away. With async iterators, creating two words means creating two promises. With stream iterators if you have the data available there's no need for promises at all, you just yield it.

- Stream iterators can help you manage concurrency, which is a huge thing that async iterators cannot do. Async iterators can't do this because if they see a promise they will always wait for it. That's the same as saying "if there is any concurrency, it will always be eliminated."

[Joker_vD]

> Obviously I'm gonna be biased, but I'm pretty sure my version is also objectively superior:

> - I can easily make mine from theirs

That... doesn't make it superior? On the contrary, theirs can't be easily made out of yours, except by either returning trivial 1-byte chunks, or by arbitrary buffering. So their proposal is a superior primitive.

On the whole, I/O-oriented iterators probably should return chunks of T, otherwise you get buffer bloat for free. The readv/writev were introduced for a reason, you know.

[robby_w_g]

> So their proposal is a superior primitive.

This lines up with my thinking. The proposal should give us a building block in the form of the primitive. I would expect the grandparent comment’s API to be provided in a library built on top of a language level primitive.

[conartist6]

How would you then deal with a stream of UTF8 code points? They won't fit in a UInt8Array. There will be too many for async iterators to perform well: you'll hit the promise thrashing issues discussed in the blog post

[Joker_vD]

No, you'll just need to (potentially) keep the last 1-2 bytes of the previous chunk after each iteration. Come on, restartable UTF-8 APIs has been around for more than 30 years.

[conartist6]

But those code points were just inputs to another stream transformation that turns a stream of code points into a stream of graphemes. Rapidly your advice turns into "just do everything in one giant transformation" and that loses the benefits of streams, which are meant to be highly composable to create efficient, multi-step transformation pipelines.

[idle_zealot]

What's stopping you from implementing a stream transformation that reads the raw stream like a parser, outputting a grapheme or whatever unit you want only when it knows it's done reading it from the input?

[Joker_vD]

No, it doesn't turn into this. Those two bytes of leftovers plus a flag are kept inside the stream generator that transforms bytes into code points, every time you pull it those two bytes are used as an initial accumulator in the fold that takes the chunk of bytes and yield chunk of code points and the updated accumulator. You don't need to inline it all into one giant transform.

Come on, it's how (mature libraries of) parser combinators work. The only slightly tricky part here is detecting leftover data in the pipeline.

[conartist6]

As an abstraction I would say it does make mine superior that it captures everything theirs can and more that theirs can't.

Plus theirs involves the very concrete definition of an array, which might have 100 prototype methods in JS, each part of their API surface. I have one function in my API surface.

[flowerbreeze]

I think the more generic stream concept is interesting, but their proposal is based on different underlying assumptions.

From what it looks like, they want their streams to be compatible with AsyncIterator so it'd fit into existing ecosystem of iterators.

And I believe the Uint8Array is there for matching OS streams as they tend to move batches of bytes without having knowledge about the data inside. It's probably not intended as an entirely new concept of a stream, but something that C/C++ or other language that can provide functionality for JS, can do underneath.

For example my personal pet project of a graph database written in C has observers/observables that are similar to the AsyncIterator streams (except one observable can be listened to by more than one observer) moving about batches of Uint8Array (or rather uint8_t* buffer with capacity/count), because it's one of the fastest and easiest thing to do in C.

It'd be a lot more work to use anything other than uint8_t* batches for streaming data. What I mean by that, is that any other protocol that is aware of the type information would be built on top of the streams, rather than being part of the stream protocol itself for this reason.

[conartist6]

Yeah it makes sense to me that the actual network socket is going to move data around in buffers. I'm just offering an abstraction over that so that you can write code that is wholly agnostic to how data is stored.

And yes, because it's a new abstraction the compat story is interesting. We can easily wrap any source so we'll have loads of working sources. The fight will be getting official data sinks that support a new kind of stream

[paulddraper]

Your idea is flatten the UInt8Array into the stream.

While I understand the logic, that's a terrible idea.

* The overhead is massive. Now every 1KiB turns into 1024 objects. And terrible locality.

* Raw byte APIs...network, fs, etc fundamentally operate on byte arrays anyway.

In the most respectful way possible...this idea would only be appealing to someone who's not used to optimizing systems for efficiency.

[conartist6]

JS engines actually are optimized to make that usage pattern fast.

Small, short-lived objects with known key ordering (monomorphism) are not a major cost in JS because the GC design is generational. The smallest, youngest generation of objects can be quickly collected with an incremental GC because the perf assumption is that most of the items in the youngest generation will be garbage. This allows collection to be optimized by first finding the live objects in the gen0 pool, copying them out, then throwing away the old gen0 pool memory and replacing it with a new chunk.

[softfalcon]

What happens when I send an extremely high throughput of data and the scheduler decides to pause garbage collection due to there being too many interrupts to my process sending network events? (a common way network data is handed off to an application in many linux distros)

Are there any concerns that the extra array overhead will make the application even more vulnerable to out of memory errors while it holds off on GC to process the big stream (or multiple streams)?

I am mostly curious, maybe this is not a problem for JS engines, but I have sometimes seen GC get paused on high throughput systems in GoLang, C#, and Java, which causes a lot of headaches.

[conartist6]

Yeah I don't think that's generally a problem for JS engines because of the incremental garbage collector.

If you make all your memory usage patterns possible for the incremental collector to collect, you won't experience noticeable hangups because the incremental collector doesn't stop the world. This was already pretty important for JS since full collections would (do) show up as hiccups in the responsiveness of the UI.

[softfalcon]

Interesting, thanks for the info, I'll do some reading on what you're saying. I agree, you're right about JS having issues with hiccups in the UI due to scheduling on a single process thread.

Makes a lot of sense, cool that the garbage collector can run independently of the call stack and function scheduler.

[vlovich123]

OP doesn’t know what he’s talking about. Creating an object per byte is insane to do if you care about performance. It’ll be fine if you do 1000 objects once or this isn’t particularly performance sensitive. That’s fine. But the GC running concurrently doesn’t change anything about that, not to mention that he’s wrong and the scavenger phase for the young generation (which is typically where you find byte arrays being processed like this) is stop the world. Certain phases of the old generation collection are concurrent but notably finalization (deleting all the objects) is also stop the world as is compaction (rearranging where the objects live).

This whole approach is going to be orders of magnitude of overhead and the GC can’t do anything because you’d still be allocating the object, setting it up, etc. Your only hope would be the JIT seeing through this kind of insanity and rewriting to elide those objects but that’s not something I’m aware AOT optimizer can do let alone a JIT engine that needs to balance generating code over fully optimal behavior.

Don’t take my word for it - write a simple benchmark to illustrate the problem. You can also look throughout the comment thread that OP is just completely combative with people who clearly know something and point out problems with his reasoning.

[conartist6]

It's not blazingly fast, no, but it's not as much overhead as people think either when they're imagining what it would cost to do the same thing with malloc. TC39 knew all this when they picked { step, done } as the API for iteration and they still picked it, so I'm not really introducing new risk but rather trusting that they knew what they were doing when they designed string iterators.

At the moment the consensus seems to be that these language features haven't been worth investing much in optimizing because they aren't widely used in perf-critical pathways. So there's a chicken and egg problem, but one that gives me some hope that these APIs will actually get faster as their usage becomes more common and important, which it should if we adopt one of these proposed solutions to the current DevX problems

[burntcaramel]

The allocation of each object still has overhead though, even if they all live side-by-side. You get memory overhead for each value. A Uint8Array is tailor-made for an array of bytes and there’s a constant overhead. Plus the garbage collector doesn’t even have to peer inside a Uint8Array instance.

[conartist6]

The engine can optimize all those allocations out of existence so they never happen at all, so it's not a problem we'll be stuck with forever, just a temporary inconvenience.

If a generator is yielding values it doesn't expose step objects to its inner code. If a `for of` loop is consuming yielded values from that generator, step objects are not exposed directly to the looping code either.

So now when you have a `for of` loop consuming a generator you have step objects which only the engine ever can see, and so the engine is free to optimize the allocations away.

The simplest way the engine could do it is just to reuse the same step object over and over again, mutating step.value between each invocation of next().

[burntcaramel]

Can optimize in theory, or are you saying this is what JavaScript engines do today?

[conartist6]

I don't think any engines today do this. It has not been worth spending the time optimizing because so few people are using iterators heavily.

[paulddraper]

Brother, you are talking about one object for every byte.

That is a madness. And often for no reason...you're copying or arranging bytes in lists anyway.

[conartist6]

That's just how string iterators work in Javascript: one object for every byte. For now it's fast enough: https://v8.dev/blog/trash-talk. I'd put the GC overhead at around 10-15%, even with 20+ objects per byte when you add up all the stages in a real text processing pipeline. It's that cheap because the objects all have short lifespans and so they spend all their lives in the "tiny short-lived objects" memory pool which is super easy to incrementally GC.

In the future it should be entirely possible for the engines to optimize even more aggressively too: they should be to skip making the object if the producer of values is a generator function and the consumer is a for loop.

[paulddraper]

One object for every UTF-16 code unit.

And yes, if that were the sole string interface, things would be very slow. String iterators are relatively less common.

[conartist6]

Code unit, yeah. Isn't for/of over an array faster than an indexed loop these days? It's fast because they took the time to optimize it.

[fwip]

I agree with your post, but in practice, couldn't you get back that efficiency by setting T = UInt8Array? That is, write your stream to send / receive arrays.

My reference point is from a noob experience with Golang - where I was losing a bunch of efficiency to channel overhead from sending millions of small items. Sending batches of ~1000 instead cut that down to a negligible amount. It is a little less ergonomic to work with (adding a nesting level to your loop).

[paulddraper]

Yes, then you are back to Cloudflare's suggested interface.

An async iterator of buffers.

[hinkley]

I did a microbenchmark recently and found that on node 24, awaiting a sync function is about 90 times slower than just calling it. If the function is trivial, which can often be the case.

If you go back a few versions, that number goes up to around 105x. I don’t recall now if I tested back to 14. There was an optimization to async handling in 16 that I recall breaking a few tests that depended on nextTick() behavior that stopped happening, such that the setup and execution steps started firing in the wrong order, due to a mock returning a number instead of a Promise.

I wonder if I still have that code somewhere…

[billywhizz]

that sounds way off. there is a big perf hit to async, but it appears to be roughly 100 nanoseconds overhead per call. when benchmarking you have to ensure your function is not going to be optimized away if it doesn't do anything or inputs/outputs never change.

you can run this to see the overhead for node.js Bun and Deno: https://gist.github.com/billywhizz/e8275a3a90504b0549de3c075...

[rrr_oh_man]

> I did a microbenchmark recently and found that on node 24, awaiting a sync function is about 90 times slower than just calling it. If the function is trivial, which can often be the case.

I dabble in JS and… what?! Any idea why?

[hinkley]

Any await runs the logic that attempts to release the main message pump to check for other tasks or incoming IO events. And it looks like that takes around 90 instructions to loop back around to running the next line of the code, when the process is running nothing else.

If you’re doing real work, 90 instructions ain’t much but it’s not free either. If you’ve got an async accumulator (eg, otel, Prometheus) that could be a cost you care about.

[conartist6]

How did you come up with 90? Can you shed any might on the difference between the cost of promise resolution and the cost of await? Is there any cost component with how deep in the call stack you are when an await happens?

[hinkley]

Essentially for loop of 10k iterations comparing `fn()` versus `await fn()` fed into a microbenchmark tool, with some fiddling to detect if elimination was happening or ordering was changing things.

I was bumping into PRs trying to eliminate awaits in long loops and thinking surely the overhead can’t be so high to warrant doing this, especially after node ~16. I was wrong.

[hinkley]

Clarifying:

‘fn()’ is the exact same function call in both cases. Only the benchmark itself was async versus sync.

Because the case under test is what’s the cost of making an async api when 90% of the calls are sync? And the answer is a lot higher than I thought.

[koito17]

I assume the reason is that `await` de-schedules the current microtask. In fact, even if you immediately return from an `await`, the de-scheduling can introduce behavior that otherwise would be absent without `await`. For this reason, code optimizers (like the Google Closure Compiler) treat `await` as a side-effect and do not optimize it out.

[conartist6]

Here is my test harness and results: https://github.com/conartist6/async-perf

[paxys]

There is no such thing as Uint8Array<T>. Uint8Array is a primitive for a bunch of bytes, because that is what data is in a stream.

Adding types on top of that isn't a protocol concern but an application-level one.

[85392_school]

A Uint8Array can be backed by buffers other than ArrayBuffer, which is where the types [0] come from.

[0] https://github.com/microsoft/TypeScript/blob/924810c077dd410...

[softfalcon]

> Adding types on top of that isn't a protocol concern but an application-level one.

I agree with this.

I have had to handle raw byte streams at lower levels for a lot of use-cases (usually optimization, or when developing libs for special purposes).

It is quite helpful to have the choice of how I handle the raw chunks of data that get queued up and out of the network layer to my application.

Maybe this is because I do everything from C++ to Javascript, but I feel like the abstractions of cleanly getting a stream of byte arrays is already so many steps away from actual network packet retrieval, serializing, and parsing that I am a bit baffled folks want to abstract this concern away even more than we already do.

I get it, we all have our focuses (and they're ever growing in Software these days), but maybe it's okay to still see some of the bits and bytes in our systems?

[conartist6]

My concern isn't with how you write your network layer. Use buffers in there, of course.

But what if you just want to do a simple decoding transform to get a stream of Unicode code points from a steam of bytes? If your definition of a stream is that it has UInt8 values, that simply isn't possible. And there's still gonna be waaay too many code points to fall back to an async iterator of code points.

[softfalcon]

I think we're having a completely different conversation now. The parent comment I originally replied has been edited so much that I think the context of what I was referring to is now gone.

Also, I wasn't talking about building network layers, I was explicitly referring to things that use a network layer... That is, an application receiving streams of enumerable network data.

I also agree with what you're saying, we don't want UInt8, we want bits and bytes.

I'm really confused as to why the parent comment was edited so heavily. Oh well, that's social media for you.

[whaleofatw2022]

Not the person originally replying, but as someone who avoids JS I have to ask whether the abstraction you provide may have additional baggage as far as framing/etc.

Ironically, naively, I'd expect something more like a callback where you would specify how your input gets written to a buffer, but again im definitely losing a lot of nuance from not doing JS in a long while.

[softfalcon]

> I'd expect something more like a callback where you would specify how your input gets written to a buffer

Yes, and this is the way it works JS currently.

The reason I'm commenting is because it appeared folks were advocating to stop doing this (despite the fact it seems to work just fine).

[pgt]

This is similar to how Clojure transducers are implemented: "give me the next thing plz." – https://clojure.org/reference/transducers

[lucideer]

Other angles of critique & consideration already covered well by sibling commenters. One extra consideration (unrelated to streams, more general) is the API design & dev UX/DX:

  type Stream<T> = {
    next(): { done, value: T } | Promise<{ done, value: T }>
  }

the above can effectively be discussed as a combination of the following:

  type Stream<T> = {
    next(): { done, value: T }
  }
  type Stream<T> = {
    next(): Promise<{ done, value: T }>
  }

You've covered the justifications for the 2nd signature, but it's a messy API. Specifically:

> My way, if I define a sync transform over a sync input, the whole iteration can be sync making it possible to get and use the result in sync functions. This is huge as otherwise you have to write all the code twice: once with sync iterator and for loops and once with async iterators and for await loops.

Writing all the code twice is cleaner in every implementation scenario I can envisage. It's very rare I want generalised flexibility on an API call - that leads to a lot of confusion & ambiguity when reading/reviewing code, & also when adding to/editing code. Any repetitiveness in handling both use-cases (separately) can easily be handled with well thought-out composition.

[conartist6]

How is it cleaner? I used to actually do that. I wrote everything twice. I even built fancy tools to help me write everything twice.

But the bigger problem here is that sync and async aren't enough. You almost need to write everything three times: sync, async, and async-batched. And that async-batched code is gonna be gnarly and different from the other two copies and writing it in the first place and keeping it in sync is gonna give you headaches.

To see how it played out for me take a look at the difference between:

https://github.com/iter-tools/regex/blob/a35a0259bf288ccece2... https://github.com/iter-tools/regex/blob/a35a0259bf288ccece2... https://github.com/iter-tools/regex/blob/a35a0259bf288ccece2...

[soulofmischief]

In the language I've been working on for a couple months, Eidos, streams are achieved through iterators as well. It's dead simple. And lazy for loops are iterators, and there is piping syntax. This means you can do this (REPL code):

  >> fn double(iter: $iterator<i32>) {
    return *for x in iter { $yield( x * 2 )}
  }

  >> fn add_ten(iter: $iterator<i32>) {
    return *for x in iter { $yield( x + 10 )}
  }

  >> fn print_all(iter: $iterator<i32>) {
    for x in iter { $print( x )}
  }

  >> const source = *for x in [1, 2, 3] { $yield( x )}

  >> source |> double |> add_ten |> print_all
  12
  14
  16

You get backpressure for free, and the compiler can make intelligent decisions, such as automatic inlining, unrolling, kernel fusing, etc. depending on the type of iterators you're working with.

[hinkley]

I think the context that some other responders are missing is that in some functional languages, like Elixir, streams and iterators are used idiomatically to do staged transforms of data without necessitating accumulation at each step.

They are those languages versions of goroutines, and JavaScript doesn’t have one. Generators sort of, but people don’t use them much, and they don’t compose them with each other.

So if we are going to fix Streams, an implementation that is tuned only for IO-bound workflows at the expense of transform workflows would be a lost opportunity.

[conartist6]

There's one more interesting consequence: you rid yourself of the feedback problem.

To see the problem let's create a stream with feedback. Lets say we have an assembly line that produces muffins from ingredients, and the recipe says that every third muffin we produce must be mushed up and used as an ingredient for further muffins. This works OK until someone adds a final stage to the assembly line, which puts muffins in boxes of 12. Now the line gets completely stuck! It can't get a muffin to use on the start of the line because it hasn't made a full box of muffins yet, and it can't make a full box of muffins because it's starved for ingredients after 3.

If we're mandated to clump the items together we're implicitly assuming that there's no feedback, yet there's also no reason that feedback shouldn't be a first-class ability of streams.

[amelius]

How do you send multiple sub-streams in parallel?