Hacker News new | ask | show | jobs
by hakunin 523 days ago
Lazy enumeration can also save memory, because you aren’t storing entire collections during intermediate steps, and it works with infinite/unknown size collections. Such as streaming data.

Some examples:

I wrote a utility gem a while ago that lets you lazily intersect, union, etc various potentially infinite streams of data. https://github.com/maxim/enum_utils/

I also used lazy enumeration for traversing the wordmap in my no-RAM static storage gem. https://github.com/maxim/wordmap/

2 comments

In the worst case, that must have intermediate space requirements equal to the entire collections, right?
I don’t think that can happen because all those functions assume the streams are consistently sorted.
Ohhhhh, nice. Yeah, sorted data is powerful. Thanks for pointing that out.
How is it different than a window, rolling window?
It's probably a version of that. But since data is assumed sorted, memory requirements almost never grow beyond one item per stream.