|
|
|
Show HN: An ETL engine with hard memory guarantees
(github.com)
|
|
2 points
by logannyeMD
222 days ago
|
|
This is an external-memory ETL engine that enforces hard memory bounds (as opposed to systems that can only try to stay within limits). It's the result of playing around with a recent idea from complexity theory, tinkering to find practical applications of where efficient time-space simulation could be useful. It enforces a strict memory cap via RAII tracking and scheduling, so you can process arbitrarily large datasets with a fixed, small memory footprint. Operators automatically spill to disk when needed. Might be helpful for serverless and edge environments where you need predictable resource usage. |
|