Hacker News new | ask | show | jobs
by SdtEE 44 days ago
This starts from my frustration when opening large CSV files, but later evolved to a log/data analyzer that loads arbitrary format in constant time (O(1)).

https://github.com/Verticalysis/Hitomi

The secret: I engineered an incremental combinatorial parser capable of processing customized format from a steam. Any inputs, including file or the stdout from a command, are first chunked and then fed to the pipeline. The UI is ready when the first small chunk is processed.

Other highlights: 2-mode filter, one with a convenient UI and the other is based on an extensible DSL for complex cases;

Timeline mode scrollbar, a secret weapon for log or time series analysis;

Column widths fit to content automatically;

Native code, no web bloat;

Cross-platform (currently Windows and Linux, MacOS WIP).

If you are tired of all the quirks Excel have when working with CSV files, you'll gonna love it!

1 comments

Does it update the parser incrementally?
I'm not sure what do you mean by "update". If you mean the state of the parser - of course, and I did some trick to reduce the supposedly infinite states for arbitrary format to a fixed set of 4. If you mean by the parser program it self -No. The parser was developed with Dart, a statically compiled language, with no ability to modify the program itself.