[zamalek]

It's been a while, so I'm a bit hazy on the details. Every iteration of the code had the scanner+parser approach. The real problems started when testing the parser (because that was the double-encapsulated `Parser<Scanner<IO>>`). This means that in order to test the parser I had to mock complete VT100 data streams (`&mut &[u8]` - `&mut b"foo"` - is fortunately a Reader, so that was one saving grace). They were by all standards integration tests, which are annoying to write. Past experiences with fighting the borrow-checker taught me that severe friction (and lack of enjoyment) is a signal that you might be doing something wrong even if you can still get it to work, which is why I kept iterating.

My first few parser designs also took a handler trait (the Alacritty VT100 stuff does this if you want a living example). Because, you know, encapsulate and DI all the things! Async traits weren't a thing at the time (at least without async-trait/boxing/allocation in a hot loop), so fn coloring was a very real problem for me.

The new approach (partial, I haven't started the parser) is:

      input.read(&mut data);
      tokens = scanner.push(&data);
      ops = parser.push(&tokens); 

Maybe you can see from that how much simpler it would be to unit test the parser, I can pass it mock token streams instead of bytes. I can also assert the incremental results of it without having to have some mock handler trait impl that remembers what fns were invoked.

I'm not sure if that really answers your question, but as I mentioned: it's been a while. And good luck with the learning!

[binary132]

Correct me if I’m wrong, but would another way of saying this be: write parsers in terms of a buffer, not in terms of IO?

[zamalek]

Yup, that makes sense.

[j1elo]

Thanks a lot! Yeah I see how the simpler design that has non-stacked implementations one on top of another is easier to understand and test. Yours is not only a lesson in design for Rust, but in general for any technology! This later idea is just to compose parts together, but those parts are able to work independently just fine (given properly formatted inputs). A simpler and more robust way of designing components that have to work together.

[zamalek]

Totally, Rust has substantially affected how I write code at my day job (C#). If I ever get round to learning a functional language, I'm sure that would have a much bigger effect.