From a project management perspective, the 5 examples don't help me understand how/why I might switch from Playwright/Cypress to this framework. It seems like Bombadil is a much lower-level test framework focusing on DOM properties but in the "Why Bombadil?" introduction you say "maintaining suites of Playwright or Cypress tests takes a lot of work" ... I'd like if there was an example showing how this is true, perhaps a 1:1 example of Playwright vs Bombadil for testing something such as notifications clearing when I click clear. Basically, beefing up examples with real-world ones that Playwright users might have written is a good way to foster adoption.
This is a great point. Bombadil _is_ also tied to the DOM much like those tools, but as you focus on providing just a set of generators (which can be largely the defaults already supplied by Bombadil), you get a lot of testing from a small spec. You might need to specify some parts in terms of DOM selectors and such, and that has coupling, but I think the power-to-weight ratio is a lot better because of PBT.
The latter. For example, the concept here seems like it ought to work with the Playwright test runner for Node.js (and then you'd have compatibility with the Playwright ecosystem).
OK. Yeah so I made the decision early to go one level below Playwright and target CDP directly, because I believe I need the tight connection with Chrome/Chromium to make it as fast as possible. Also I really value the distribution aspects of writing this in Rust and compiling to a single executable with everything in it.
All I can think of is "the Token Ring had no power over him" but then I realized that "token ring" has a completely different meaning now in the age of AI.
Thanks! I don't know much about it, but from what I can tell from their site, it does differential testing based on screenshots and behavior, and mocks out network IO, focusing on the client side. I'm not sure what the AI part is, if it's for extracting information from the page, for generating actions, for smarter diffing, or some combination of those.
Bombadil doesn't mock the network (although I'm working with a contributor to add something like Playwright's Route system for manipulating responses). It runs on your full stack (if you want) and if you need determinism for perfect bug reproducibility, you should run it all in Antithesis.
On the AI front, my view is that Bombadil should be optimized for using LLMs to generate and refine specs over time (properties and action generators), but that it should be an offline process. It's not like a "browser testing agent". It's analogous to having the LLM generate a program, rather than having the LLM itself perform actions. For some problems, the latter makes more sense, but for many problems the former is better, and I believe that PBT (and things like proof systems, but that's not my domain) are suited to the former strategy.
I have some sketches on how the refinement-with-human-in-the-loop thing should work with Bombadil and LLMs, but it's only a roadmap item for now.