[DanyWin]

Here we just provide natural language instructions and the LLMs generate the code appropriate at a given time. If the site changes, we can regenerate the code using the same instruction, so unless the site changes a lot, it is quite robust

[atonse]

Right so in general I can see this in use by development teams itself cuz we don't want to sit there and manually write tests.

I'd love to tell it to just log in to my own website, click on certain pieces of functionality and repeat that. Especially with more casual day to day tasks.

Heck, we could even auto-generate tests from a bug report (where the steps to reproduce are written in plain english by non-technical testers).

That means less time for a dev to actually reproduce those steps, right?

[DanyWin]

Exactly! In the future, testers could just write tests in natural language.

Every time we detect, for instance with a vision model, that the interface changed, we ask the Large Action Model to recompute the appropriate code and have it be executed.

Regarding generating tests from bug report totally possible! For now we focus on having a good mapping from low level instructions ("click on X") -> code, but once we solve that, we can have another AI take bug reports -> low level instructions, and use the previously trained LLM!

Really like your use case and would love to chat more about it if you are open. Could you come on our Discord and ping me? https://discord.gg/SDxn9KpqX9

[atonse]

I don't use discord much but joined to provide any additional thoughts.