Hacker News new | ask | show | jobs
by joshuanapoli 601 days ago
I think that the hope/dream here is to make end-to-end tests less flakey. It would be great to have navigation and assertions commands that are robust against simple changes in the app that aren't relevant to the test case.
1 comments
chairhairair 601 days ago
It's just a dream then.

It's completely at-odds with the strengths of LLMs (fuzzy associations, rough summaries, naive co-thinking).

link
yorwba 601 days ago
Fuzzy associations seem relevant? Interact with the UI based on what it looks like, not the specific implementation details.
link
chairhairair 601 days ago
No. Both of the requirements "to interact" and "based on what it looks like" require unshakable foundations in reality - which current models clearly do not have.

They will inevitably hallucinate interactions and observations and therefore decrease reliability. Worse, they will inject a pervasive sense of doubt into the reliability of any tests they interact with.

link
tomatohs 601 days ago
> unshakable foundations in reality

Yes, you are correct that it entirely lays in the reputation of the AI.

This discussion leads to interesting question, which is "what is quality?"

Quality is determined by perception. If we can agree that an AI is acting like a user and it can use your website, we can assume that a user can use your website and therefor it is "quality".

For more, read "Zen and the Art of Motorcycle Maintenance"

link