[tim1994]

It's not quite that simple. For example the button doesn't necessarily have to have an aria label. Could be the (nested!) content. There could be multiple candidates (buttons). The speech recognition thing is also not trivial. Yes there are WebAssembly based solutions but those have to be properly integrated and tested (not sure how good they actually are)

[bryanrasmussen]

for a Cucumber based testing solution back in 2013 the company I was working for had all element interaction commands go to a function that tried to identify what element it could be so you could write "I click 'home'" and that function would do

1. look in a object we had in which a bunch of DOM objects have been identified via CSS selectors of XPaths to see if there was an object named 'home' if that not exist

2. look for an element with the id home if that not exist

3. look for an element with the class home - take the first one. If that not exist

4. look for an element in which the text node was 'home'

There were a few other heuristics to determine what to do also dependent on if you said I select home, I click home, I move the mouse over home etc.