[datagram]

There's already companies working on this: https://axiom.ai/

I would guess that it's just a matter of converting the DOM accessibility tree into text descriptions, e.g. "There is a button that says 'Start'" And then converting text like "Click the Start button" back into an actual action on the page.