[adamsiem]

Anyone using vision to parse screenshots? QVQ was too slow. Will give this a shot.

[logankeenan]

I used molmo to parse screenshots in order to detect locations of UI elements. See the repo below. I think Omni parser from Microsoft would also work well.

https://github.com/logankeenan/george

https://github.com/microsoft/OmniParser

[abrichr]

You might be interested in https://github.com/OpenAdaptAI/OpenAdapt