[mikestorrent]

Yes, and we haven't even really begun to nail down computer-use agents yet (can you believe they're still basically just OCR'ing screenshots?)

Once we have something that experiences a desktop interface more like a human does, an entire swathe of tooling that has heretofore been nigh-impossible to automate moves into the fold, and that'll be another explosion of folks finally getting to join the agentic workflow world on their industry specific apps...

[Timwi]

How do you think humans experience desktop interfaces? “Basically just OCR'ing screenshots” is exactly what humans do.

[reasonableklout]

It's not the same thing. For example, given a GUI with a titlebar, title, subtitle, text, and buttons, a human can instantly understand spatially the relationship between these items. But a naive OCR of such a GUI would be a flat stream of text that loses a ton of information.

[kortilla]

But that’s not how models handle images either. They spatially segment and reason about title bars, placement, etc.

[gabriel-uribe]

I was also under the impression modern AI agents have moved on from just OCR'ing screenshots to leveraging native vision model capabilities.

[ewild]

They do. They all use ViTs and have for quite a while.