[thorum]

The “correct”, elegant way for AI to interact with existing software would take decades and billions of dollars to build. Someone would have to do the hard work of building new APIs, solving decades of accessibility issues, etc.

Or you can show an AI screenshots and ask it where to click.

[sarreph]

I disagree if your application is networked. Most SaaS is built on RESTful APIs that can be converted trivially into interfaces / contracts for tool use.

[chatmasta]

So you can either wait for every application to do that, or at least make it possible for an LLM to do it… or you can make the LLM use a computer interface that works with every application by definition.

[Chu4eeno]

The middle ground would be leveraging e. g. standard a11y APIs, and/or hooking into applications like Squish does.

Then you get a nice textual world that fits the LLM without having to rewrite every application to have a fullblown HTTP server.

[jubilanti]

it takes decades and billions of dollars to develop APIs?