[a-dub]

i'm curious: how does the steady state error rate of a stochastic automated system like this compare with the downtime and errors that come from a (brittle) deterministic bridge that can fail with upgrades? what does the observability look like? (i'm guessing one feature is that the execution log including images/screenshots for each transaction gets saved, which is probably a huge improvement.)

[fchishtie]

it’s a good q - we experimented a lot with computer use / agentic automation and found that at scale a hybrid solution where the automations run as deterministic code with agents for recovery is the best - running automations as code is faster & cheaper & when you’re doing critical tasks (like updating patient records) you don’t want an agent to potentially mess something up.

previously writing RPA code used to take a long time - using AI (and its infinite patience) we can write more durable code that covers more edge cases

And since they’re code based it’s pretty straightforward to an agents monitor them and update their code when upgrades to the underlying system happen etc…

for observability - we have workflow execution logs that store text, videos and screenshots so an agent or a human can debug them - lots and lots of webhooks when things break ! (:

[_crowecawcaw]

I also experimented with vision/screenshot based computer use tools for similar use cases but had inconsistent results. LLMs had trouble getting precise pixel coordinates from a screenshot to move a mouse. And the screenshots took extra tokens. I had a lot more success using accessibility APIs to replace screenshots + input simulation since accessibility data is easier for LLMs to process. The accessibility functionality is now released as a separate library for building automation tooling: https://xa11y.dev/

[fchishtie]

cool! thank you for sharing - will check it out

[a-dub]

that sounds like the way. also keeps runtime costs down. seems like the trick there would be to build (a|on top of) durable rpa librar(y|ies) that the agents and humans work in so that the automation recipes and their automated updates can be quickly skimmed and sanity checked when needed. add in some live automated testing (assuming you can make this happen with the legacy systems) and maybe you could get really close to fully automating all of it.