[jtemplestein]

I wonder if this screen + mouse + keyboard (+ camera + speaker + mic) interface is really the right level of abstraction to model a “digital entity”

Sure, you can do everything a human can, but it also seems VERY inefficient

As an alternative, maybe you could just do network in/out?

[evanjrowley]

It's the same approach as Windows Recall, but all data remains sovereign to the company generating it.

[vorticalbox]

for agent agents we have ACP [0] surely their time would be better spent builing this sort of abstraction for computer use then simple teaching an AI to use a mouse?

The computer UI is the way it is because that is optimal for humans, if your plan is to replace humans why not just replace the whole stack os and all to something these models already know how to use?

[0] https://zed.dev/blog/acp-registry