[dbish]

We've done a bit of both for our screen seachable loom-like screen recorder, the problem is that the accessibility APIs differ greatly between Mac and Windows if you want to be OS agnostic and even on Windows all the apps tend to do things a little differently making it hard to say what did you actually "see", with some apps missing key data or implementing it incorrectly. OCR ends up being easier many times desptie thinking accessibility would be.

[Sephr]

OCR is easier for the developer, but worse for the user in terms of battery drain / energy use.

[dbish]

For sure, we made a privacy tradeoff to do it server side (given some screen change delta) because of this. Accessibility is a good "in addition to" but there are just so many apps that don't handle it well