[torginus]

I did desktop UI testing a couple years ago on Windows apps, and the standards solution there is to use UI Automation, which itself works by sending messages to each app that makes them run internal queries to find elements.

It seems like quite the intuitive approach, but we quickly discovered that due to differing implementations, and the reliance on the apps actually cooperating with you, it's actually so much more reliable and much faster to use OpenCV to physically detect UI elements by appearance.