[stared]

Is it problematic to capture screen and generate mouse/keyboard input?

In that way there would be no need of any privileged API... and it would be more natural (i.e. interacting with the real way, instead of its abstracted version).

[tantalor]

Indeed, DeepMind has already demonstrated mastering a game using only screen input and controller output, e.g., Breakout https://www.youtube.com/watch?v=V1eYniJ0Rnk