Hacker News new | ask | show | jobs
by hanrelan 3034 days ago
We tried matching against known elements (template matching), but unfortunately that didn't hold up well given the large number of resolutions games support.

One problem is that the elements themselves don't necessarily scale proportionally. For example, if you have a button with some text in it and you reduce the resolution, often the game will keep the text larger but reduce the padding around the text. This makes template matching tough.

We found blob detection works pretty reliably and uses minimally CPU, but we're not CV experts so if anyone has other ideas we'd love to hear them!
3 comments
chatmasta 3034 days ago
I’m assuming you can’t just look at the active process because the game might be in the foreground but the user is in a lobby. I would look at the bandwidth profile of the game over 30 second windows. I bet it looks way different while playing a game vs. sitting in the lobby, and you can watch the traffic without modifying the binary of the game (at least, you can watch the relative volume of traffic, and the IP addresses it interacts with).
link
clemmakesapps 3034 days ago
It would be nice to see a metric for performance impact of this blob detection so that gamers with less beefy computers don't have to compromise their gameplay settings to accommodate for this app
link
hanrelan 3034 days ago
We don't have official benchmarks, but you shouldn't see much of an impact at all since we only do it once a second and on a small portion of the screen. If you do notice a difference please let me know.
link
ngokevin 3034 days ago
Can you read from memory? Speedrunners use programs that auto-split (detect when the runner has reached a certain stage of the game), I think by reading from memory.
link
hanrelan 3034 days ago
We thought about doing that but didn't explore it much to be honest. One issue we have to be careful of is not triggering the anti-cheat features of these games. I'm not sure if looking at memory without modifying it does that.

The blob detection does work surprisingly well though, doesn't take us very long to add a new game using it. Most of the time is spent making sure we understand the flow of games we don't play ourselves.

link