[kadoban]

For shooters, bots or assists that only use vision and normal input (special mouse/keyboard) aren't hard to imagine and I'm positive already exist. I know specifically I read a while back about mice that do recoil correction to improve aim, and things like "click when something that looks like a head is in the crosshairs" would be a weekend project at most for a bored college student.

Strategy games may be harder, but still not going to take too long before those exist.

[perth]

Funny thing, there are already YOLOv3 derived aimbots, and a lot of anti cheat now scans pci-e cards to try to detect them. They intercept video input and act as mouse and keyboard devices, acting almost completely independent of the host computer.

[kadoban]

Yeah that's not far at all from just running as a separate machine, either grabbing the hdmi output or going full webcam pointed at the screen.

[patmcc]

It's not hard to make cheating tools, it's hard to run them on the same hardware as the game and not get caught. Chess cheating tools don't need to be on the same hardware, so they're not detectable in the same ways.

[krageon]

It is not hard to run cheats for basically any game that uses video on another machine. Just a little more expensive (i.e. it is harder for someone in high school, but not anyone over 20).

[Orothrim]

Due to the need for real-time input that's just not true.

[bombcar]

There are/have been elaborate cheating setups for online shooters that involved a second computer using the VGA or HDMI video stream.

It’s doable to get around kernel detection tools.

[mminer237]

Splitting the video stream to a second computer to run real-time video analysis and then send back commands through an emulated mouse may be doable, but it's certainly harder for anyone to do than copying chess moves on your phone.

[bombcar]

Certainly! But it does show the amount of effort people will go to cheat - though lots of the "online shooter" cheating is people selling cheating tools/kits to idiots who want to win (basically turning games into pay to win).

The number of people who can build that kind of system from scratch is small.

But anyone can cheat at chess online.

[robswc]

That’s fascinating… do they work long term? With only certain games?

[bombcar]

I don't have details, but they're originally were designed for specific games - but the mechanisms can be adopted for others. The goal is to get a slight advantage on headshots, I believe.

https://arstechnica.com/?p=1779166

[kadoban]

My point was that video game cheats _don't_ need to be on the same hardware.

[rowanG077]

I doubt that since a weekend project. You need to analyze a video stream of hundreds fps in real time with very low latency requirments. You'd probably need an FPGA to do that. Custom training an AI to only trigger on heads.

[rippercushions]

No, it's trivial these days, and this very task is one of the introductory tutorials to Google's Coral AI Tensorflow boards:

https://coral.ai/models/object-detection/

Which you can buy as a USB stick for $59:

https://coral.ai/products/accelerator/

Sure, it's a little more complex if you want to train on specific heads as opposed to a generic model, but not hugely so. The biggest challenge would likely be downsampling the video stream into something the model can process quickly enough.

[WhitneyLand]

When you say it’s a weekend project for a college student it suggests it’s mundane or easy for most professional developers within a weekend.

This sounds exactly like the old “Twitter would be easy to implement give me a month” naivety.

Development is hard, even small projects aren’t usually finished in a weekend, let alone something there’s no reference implementation for and that we’re merely speculating about.

If such a project was built in a weekend to work reliably enough others could use it successfully in gameplay, it would be highly impressive. Maybe we’d call it an extraordinary effort, if for no other reason “because software”.

[rowanG077]

That's a generic AI. I would be very surprised you could get it to work accurately for games and hyper specific the heads of (possibly non-human)models. The box you see in the marketing material is completely useless for this use case. Besides how do you get the data to your standard second PC, scale the input the USB thing expects. Send it over USB and back, and then you have to send it to a custom mouse that receives the click command over bluetooth? I don't think 7ms just for the inference step is going to cut it.

A weekend project were you have to design an ultra low-latency software + hardware stack, a custom mouse and train a custom AI. I would be incredible if anyone manages this in a weekend.

[krageon]

If you want absolutely olympic response times (110-120ms), it is a little tight but nowhere near impossible. 7ms as the other poster says is absolutely insane. Most videogames will already trigger on consistent peak human response times, so you'll want to trim down your bot to go to a more average person (160ms, maybe a bit higher) to fly under all the heuristics they might (but probably don't) use. Plenty of time to compute things.

[rowanG077]

7ms is enough. But that is one in a chain of latencies. 110ms is way too long if you want your trigger to be effective. In 110ms the enemy has moved out of your crosshair. Humans deal with that with dynamic adjustment but the AI will just click the mouse. I'd think you'd need to hit less then 25ms end to end and even that might not be enough for far away targets.

[xxs]

I mentioned earlier - reaction time under 100ms on track events is considered a foul start and the offender disqualified.

[barrkel]

7ms isn't going to be the bar. Human reaction times are in the 150+ms range, you have 20x your budget.

I agree it's still tough though, most I/O paths on generic hardware will have buffers at every step that you'll have to fight to eliminate.

[rowanG077]

7ms is enough. But that is one in a chain of latencies. 150ms is way too long if you want your trigger to be effective. In 150ms the enemy has moved out of your crosshair. Humans deal with that with dynamic adjustment but the AI will just click the mouse. I'd think you'd need to hit less then 25ms end to end and even that might not be enough for far away targets.

[swores]

25ms would be quite obvious, and absolutely wouldn't be necessary.

Humans regularly manage to hit heads without that reaction time, and a human will still be involved in the aiming / predicting where to aim.

[barrkel]

The AI would need to lead the target under any reasonable implementation. Just clicking the mouse when crosshairs are over a target would scarcely deserve the name AI.

[kadoban]

I was thinking a decent proof-of-concept. All you'd really need is a generic object detector and fake mouse.

Latency _could_ make it a lot more difficult, but beating a human I don't expect would be hard.

You don't need to do hundreds of fps. Just grab the newest frame, process it, repeat. Missing frames at most increases effective latency or means your cheat isn't 100% effective if you miss a head. It's a sliding scale of improvements, not a deal breaker.

You also don't even necessarily have to process the whole frame. Just the bit actually _at_ the crosshairs is probably going to be enough for a crappy version.

And a fake mouse is just usb-hid, usb gadget whatever search terms, not like you'd have to break any new ground there.

I consider it a "weekend project" because it's just throwing together a couple of existing libraries in a fairly standard way. Like most things, cleaning it up enough to be perfect could/would take far longer.

[xxs]

You don't need 'hundreds' fps - you can do it w/ 10sec and get better reaction time than humans (reaction under 100ms in track events is consider foul). It should not be so difficult with a separate GPU and a capture card. Serial mouse/keyboard inputs are trivial as well.

[rowanG077]

How is Serial mouse/keyboard trivial? You'd need to make a custom mouse that receives data from your second PC. You'd probably want it to be wireless else you will have 2 cables coming out of it. And no just having a second fake mouse is not fair. That's easily detectable by software and would definitely raise a lot of red flags.

[xxs]

This is how: take ESP32 - it has bluetooth, wifi and serial port all integrated and it costs couple of dollars. It's small and it runs on 3.3V, it has a voltage regulator that it allows to connect to 5V.

I can do the code and place the esp32 in a mouse on a Saturday (my C is always rusty, not using it professionally). And I am a hobbyist at best. So the original serial mouse/keyboard are connected to ESP32 that normally proxies the signal of the hand movement to the PC.

If I press a pedal (w/ foot) the second computer would receive a signal, calculate a human alike trajectory to move the mouse and send it to the ES32. The latter will execute it along with the left click to shoot.

Like I said - trivial. The same can be done with the USB port as well, and it's not harder. Just that for PS/2 I have the tools laying around.

[seba_dos1]

It is trivial. All you need is a device that can work as a USB gadget that you can plug your keyboard into. I could do it with my phone, many Raspberry Pi-like SBCs, or even Arduino-like boards...

[perth]

YOLOv3 can do it

[Semaphor]

Those have been common for decades and are pretty much the lowest bar for detection.

[socialismisok]

Using computer vision on the video stream (eg via a capture card) and then sending valid mouse and keyboard inputs most definitely has not been around for decades. It's a mechanism to cheat that has zero binaries running on the host computer.

[eru]

You could do statistical analysis on the mouse and keyboard inputs to try and detect this.

(Very similar to how you might detect chess cheaters, actually.)

[erwincoumans]

You can use (the data of) statistical analysis to train a deep neural network to avoid detection. That may also help avoiding chess detection cheaters, so the moves are not perfect anymore.

[eru]

Yes. You can't make cheating impossible, but you can make it harder to pull off.