[xtracto]

I probably will be burned for this, but with the help of an LLM I wrote a tiny program that captures video from a browser screen (Xbox live online FPS game), passes the video images through a small trained NN that recognizes people forms and presents the video on another screen. That way I can place a green overlay on enemies and they are easier to see on PVP matches.

All that in around 100 lines of code, including the training/fine-tuning of the tiny YOLO nn.

[jerryoftheyear]

I'm curious, what amount of input lag does this introduce?

[naikrovek]

None.

It introduces output latency, not input latency.

[jerryoftheyear]

That's not how that term works. Input latency/lag refers to the time differential between triggering some kind of input and the action of that input showing up on your screen.

[naikrovek]

you described end-to-end latency, input latency + processing latency + output latency.

input latency is a measure of latency between the human input and the computer receiving the input.

output latency measures the time between a computer commanding a certain pixel to change color on the screen and the color change actually taking place.

It's not my fault these terms are often used incorrectly.

Most of the time, what you see when you see someone test "input latency" is that person actually testing end-to-end latency, which is input latency + processing latency + output latency, as it is difficult (but not impossible) to test only one of these without special hardware. testing all three at once is easy.

A proper input latency test would be (for example) some external tool sending keypresses to a computer and measuring (via a hardware debugger or some other hardware-level tooling) how long it takes for the program you are interested in to receive that input.

As stated previously, output latency is the time between your program commanding something on the screen to change and that change actually happening.

there's a third latency in this stack, and that's your program itself. how long between the time it has received an input before it commands an output device to change its output. processing latency.

for the purposes of end-to-end latency testing hardware, the processing latency is effectively zero.

all three of those stack up to become "end-to-end latency" which is what most tooling available to end users measures.

[darksim905]

Why would you get burned for this? For people who don't know, sure, this smells of 'cheating' but it's essentially just an exercise in computer vision. Also, I'd posit that using various DMA type tools in the video game modding arena are impressive and unusually secretive because they make money on subscription services. It's fascinating to me how well some of those tools and mux hardware works. When you combine that with hardware we can now buy like the DGA with more onboard memory and faster interconnects, it's going to make reverse engineering, malware analysis, forensics -- a very interesting time for those who know what they are doing.

[bombcar]

Go the next step and have the LLM simulate the controller and play the game for you.

[mft_]

Would you either share the code, or describe the approach? I’ve asked LLMs how to do something similar a few times recently, and received several different potential approaches. I’m still no clearer which route to take!