[LinuxBender]

From my experience running a Garry's Mod server a slow methodical and tedious approach is to capture some of the abuse traffic and see what is different from it and legit game traffic using tcpdump and wireshark as you mentioned. It's not the most efficient way to stop it but one can use assorted iptables modules to block the traffic if it sticks out from legit gaming traffic. Some would argue it will look the same but I have never found that to be the case. Look for things like packet length and burst rates of legit traffic then apply xt_recent or limit per second rules for different packet lengths using the iptables length module in combination with the other modules. Use your own game client IP as the normal traffic. Play aggressively.

e.g. Normal traffic may have a rate of {n} packets of 22:66 length in {t} time window. And then perhaps 67:80 length in another interval, and so on... Combine the length module with the limit module to allow normal traffic. This won't help with volumetric attacks but skiddies rarely do volumetric. It's usually just a handful of VPS nodes from some angry gamer. In those cases one could even create ipset hash tables of the attacker IP's or CIDR blocks and just silently drop them in the iptables raw table. In some cases the abuse traffic scripts will fixate on a particular string. This can be leveraged by the iptables string modules.

First use logging rules before using DROP rules so that you can see if your tuning is going to affect legit players.

The ideal solution would be to work with the game developer and show them the abuse traffic and have the game code start ignoring nonsensical traffic. I did not have this option with Garry's Mod as the SRCDS server it runs on is abandonware for all intents and purposes.