I'am a Developer of an larger custom modded GTA Server (Roleplay Server), with 500-700 online users in peak hours. The community is very toxic. In the beginning, we received several DDoS Attacks daily, so we decided to build our own protection. We are working with IPtables and IPSets, as you are. But we dont bother ispecting the contents of a packet:
Our users are logging in to the Server over HTTPS to a dedicated Server, which is protected by Cloudflare. Once they verified their identify, the Server sends a Request to the Main Game-Server, which then adds the User's IP (from which the Login Request came) to an IPSet. We setup IPTables to only accept packets from IP Adresses in this IPSet. When the IPSet is updated, the Login requests finishes and the User is connecting to the Main Gameserver. Also we are unsing UDP only for Game related traffic, so we completly blocked SYN and TCP related packets in our Hardware Firewall on the Gameserver Port (its a Hetzner Feature).
Since then, we had no more major outage, besides sometimes some clients are dropping, because an attacker clogs our 10GB line. Of corse, this can't hold against every attack, but so far this is enought to protect us.