Packet Loss increase with more than 1500 players

We are currently experiencing an issue where the packet loss percentage is increasing.

This problem occurs when the number of players exceeds 1500 connected simultaneously.

When it is around 1600 players connected the packet loss is at 15%, where we start to face complaints.

When the amount of players is around 1700, the packet loss is between 30%, making it impossible and unfeasible for players to continue on the server.

The machine’s resources do not exceed 40% of use, being a 10th I9, with 64GB of RAM, SSD storage and 2GB of dedicated internet network.

We have already tested disabling several scripts that consume more, such as anti-cheat and voip.

We already have an Nginx proxy server to cache download scripts and mods from the server.

We temporarily disabled anti-ddos, which could be the cause of this problem, but we didn’t get any improvements.

Several tests and changes were performed, but we didn’t get any improvements in relation to this increase in packet loss.

  • vrpserver is the anticheat, we have already tested disable it

I don’t think anyone else even ever had ‘more than 1500 players’, but you’re likely maxing out the network thread here.

Providing an ETW trace made on the server may be a starting point.

I need to find FXServer.exe to view thread lifetimes using Windows Performance Analyzer (ETW).
But as I run the server using a .bat file, it doesn’t appear.
I tested using txAdmin, but it doesn’t appear either.
Is there any other way or suggestion to help me?

It shouldn’t ‘appear’ in the ‘Generic Events’ section, only in the ‘CPU Usage’ section. Upload the trace file if you don’t know how to interpret them, though you might’ve configured something incorrectly as the ‘CPU Usage’ chart seems fairly empty, or you’ve not captures your actual server.

Here is the link to download the ETW tracking, currently with 1000 players connected to the server.

The other trace I did for 5 seconds just to show it wasn’t showing.

Quickly looked at the trace - a lot of time spent in the network card driver on svNetwork while handling UDP datagram sends.

Trying sv_useAccurateSends false (if not already set) might mitigate this, but may lead to some sync accuracy issues as it reduces some update rate.

I believe this value can be changed at runtime, so in theory it could be enabled when at a very high player count, and then disabled later.

I didn’t know this option existed, I’ll leave it set to false like you mentioned to see the changes that will occur.

image

Do you have any other suggestions that we could make to reduce this time spent on the network card? Currently we only have one other machine for the nginx pro proxy to download the scripts and mods.
Maybe host the voip and database externally? Would there be options to reduce this time spent on the network card?