Packet Loss increase with more than 500 players.

We are experiencing an increase when we reach 450 users/500 users online. This problem only happens when we exceed and maintain this amount of users. When we reach 500 exactly, the server exceeds almost 25% of packetloss and makes it impossible to play and stay on the server.

The dedicated hardware does not even reach 50% usage, having, we are not under ddos attack, nor any other reason why it may cause this problem.

Hardware:

  • 12th Gen Intel(R) Core™ i9-12900, 24 cores.
  • 128 GB RAM
  • 1TB SSD
  • 1Gbps dedicated internet network.

Actually we already tried to disable many random things that can cause this, such as: anticheat, voip, different scripts, etc, etc. And we are still without any solution.

We also tried temporarily disabling proxy + anti-ddos mitigation, but this had no effect either.

We tried several other things to diagnose the problem, taking out different profilers, pcaps, etc, etc, and we do not see any anomaly, neither in the network traffic, nor in the server scripts (profiler).


Profiler: https://bair.es/profileData.json

Is there any way to try to “diagnose” or “trace” this problem, in case it is hardware, or server scripts, to be able to guide us to have a solution?

Since you’re on Linux, not in any trivial way, no.

Update:

We tried to start the server without resources stremeable, and it kept happening.

Today we formatted the computer from Ubuntu 22.04 to Debian 10, and it keeps happening. At the same time we installed txAdmin to have a log of “svMain/svSync/svNetwork”, which I attach its graph.



Environment:
OS: Linux 5.10.0-21-amd64
CPU Model: Intel Gen Intel® Core™ i9-12900
CPU Stats: 16c/24t - 0.8 GHz
CPU Usage: 39%
Memory: 19% (24.29/125.60)
Version: FXServer-master v1.0.0.6231 linux


We don’t know if it’s a hardware problem, or a wrongly programmed resource inside the server. But it is completely unfeasible to track this problem. Attached are pictures of the current performance of the server.


I think it is not a hardware problem, but the packet loss would be with the dedicated one, not only with the FXServer…
I also attach a graph of WinMTR that we have not lost any packet at times when in the FXServer if we lose (+6 hours of WinMTR).

Host - % Sent Recv Best Avrg Wrst Last
192.168.0.1 - 0 38555 38555 0 0 3 0
No response from host - 100 7787 0 0 0 0 0
No response from host - 100 7787 0 0 0 0 0
No response from host - 100 7787 0 0 0 0 0
No response from host - 100 7787 0 0 0 0 0
host39.181-89-51.telecom.net.ar - 6 31413 29569 5 37 1819 116
200.0.17.173 - 0 38575 38575 4 11 26 10
No response from host - 100 7787 0 0 0 0 0
No response from host - 100 7787 0 0 0 0 0
146.99.235.45.in-addr.arpa - 1 38566 38564 4 11 30 11
________________________________________________ ______ ______ ______ ______ ______ ______

Image from the server-side where we lost only 1 packet with the server-dedicated.


I know that with all this you will not be able to give me any solution, I am just trying to look for help to see where to trace this problem. I don’t know what else to do to try to get a diagnosis of where the high packet loss is coming from when reaching 350 players, 400 players, 500 players, etc, etc, etc.

If there is a way to do it in Windows, which one is the best and how exactly is it done?

Generally you’d use UIforETW to record a trace and then upload it for someone else to analyze in detail.

this is not the right resource monitor, you should be looking at the server side resource monitor via svgui which i am not sure if it even exists in linux