Packet Loss increase with more than 1500 players

We are currently experiencing an issue where the packet loss percentage is increasing.

This problem occurs when the number of players exceeds 1500 connected simultaneously.

When it is around 1600 players connected the packet loss is at 15%, where we start to face complaints.

When the amount of players is around 1700, the packet loss is between 30%, making it impossible and unfeasible for players to continue on the server.

The machine’s resources do not exceed 40% of use, being a 10th I9, with 64GB of RAM, SSD storage and 2GB of dedicated internet network.

We have already tested disabling several scripts that consume more, such as anti-cheat and voip.

We already have an Nginx proxy server to cache download scripts and mods from the server.

We temporarily disabled anti-ddos, which could be the cause of this problem, but we didn’t get any improvements.

Several tests and changes were performed, but we didn’t get any improvements in relation to this increase in packet loss.

  • vrpserver is the anticheat, we have already tested disable it
1 Like

I don’t think anyone else even ever had ‘more than 1500 players’, but you’re likely maxing out the network thread here.

Providing an ETW trace made on the server may be a starting point.

I need to find FXServer.exe to view thread lifetimes using Windows Performance Analyzer (ETW).
But as I run the server using a .bat file, it doesn’t appear.
I tested using txAdmin, but it doesn’t appear either.
Is there any other way or suggestion to help me?

It shouldn’t ‘appear’ in the ‘Generic Events’ section, only in the ‘CPU Usage’ section. Upload the trace file if you don’t know how to interpret them, though you might’ve configured something incorrectly as the ‘CPU Usage’ chart seems fairly empty, or you’ve not captures your actual server.

Here is the link to download the ETW tracking, currently with 1000 players connected to the server.

The other trace I did for 5 seconds just to show it wasn’t showing.

Quickly looked at the trace - a lot of time spent in the network card driver on svNetwork while handling UDP datagram sends.

Trying sv_useAccurateSends false (if not already set) might mitigate this, but may lead to some sync accuracy issues as it reduces some update rate.

I believe this value can be changed at runtime, so in theory it could be enabled when at a very high player count, and then disabled later.

1 Like

I didn’t know this option existed, I’ll leave it set to false like you mentioned to see the changes that will occur.

image

Do you have any other suggestions that we could make to reduce this time spent on the network card? Currently we only have one other machine for the nginx pro proxy to download the scripts and mods.
Maybe host the voip and database externally? Would there be options to reduce this time spent on the network card?

VOIP would probably help. But database, no. (Assuming you’re running it locally currently).

I’m not sure how the spacial stuff is worked out for voip and how much traffic that is, but offloading voice to another server would reduce traffic on the game servers nic.

As for the database stuff, if hosted locally, you won’t be using the nic for local traffic. But if you offloaded it to an external server, that would increase latency and also network traffic doing those queries.

I could be wrong about all this, but as far as a networking/server aspect of things the above should apply. However, I’m not 100% certain about the VOIP stuff as I dunno how fivem handles that data/traffic.

In the course of these months, only today has progress been made in relation to this problem.

I validated that whenever I run it on the server-side, GetPlayerPed(src) returns me as 0 for all players that are in this solo session.

I wonder if anyone knows a possible change or solution to this problem.

1 Like

Hi everyone,

I’ve also been having this issue. With more than 800 players some data isn’t getting synced with the server.

Like @yWilliam said, for some players their GetPlayerPed(src) returns 0.

I’ve tried switching to linux but the issue was still exactly the same, so I came back to Windows.

Also this RPC native GiveWeaponToPed after that player count sometimes doesn’t work (and the ped is valid)

Some other issues are vehicles getting deleted with a player inside, and vehicles being only visible to a certain player altough they are Networked. Both of this issues occur with valid players that are not in that weird state solo session.

I’ve recorded some ETW Traces with the hope of finding the cause of this issue.

https://drive.google.com/file/d/1hkGPEnG8_CgI8LiBWv6BhmtiUJQXClra

They both were recorded with 850 players, one with sv_useAccurateSends false and the other one sv_useAccurateSends true

Thanks in advance!

I don’t think there is anything we can really do in this regard here (regarding non-performance bits, i.e. everything after the recent bump in this topic: nowhere before was any sort of ‘solo session’ mentioned, perhaps @yWilliam bumped the wrong topic?).

‘Traces’ or other logs that commonly are provided won’t help at this scale, and there’s so many different issues going on with multiple potential different root causes, that without direct access to an environment experiencing these issues (at the exact moment they start occurring!) there’s really nowhere we can even start, and even with access it’d be a lot of wild guesses at best.

If anyone in an aligned time zone and without much of a language barrier would be up for such investigation and could give access to whatever (and ideally is in touch sufficiently with players that they can ask some players for followups) that’d really be appreciated since a lot of these issues are impossible to experience let alone replicate outside of ‘real’ game scenarios.

1 Like

Hi @nta,

We can make that work! I have only 2 questions,

What time zone are you at? And where do you want to talk? Discord?

Cheers

Ik am having the same issue but this happens when I reach 240 players, I have a roleplay server which has a decent a lot of scripts.

I have used several tools to check several scripts using TriggerClientEvent which has been found to cause network issues in the past but all of my scripts seem to not overuse it nor sends to many players( -1) too often.

I have talked with the owner of my hosting company several times and he keeps saying it is a script and not a problem on his end. I have tested rigorously and cannot find a solution to this.