FiveM Server Problems at 450+ players (Everyone times out)

FiveM Server Development Server Discussion
, , ,
1

Hello,
For more than a month now I have had problems with server crashes if the server has more than 450 players for more than 20-40 minutes. But these are not really crashes.
I have already written a lot with Tabarra from txAdmin, but the problem has not yet been resolved.

What happens?
The network thread starts to hitch at the same time as the sync thread and all players gradually lose their connection. Strangely, the whole thing only happens after 30-50 minutes, when the server has just started, 540 players can be on it for a relatively short time without problems (I had it once)

What i am running

  • a RP server with ESX (Much rewritten and optimized)
  • 180 Scripts (Less than 100 are real scripts, the rest are addon cars, maps and car packs)
  • i dont know the exact number but around 500 Addon Cars
  • 4 bigger maps, 15 very small maps with only GTA Props
  • the integrated mumble voip server with the mumble voip script

What i have tried

  • I used to have this problem with 250-320 players when I was still using Debian 10, now, with Windows Server 2019 the exactly same thing happens with more than 450 players.
  • duk analyzed a full dump and said it was heap corruption
  • Today I switched from mysql-async to oxmysql, my last hope. The whole thing really helped the performance, but unfortunately didn’t solve the problem.
  • I have already seen many of these crashes live and had the Resource Monitor open, but the load only went down there
    Not really often more than in this picture:

I gradually increased the slots and then set them to 512 at the end just before the “crash”. I did the restart manually because everyone was down and the laggs are not stopped (so that you can play again)

I have a ton of mysql monitoring but that’s all completely harmless (I’ve also added indexes etc. everywhere)

I would be happy to receive help, if you have any questions please ask :slight_smile:

2

Can you upload the .etl file?

Also, an initial suggestion at these player counts would be to move out the Mumble bits to an external server process, we currently don’t have any obvious way to offload these from the main UDP thread and even despite lots of optimization it often does lead to a ~20% consumption of the network thread budget, where even a small interruption can eventually lead to a snowball effect of packet resends.

3

So there is no way tu turn ot off in the thread after moving the Mumble to an external server process?

4

Not sure what you’re asking here, actually. :stuck_out_tongue:

5

Whether you can deactivate the Mumble server in the main process if you have moved it to the secondary process

6

Not sure, but if you move clients to the new one using the native command it shouldn’t directly get used anymore.

7

But that won’t fix the problem so much that 1024 players suddenly go, will it?

8

… huh?

9

i will try that with mumble, have you looked at the etl files?

1 Like