Random heap corruption (regression?) on servers

FiveM Discussion Bug reports
21

Okay, this is indeed another heap corruption crash except WER ended up acting slow and instead it tripped the live dump.

(this specific case of heap corruption seems to have come from a different place than usual - attaching procdump to catch a future case may be helpful)

22

Will do, iā€™ll let you know when i have a few more dump

1 Like
23

For what itā€™s worth, once 6183 is out it should fix the crash handler not getting invoked for these crashes so that it should act in a bit more predictable/obvious a way.

1 Like
24

Just wanted to share update if anyone is interested.

On the latest scheduled restart, I completely closed the server process ( closed TXADMIN ) and started it again, until now, no crashes or errors of any kind.

Ill also do this for the next 2 scheduled restarts and then let the TXADMIN do its restart procedure and notify here if then crashes occur.

Thanks!

25

Updating the server without restarting txAdmin probably led to the old version still being started somehow.

26

@Plouffe Do you have, by any chance, multiple FiveM servers on that same machine?

27

No i do not

28

So like i told you i kept monitoring, i have a few more dumps but i havenā€™t seen anything more. Im still running 6181, iā€™ll update to 6185 today.
Would it be of any help if i provided dumps from 6181 or would you rahter have the ones from 6185?

29

Okay, something that I noticed in my case. Nobody from my team launched remote desktop to do something on server for 2+ days.

Server did fine, no restarts, not issues or anything. Few minutes ago, I opened and established remote connection to turn on second test server on it.

After launching the server and then quitting remote connection, main server (115+ players) instantly crashed (while test server was still on and didnā€™t crash, both are using txadmin)

Maybe its time to switch to linux again :smiley:

30

After updating to the latest artifact a actually have a Fx crash message in the fx server console and i also have actual crash logs!
This looks like some kind of progress wich is great.

Here is the Link for the latest crash logs from fx server.

Looking foward to see whats up with it !

31

Do you happen to have a crash dump for that? Also, are you running the server from an interactive session or as an NT service?

Running background apps on an interactive session is a bad idea no matter whatā€¦

32

Hello,

I do have crash from that specific time - here is the link. Both of the servers have exact same resources turned on and are sharing same database.

To be honest, I do not know what you mean by thatā€¦ I was running both main and test server same way, by clicking on the FXServer.exe in the artifacts folder (making shortcut to it), both are separated of course.

Thanks!

33

Try using something like NSSM (https://nssm.cc/) to run it decoupled from the actual interactive session.

34

Throwing that out there but, could this be cause by ā€˜SaveResourceFileā€™ ?

35

What makes you think this to be the case?

36

I have a resources that once the server is started creates 3 pretty big json files has ā€˜Backupā€™ and it backups once every 30 minutes.

As the server crashes right on startup sometimes, i figured this could be linked to the issue.
I also remember seeing an error that mentioned something about write acces.
It might not be anywhere near my issue but im still searching.

I just rewrote the whole thing to save this data with mysql to test it out.

Also im not sure if you saw or maybe it wasnā€™t concluding but i uploaded a crash dump that was log by fxServer in my last post.

37

We donā€™t use too much SaveResourceFile. We handle everything through database mysql but it still sometimes crash

38

@Plouffe Do you have any luck with this one, did you fix crashing? :confused:

I did pretty much everything, removed most of the StateBags, changed hosting, used older artifacts.
Only difference is that I now get

[ citizen-server-impl] Server list query returned an error: System.Threading.Tasks.TaskCanceledException: A task was canceled. <- System.TimeoutException: A task was canceled. <- System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 30 seconds elapsing.

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
Unhandled exception in Mono script environment: System.NullReferenceException: Object reference not set to an instance of an object
(null)> txaEvent "serverShuttingDown" "{ļ¼‚delayļ¼‚:5000,ļ¼‚authorļ¼‚:ļ¼‚txAdminļ¼‚,ļ¼‚messageļ¼‚:ļ¼‚Server is shutting down: (Server stopped).ļ¼‚}"

Also, @nta does this line have any meaning to you

Feb 10 18:44:13 5600x kernel: [97146.307251] traps: luv_tcp5[212245] general protection fault ip:7fd02ecf09aa sp:7fd008e7d360 error:0 in ld-musl-x86_64.so.1[7fd02ecdf000+4b000]

That is the message that appears in the kernel log right when server crashes

EDIT: Also I noticed that crash usually occurs RIGHT AFTER heartbeat:

[ citizen-server-impl] Sending heartbeat to https://servers-ingress-live.fivem.net/ingress
[ citizen-server-impl] sync thread hitch warning: timer interval of 102 milliseconds


=================================================================
FXServer crashed.
A dump can be found at /root/FIVEM/MAIN/alpine/opt/cfx-server/crashes/7a3e30b6-6309-4a1e-12abd89a-da288941.dmp.
Crash report ID: bc54b898-55ee-417b-81d0-bfc57a5c0d20
=================================================================
> txaEvent "serverShuttingDown" "{ļ¼‚delayļ¼‚:5000,ļ¼‚authorļ¼‚:ļ¼‚txAdminļ¼‚,ļ¼‚messageļ¼‚:ļ¼‚Server se restartuje: (Server se zaustavio).ļ¼‚}"
39

On our server, it happens when there is Ā± 190 players And people say server crashes when a blimp drops down or someone explodes the gas station.

40

Entirely unrelated to the crashes that have been discussed in this topic so far. If that is a thing you should provide dumps for that scenario separately.

1 Like