Crash, Crash, Punch

hallux · 2019-06-16T04:41:12.436Z

Vulkan:

And at no point that I noticed have you told me how to provide crash information for a server that keeps no logs, and creates no dump files? Where is this information. How do I access it?

For what it’s worth, @bndzorro asked you for a few specific things you have yet to provide…

Vulkan · 2019-06-16T06:03:02.693Z

Nathan_C:

Well first off, all the crash dumps are from clients which tells me you’re complaining about client crashes, not server.

You have not read anything said. This is not about client. This is about server. And everyone is insisting i provide server logs and crash dumps, but there are none. FiveM doesnt record these things.

bndzorro:

Can you please provide some real info? Resources, ip:port etc ? What does the crash say ?

Resources are all custom, none have been changed since January 16, 2019. Which was only a minor upgrade. mostly the package is unchanged for two years.

45.58.32.76:what ever default ports are…

the crash says in client: “You have been disconnected, trying to reconnect” or whatever that exact message in red letters is, to all users, as the server crashes and restarts. A new pid results as the server starts, and that is all the information it displays.

You guys speak of server crashes as though they are well documented and I am failing to upload the logs and crashes… there are none. If there are, where are they located?

Thank you, I did over look that.

Vulkan · 2019-06-16T12:16:39.957Z

Downgrade to 1317 Crashed 8 hours later, 2 hours after it got busy.
Upgraded to 1337… Fingers crossed.

You can’t image how fatal these crashes are to a small community. We don’t all have 129 people in queue.

Nathan_C · 2019-06-16T13:00:27.869Z

What exactly happens when the crash occurs? What displays on the server console when it happens?

There are very few reasons it crashes on Linux in general so it’s interesting you’re having such trouble with it. Unfortunately due to how FiveM runs in linux there are no crash dumps or things like procdump like the windows version.

Even if it’s literally a SIGSEV in your console, post it.

Vulkan · 2019-06-16T23:54:48.163Z

Nathan_C:

What exactly happens when the crash occurs?

The red auto reconnect message comes up. All players disconnects. 10% rejoin.

Nathan_C:

What displays on the server console when it happens?

The server is started at VPS boot, the ocnsole is down, i do not know how to start at boot from script in a window. I will try to run it with console up and in a window. I cannot see the console in this manner, and do not know what it says.

Nathan_C:

Unfortunately due to how FiveM runs in linux there are no crash dumps or things like procdump like the windows version.

This is now a serious short coming for FiveM. The uncrashable server now crashes readily. A server log system should be produced. A server dump system should be produced. The reasons they cannot should be overcome by whatever means possible, even unconventional means like a secondary watcher process should be considered.

Nathan_C:

Even if it’s literally a SIGSEV in your console, post it.

I will do that as soon as I can.

Thank you for your productive answer, in a sea of accusations. You clearly read everything about this issue and are well-informed.

Linux Server 1337 Crashed at least 1 time last night. It had a definite uptime of over 4 hours and under 12 hours, quite similar to the three post-1317 previous versions. On to 1338, in a window this time…

Vulkan · 2019-06-17T09:51:58.091Z

It took three crashes on 1338, but i finally logged it. Below is the PuTTY log from the last regular join to the First resource load of the new restarted server. The server auto restarts after a crash, with a script. Everything above and below this point looks quite normal.

...
Sending heartbeat to live-internal.fivem.net:30110
Connecting: D a n i e l Key-Drop.pl
Sending heartbeat to live-internal.fivem.net:30110
server thread hitch warning: timer interval of 274 milliseconds
network thread hitch warning: timer interval of 174 milliseconds
network thread hitch warning: timer interval of 157 milliseconds
server thread hitch warning: timer interval of 151 milliseconds
network thread hitch warning: timer interval of 325 milliseconds
network thread hitch warning: timer interval of 152 milliseconds
network thread hitch warning: timer interval of 152 milliseconds
network thread hitch warning: timer interval of 280 milliseconds
network thread hitch warning: timer interval of 189 milliseconds
server thread hitch warning: timer interval of 152 milliseconds
server thread hitch warning: timer interval of 573 milliseconds
/home/_fivem/fx-server-data/Server-Starter-fivem.sh: line 1:  2991 Killed                  /home/_fivem/fx-server/run.sh +exec server.cfg +set con_disableNonTTYReads 1
FiveM server has stopped.
restarting FiveM server in 5 seconds
Starting 5M Server
Creating script environments for _cfx_internal
Found new resource...

This is the above mentioned Server-Starter-fivem.sh:

Linux Server 1141 crontab start restarts endlessly FiveM Client Support

Like this?

Nathan_C · 2019-06-17T14:31:49.791Z

Vulkan:

/home/_fivem/fx-server-data/Server-Starter-fivem.sh: line 1: 2991 Killed

So this is telling me some external process is killing your server. I was expecting a stack dump at the very least - this is pointing at something else.

Check syslog and dmesg for anything interesting. It could be any number of things, like cron killing off your server, the server itself running out of resources, etc.

Vulkan · 2019-06-18T01:08:38.936Z

Nathan_C:

syslog and dmesg

A script which works faultlessly on all the above listed mps for 5 years. One that has worked well on FiveM for several years, until 1141 when it was upgraded by a FiveM element. No harm in trying FiveM without it… thanks for your help.

FAXES · 2019-06-18T01:18:59.063Z

So is this fixed?

Vulkan · 2019-06-18T01:21:18.051Z

I will upgrade the server to 1347 any minute now, and try a normal start without the script, and see over the next 48 hours if it continues to crash.

UPDATE: 1347 is running via the common method:

bash /home/_fivem/fx-server/run.sh +exec server.cfg

Vulkan · 2019-06-18T02:13:37.844Z

Following the rebuild to 1347 i started the server with no restarting script via the normal method: bash /home/_fivem/fx-server/run.sh +exec server.cfg

several minutes elapse here, then the red disconnect message appears at the same time on two different clients, and i presume on the clients of those playing.

i close PuTTY and relogin to check top, and the fivem server process is no where to be seen in top over several minutes. (See attached PuTTY log)

top displays system processes at 6% with .7mb of ram usage without FiveM. The last top at the base of this file is my second run without the restart script. With FiveM server running, at roughly the same time as the previous crash, showing CPU at 6% and ram at 12mb which is typical for an empty server by my observations.

1347 Crashed in an uprecident short time. Under 30 minutes with 4 players on it, previous versions have given me 4-8 hours and only crash when the population exceeds 10. Maybe just bad luck, trying it a second time to see.

But what can be assumed at this point, is that the server restarter is not to blame. A once stable FiveM server, package, and startup system now crash frequently. I have made no changes to this system since January, apart from the upgrade to the restart script as advised by a FiveM element, which works perfectly.

UPDATE: the attached…

putty_1348-1.log (180.5 KB)

Nathan_C · 2019-06-18T02:24:48.410Z

I’m unfortunately not sure what to tell you since I don’t run the Linux version of FiveM. I do know that it’s not as stable and that error reporting is very …well, not there.

At this point all I can recommend is attempting the same on a different VPS. Everything seems normal based on the logs you sent (aside from a 2 year old kernel, but not the main issue here).

The only reason I can’t blame FXServer itself is because many others run on Linux with no problems at all.

Vulkan · 2019-06-18T03:03:11.787Z

Not to be repeating myself…

Vulkan:

My server was perfectly stable for two years. I made no changes for over six months. It never crashed one time. FiveM makes erratic, sudden, irreversible changes, with serious critical problems for a user, to avoid exploitation of FiveM at all costs, by banning all previous servers to date. And you want this to be my fault?

Remove the no previous servers allowed ban. And this would seem less like FiveM’s doing.

I am clearly suffering a problem with your new build that is systemic across every version since the new build, and you are not, yet you have created this error for me. And exploits are not universal, they effect some horribly and others not at all. I will take my chances with the exploits. They can’t possibly be as horrible as 100s of users being crashed every few hours.

Vulkan · 2019-06-18T05:06:25.877Z

Nathan_C:

Check syslog and dmesg for anything interesting.

the what and whatsit…

logs.rar (567.4 KB)

Vulkan · 2019-06-18T06:27:31.169Z

The server is running without the restart script. Crashed terminally without restarting. The second time this experiment has been run. Twice now FiveM 1347 has crashed without the Starter Script being active.

So what I am seeing, is that the server during a calm period where only heartbeats are registering, after a lengthy hitch session says “Killed” which the first time seems to have meant nothing. The server continues to send heatbeats, but now looks like it is failing to connect, followed by a second hitch session, and a second “Killed” which is the last thing in the log.

putty_1348-2.log (18.2 KB)

1347 not 1348^^ can’t change it

Which raises the question of linux server logging. If it can say heartbeat, and it can say Killed, why can’t it say a lot more things? If it can’t log, why can’t it create a dump file?

beryllium · 2019-06-18T06:37:15.985Z

Your dmesg is literally full of the server process (and other processes, such as apache2) getting killed due to being “out of memory”. It seems your server has only 512 MB of RAM, and you’re running multiple intensive server processes? This was a common amount of memory for a PC in 2003, not in 2019. Due to all the background processes going on, the server is getting killed when its memory usage is merely 128 MB, which is a perfectly normal amount of memory to be used by a game server.

Try shutting down stuff like apache2 (which takes around 70% of your server’s memory), exim4, and other things you don’t need, or get more RAM.

Vulkan:

If it can’t log, why can’t it create a dump file?

Even if this’d be a normal crash (which it isn’t - it’s the OS forcibly terminating the server because you’re running with only 512 MB of memory with random other processes using 70% of it), making a crash dump file on Linux would be completely futile since there’s no software that can open it due to the way Linux works with dynamic linking.

Vulkan · 2019-06-18T06:56:54.228Z

beryllium:

Your dmesg is literally full of the server process (and other processes, such as apache2 ) getting killed due to being “out of memory”.

While running FiveM Server and an apache2 stress test, CPU 23%, and 323MB RAM free. Apache2 under test appears to not be occupying more than 20% of ram. I additionally have 512mb of swap which goes unused 24/7 for 3 years.

Out%20of%20Ram%20top%20while%20loading%20apache2

free -m appears to confirm this

Out%20of%20Ram%20free%20-m

The only tasks running on this server is 1 website @ 250 users a day, and 1 usually half full FiveM server. The website operates perfectly, I’ve never once seen a download or page load fail. FiveM until 1317 ran perfectly. I never once saw it crash. The VPS is largely unchanged for three years.

molybdenum · 2019-06-18T06:59:14.385Z

Vulkan:

which goes unused 24/7 for 3 years

Presumably, your server is configured to just kill processes before using it. Linux memory management is a weird one, it’ll OOM kill long before it indicates anything as being wrong.

Vulkan · 2019-06-18T07:05:24.229Z

Of interest is that 1347 seems to fail following the hitch warning sessions. I have an identically prepared VPS that I dev FiveM on, that is currently idling the same setup, on 1244, waiting for FiveM to stop blocking the masterlisting of it. And in its log i see the same behavior of the FiveM server/package a large hitch session, perhaps of my own causing. But unlike post-1317 it does not crash as a result of this hitch session, which is currently occurring with no players on the server, none ever joined:

1244%20Not%20Crashing%20with%20same%20package%20and%20same%20hitch%20sessions963×1045 322 KB

961-1244 hitches and survives, 1317-1347 hitches and crashes every time.

Vulkan · 2019-06-18T07:08:39.183Z

My VPS is a very stock Debian setup, as advised by the great tutorials at Digital Ocean. Usually anything said following the word Linux takes me several hours to understand and implement. There is no fancy configuration here. What ever Debian does stock so does mine.