Scripts freeze when entering on a server (Only Onesync)

FiveM Discussion OneSync reports
1

Client

Using canary? No
Windows version: Any
System specifications: Various

Server

Operating system: Windows Server
Artifact version: Any
IP address: Localhost and on my public server
Resources: +150
System specifications:

Incident

Summary: If a server has a good number of streamed resources, and use OneSync, when you are entering on the server, and the game start download streamed resources all scripts just freeze, being impossible to get out from the Loadingscreen.

  • When the player has a very speedy download speed, the chances to the problem happen is low
    Expected behavior: The scripts keep running fine while downloading streamed resources.
    Actual behavior: The scripts are freezes, making the loadingscreen infinite, due to spawnmanager being blocked from execute codes to ShutdownLoadingScreen.
    Steps to reproduce: Use a clean cache, enter on a server with a lot of streamed resources and with OneSync activated, and will have a good chance to scripts freeze. (Lower download speeds increase the chances to the problem happen)

Server/Client? Client
Files for repro (if any): None
Error screenshot (if any): None
.dmp files/report IDs: (Only full memory dumps)

Any additional info:

Iā€™m getting more memory dumps from real players to help on fix that.

2 Likes
2

this is a bug report like i love to see them, very detailed thank you for taking time to make fivem better :heart:

2 Likes
3

Call stack on main thread for one of the dumps:

 	ntdll.dll!NtWaitForSingleObject()	Unknown
 	KERNELBASE.dll!WaitForSingleObjectEx()	Unknown
 	FiveM_GTAProcess.exe!rage::sysIpcWaitSema(class rage::sysIpcSemaTag *,unsigned int)	Unknown
>	FiveM_GTAProcess.exe!sub_1415C9018()	Unknown
 	FiveM_GTAProcess.exe!sub_1415C0CEC()	Unknown
 	FiveM_GTAProcess.exe!sub_140913994()	Unknown
 	FiveM_GTAProcess.exe!sub_1408FE350()	Unknown
 	FiveM_GTAProcess.exe!CScene::PreSceneUpdate()	Unknown

0x1415C9018 seems to wait for a certain background task to complete; apparently related to fwAnimDirector.

How this could remotely end up hanging is hard to tellā€¦

4

ā€¦ okay, the last dump has nothing to do with that same call stack, and was actually still loading the game. Again, not ā€˜scripts hangingā€™. Not going to download the first dump as really, 100MB parts?

Sure this is not some completely different issue somewhere?

5

Iā€™m waiting for my staff members to get more full memory dump from new players to post here to make sure what is causing that problem. And sorry for my splitted dump, i really canā€™t upload a 700MB file with my poor upload speed (0.50kb/s), it was done on my own PC, using the repro provided and caused the thing on resmon.

And basicly in all cases, they stay on the infinite loadscreen, but i will get more details to make sure to the new dumps have the same problem.

6

Here is more full memory dumps from my players:

  1. https://mega.nz/#!MZVAlCYI!NL_gdMiBntXC8rgD57SMEz7EgNuOXnM1znEkyy9UzS0
  2. https://drive.google.com/file/d/1e2CdHLYL_CFvCTS2TiP_U7sQ6UI4iytF/view
  3. https://drive.google.com/file/d/1vSZ57HnoU9oDs65Vw66qvvlJkA655lAT/view?usp=sharing
  4. https://drive.google.com/open?id=1mXdFw8EfkkZzD_vciJQt-nmc7CVIFNXx

Itā€™s being difficult to get more memory dumps because players normally has a low connection speed, and need to stay 5 hours or more stopped while uploading their dumps, or they donā€™t trust on us for send this file. But we are trying to get more dumps right now.

Edit (17/01/20):

Another dump:

Edit (18/01/20):

More dumps:

Edit (21/01/20):

More dumps:

Edit (22/01/20)

Another dump:

Edit (28/01/20)

Edit (05/02/20)

Edit (18/02/20)

8

Last one is same as here. Highly confusing as to how this could be related to any core issue still.

1 Like
9

There is another thing i can provide in order to help in fixing it?

10

Iā€™m not sure about that, but testing streamed clothes these days with a friend, him has experienced a bug who cause their clothes to stop the download, showing this on F8:

image
image1809Ɨ508 432 KB

Even when i had selected other streamed clothes and asked him to look at me, their game donā€™t download these others streamed clothes, after the failure with the first one.

And why i think this can have some relation with the Scripts freezeā€¦ Because in most of times, the F8 when the scripts freeze, i can see some clothes downloading on F8, before the game freezing. Obviously i can be completely wrong, but iā€™m trying to understand the reason to this bug, because these last days, more players are being affected by this on my public server, even me. Thank you for you support. And told me if you require more Full memory dumps for this problem, because i had told my Staff team to stop collecting these dumps.

(These prints from ā€œspawnmanagerā€ was used to debug the Scripts freeze problem, all stages has timeouts to be sure any players will not get stuck due to a script problem)

11

any clue what specific failure condition was logged here for this user? recurring download failures will practically break a game session as one canā€™t ā€˜safelyā€™ dequeue a gta streamer request

12

Probably Packet loss. I remember looking at the Connection status between us. With other friends, i had this too, even without Packet loss, but just a one time. I donā€™t thing all players who has provided Memory dumps are having Packet loss, even me yesterday on my main server, but maybe it can show a possible reason to the Scripts freeze, i hope.

13

Iā€™m one of the staffs members from the topic creator and i will stay posting full memory dumps from players who are unable to enter in the server due to infinite loading (Scripts being freezed, causing infinite loadscreen

24/06/20
https://drive.google.com/file/d/1iHv-MALhLHenZ-R2wE6dzMplpltulxdw/view?usp=sharing

Edit
06/07/2020
https://drive.google.com/file/d/1IVeJyW5U__9f3nTb0misFr8t1bOL42HY/view?usp=sharing

Edit
06/07/2020
https://drive.google.com/file/d/1mh76HVk7W-SfUtudbRy424mTCwGL6QGT/view?usp=sharing

Edit
08/07/2020
https://www.mediafire.com/file/qfhpzcud8pxhwh8/FiveM_GTAProcess.rar/file

https://drive.google.com/file/d/1W4cShC00pPlu8nJpvpdg_8C5tM3jc9fL/view

https://drive.google.com/file/d/1yYl95XbI9O0YaO04OoE5KmE50WQwWmfh/view?usp=sharing

Edit
10/07/2020
https://drive.google.com/file/d/1ANhT-SucZSvGBmTfNboKosr0yOkSYHec/view?usp=sharing

https://drive.google.com/file/d/1N0MkXNf06gqsjs7ru19rXMvzFnPk2538/view?usp=sharing

14

After the update done on morning of 07/06, we got a lot more reports of Scripts freezing while entering than the normal from our players. We are collecting full memory dumps from all of them and will post here when those dumps are available.

I tried to:

  • Update the artifacts to 2681, but nothing changed.
  • I had look at the CPU, Memory and Bandwidth usage, but nothing are strange on serverside.
  • And i had a unique case where a player was looking at the ā€œ/strdbg 1ā€, all data was frozen, and after a lot of time waiting on Loadscreen, their game had unfrozen randomly, with no visible reason. (This never happened with no other person, only one time with one person)

Iā€™m running out of ideas, and after the update from yesterday, i donā€™t know what to do because a lot of people are wanting a fix from me, but i had tried everything i can, and nothing had worked.
Thanks for your support, and i hope this problem will get a fix sometime, i donā€™t know when, but i hope soonly. I will be here to provide any info to help on this, and anything to identify this bug.

15

Didnā€™t someone at one point find out something about parachutes led to fwAnimDirector hangs - assuming thatā€™s the ones youā€™re still experiencing?

Thereā€™s a lot of potential things that can hang, especially over time it might be notable to have some newer dumps.

16

I donā€™t think my players are experiencing this due to Parachutes, because itā€™s pretty rare when someone are using parachutes on my server. My staff members are collecting newer dumps from our players and will post them here soon as they can. The most recent dumps collected was the dumps posted by @Soneka (one of my staffs members) 5 days ago.

17

Didnā€™t someone at one point find out something about parachutes led to fwAnimDirector hangs - assuming thatā€™s the ones youā€™re still experiencing?

That was me and yes can confirm, removing parachute access to player fixed it, that was 100% sure it was that, did removed all scripts one by one until it never happen again, and yeah that was the parachute.
Didnā€™t found a repro but seems related to peds around you using parachute not yourself.

18

So, iā€™m gonna remove Parachute access and block it from being used by anyone today and see if those reports will stop. Hope so :pray:
The strange part is why the FiveM client update from 07/06 had increased the relats of Scripts freeze so intensively, but lets see what will happen removing parachutes. Thanks.

19

Unfortunately, the Parachutes removal has not solved my problem :frowning:.
I had removed them yesterday, and look at reports until today, but still ocurring. Iā€™m looking for possible scripts causing this, but currently none had reduced the cases of Scripts freeze.
My staff will stay collecting Full memory dumps from our players and posting here until a fix, or the reason for this be found.
Thanks for your support and the attempt in helping me.

1 Like
20

Those last days i was doing a lot of changes to spawnmanager in order to find a possible reason for this problem, but i had no luck unfortunately. But iā€™m here to share everything i had see these days:

  • I got a decent way to repro this on my server, i was using this to test some changes:
    I enter on the server (we have a Login screen who shows up when Session became active), and when the Login screen shows up, i wait 60 seconds, put my Password and press Enter, if i wait those 60 seconds, i had a >95% chance from being stuck on Loadscreen (From 100 times i had entered to test some changes for it, 96 times i became frozen), but if i log-in pretty fast, this tax drops decently, but when more and more players are connected, even connecting fast, it still happening and you can only trust in your luck to enter without being stuck on Loadscreen.
  • I have observed too, some rare cases while waiting for those 60 seconds, some times the game stucks even without trying to connect or just doing nothing, because the resmon is completely frozen, and the Login button donā€™t work, forcing me to Close the FiveM totally.
  • Some players are creating some strange (but creative) ways trying to bypass that problem, i canā€™t prove the accuracy of them (and i donā€™t believe on them), but some players are having some success with them, soā€¦
  1. Putting FiveM_GTAProccess with High priority on Task Manager had shown a good result for some peoples
  2. Logging pretty fast when the Login screen shows up
  3. Log-in with less things possible opened, only really necessary ones, like FiveM, Steam, Discord, etc

What i had done:

  • Remove all Parachutes interactions from the servers (No result).
  • Try to avoid any possible broken interactions on Login function, using only simple functions, but nothing had shown a improvement.
  • I had disabled a lot of scripts, rewritted other ones due to a suspect in how they work, but no luck too.
  • Tried to force it to happen on my local server, but even on my main server, i can only repro it while there is at least 10~15 players.
  • Use newer Canary client.

Where it getting stuck?

  • When i press the Login button, it received data from a Net EventHandler, and when i call ANY ā€œwhileā€ function, it will became stuck, and if i call ANY ā€œCreateThreadā€ function, the code inside will not be triggered, as you can see here:
    image
    image1473Ɨ554 19 KB

    image
    The game donā€™t proceding the ā€œWaiting for Load scene: 0ā€ stage, but sometimes, it donā€™t even get there, like here:
    image
    image844Ɨ475 211 KB

    And the [T0] [T1]ā€¦ prints, are checks too see when the thread will get stuck, as you can see here:
    image
    image1150Ɨ450 14.9 KB

I guess i donā€™t have forgot anything to say here, but if i had i will add more information here. Thanks all of you, and keep your good work FiveM team.

21

As suspected, plain fwAnimDirector/crmtScheduler hang. :confused: