NUI restarts to random players with more than 50+ players and after randoms mins

[Fujy]

Our server has a big issue with CEF (Nui resources) and we struggle to fix since August when we first see this issue for few players!

So what we’ve seen so far to reproduce the bug on our server:

• we need at least 50 players connected on server (we’ve tried to repro on dev with 10-20 players we couldn’t and also on live when there was less then 50)

Other things to mention:

• it happens random to players right after connect, after 30 mins
• the time is not fixed so its not happening for example every 30 mins, 5 mins or so
• is not happening to all players in the same time, so could be max 1-3 players in the same time, but mostly only 1
• for some of them they only have all NUI js scripts restarted and they can continue playing
• for others they get stuck on chat and they can’t do anything or other way
• for others they get crash because of nui-resources.dll
• i’ve seen something about of alt-tab but i cannot say i really can reproduce but i’ve seen happen a lot to the players when they alt-tab
• sometimes in F8 they get an error about FOCUS
• we’ve set few js scripts to do console.log when the script is loaded, so everytime occurs this, the clients will have in their F8 console those outputs about those js reloading

When crash to clients they quit with these errors:

• Timed out after 60 seconds.
• Game crashed: VCRUNTIME140.dll!memmove (0xca)

In most of the crashes is because of nui-resources.dll and specific to the file RPCSchemeHandler.cpp at line 134.

We’ve tried to figure it out why it happend after a CallBack and we assume is because of a timeout to a the method: RegisterNUICallback but we couldn’t fix it.

Info about our server:

• We run the server on linux server (Ubuntu 18.04 LTS)
• We use vrp2 framework
• 1Gbps Internet, 32 GB Ram, dedicate server OVH (usualy we are around 6gb/32 ram, 2-2.5 load average)
• Infinity 100(morning, mid-day) to 200(evening, midnight) out of 200 players, and when is full we have a queue of 1 up to 50+

As we don’t know from which resource we have this problem we’ve tried:

• remove all the resources and let only vrp2
• remove scripts one by one (those who have nui)
• we moved the server to a windows machine just for test because we thought maybe is because of linux build and to exclude maybe is because some firewall from linux machine, ovh etc

We also thought maybe is because of the framework. But as we’ve seen other communities with vrp2 running without this problem is obvious something from out behalf.

If need more info I will give you asap.

Thank you in advance for any clue or info or guide on how can we debug this, or fix it.

When happen all the nui resources restarts and we have some console.log to print when the script is loaded:

Some crash dumps:

CfxCrashDump_2020_10_18_16_47_45.zip (3.4 MB)
CfxCrashDump_2020_10_18_12_04_39.zip (2.4 MB)
CfxCrashDump_2020_10_15_08_13_30.zip (2.6 MB)
CfxCrashDump_2020_10_14_21_11_25.zip (2.8 MB)
CfxCrashDump_2020_10_14_08_59_36.zip (2.0 MB)

[nta]

Are you able to upload a few of the older crash dumps from crashes/ in addition to the .zips you’ve already uploaded? CEF crashes are usually not fatal (so not included in the .zip) however this seems like an odd combo of one process crashing then NUI core stuff getting invalid data right after.

Also, the CEF process crash is odd as it’s very recurrent for those who have it but 80% happen on one server… with the following stack trace, are you doing anything odd/custom with audio in NUI? Might be an actual crbug somewhere.

blink::MediaElementAudioSourceHandler::ProvideResamplerInput
blink::MediaMultiChannelResampler::ProvideResamplerInput
media::SincResampler::Resample
media::MultiChannelResampler::Resample
blink::MediaMultiChannelResampler::Resample
blink::MediaElementAudioSourceHandler::Process
blink::AudioNodeOutput::Pull

[Fujy]

Thank you for your time!

The crashes from above we’ve from a player of our server. I will try to give you my crashes (i’ve reinstall it recently fivem so I don’t have much but I can get from players a lot of them)

I did a grep in the server for: MediaElementAudioSourceHandler, MediaMultiChannelResampler, SincResampler, MultiChannelResampler, MediaMultiChannelResampler, MediaElementAudioSourceHandler, AudioNodeOutput, ProvideResamplerInput, Resample but i didn’t find any occurrence.

What i’ve found yesterday doing profiler to check i saw that vrp/client/audio.lua consumes a lot:

vrp/client/audio.lua code:

Line 24-39

Citizen.CreateThread(function()
    while true do
      Citizen.Wait(self.listener_wait)

      local x,y,z
      if vRP.cfg.audio_listener_on_player then
        local ped = GetPlayerPed(PlayerId())
        x,y,z = table.unpack(GetPedBoneCoords(ped, 31086, 0,0,0)) -- head pos
      else
        x,y,z = table.unpack(GetGameplayCamCoord())
      end

      local fx,fy,fz = vRP.EXT.Base:getCamDirection()
      SendNUIMessage({act="audio_listener", x = x, y = y, z = z, fx = fx, fy = fy, fz = fz})
    end
  end)

Line 42-86

Citizen.CreateThread(function()
    local n = 0
    local ns = math.ceil(self.voip_interval/self.listener_wait) -- connect/disconnect every x milliseconds
    local connections = {}

    while true do
      Citizen.Wait(self.listener_wait)

      n = n+1
      local voip_check = (n >= ns)
      if voip_check then n = 0 end

      local pid = PlayerId()
      local spid = GetPlayerServerId(pid)
      local px,py,pz = vRP.EXT.Base:getPosition()

      local positions = {}

      local players = vRP.EXT.Base.players
      for k,v in pairs(players) do
        local player = GetPlayerFromServerId(k)

        if NetworkIsPlayerConnected(player) or player == pid then
          local oped = GetPlayerPed(player)
          local x,y,z = table.unpack(GetPedBoneCoords(oped, 31086, 0,0,0)) -- head pos
          positions[k] = {x,y,z} -- add position

          if player ~= pid and self.vrp_voip and voip_check then -- vRP voip detection/connection
            local distance = GetDistanceBetweenCoords(x,y,z,px,py,pz,true)
            local in_radius = (distance <= self.voip_proximity)
            if not connections[k] and in_radius then -- join radius
              self:connectVoice("world", k)
              connections[k] = true
            elseif connections[k] and not in_radius then -- leave radius
              self:disconnectVoice("world", k)
              connections[k] = nil
            end
          end
        end
      end

      positions._ = true -- prevent JS array type
      SendNUIMessage({act="set_player_positions", positions=positions})
    end
  end)

More info: We have the server since 20 december 2019 and we’ve never had problem until July-August or so when random happend to different players, when there are more players on server the occurrence increase.

My crashes from Fivem\FiveM.app\crashes:

1ddbbd0f-6ef0-4095-8f7b-3f7ae57a11d1.dmp (593.6 KB)
1f0f2789-a8c3-48a4-8929-d7d353ae48a3.dmp (671.5 KB)
2bef3ec3-908d-4571-bcc9-d90e2d2e70f4.dmp (568.9 KB)
4a6fe370-28bb-4505-8309-6b4a8a4ecc3d.dmp (703.8 KB)
5b8583a2-571e-471e-a71f-39fa58d8cdd9.dmp (588.2 KB)
97f7074c-3f23-4093-b82c-3111c54aa12e.dmp (14.1 MB)
98d71215-554e-40f4-833b-a7a826da8fb2.dmp (546.2 KB)
315d2536-ceef-4baf-b6f4-2c46a980b39d.dmp (580.0 KB)
2971d49e-4890-45ca-b906-e0f94dfee3b6.dmp (16.4 MB)
cc4997de-8fe9-4a99-a1fc-5195ab465482.dmp (546.2 KB)
ce24ac3f-c07c-4da8-90a1-316ddfdd3080.dmp (548.4 KB)
e569baa5-9b55-4553-82a7-545b062ff24f.dmp (577.2 KB)
f7ceb8a0-3de7-432b-b6b3-5e8235b52752.dmp (14.1 MB)
f66c746e-846d-44b7-bcb3-db2a20d2a437.dmp (751.6 KB)
f99cb80e-0f73-434a-bc77-e30b938f439b.dmp (583.9 KB)
fcbfedbe-9da6-461e-8f4a-5f9ba8d16565.dmp (550.5 KB)

[nta]

The stack trace comes from the Chromium browser itself, I wouldn’t be surprised if some NUI stuff in vRP does some really weird audio stuff that crashes a lot for you.

[Fujy]

It helps me a lot! Before posting here, i was clueless! We had more than 20 resources that uses NUI and didn’t know exactly where should I debug! Now I have a hint and I will try to see what’s wrong with vrp2 audio.