SetVehicleDoorsLocked RPC causing client crashes (full dump included)

Client

Using canary? Multiple, both Yes and No
Windows version: 19043.1110, Multiple
System specifications: Unknown, Multiple

Server

Operating system: Windows Server 2019
Artifact version: 4304
IP address: 72.5.195.226
System specifications: Xeon E3-1270, 32GB DDR3 RAM

Incident

Summary:
Multiple players - from various geographic regions on both mainline and canary - are experiencing 166074F - california-white-muppet - crashes. Based on analyzing the dumps provided by users, it appears to be caused by the SetVehicleDoorsLocked RPC native.

Steps to reproduce:
We are unable to reproduce this in smaller scale local/staging tests, it appears with 30-50 active players, performing an unknown number of vehicle locks/unlocks which use the SetVehicleDoorsLocked RPC native.

Dump Files:
These are all full dumps ranging from 10-20 GB each.

Example Screenshot:
unknown

Right - I suspect an easy fix would be to wrap the native invocations from ClientRPC in a SEH wrapper just like ScRT invocations get, as game natives may crash/bail out for a variety of reasons.

The underlying issue may be more or less nefarious, but a good start would be to SEHify this anyway.

1 Like

Tracking the first task on GH as I’m experimenting with public issues:

2 Likes

probably fixed ?

1 Like

@nta Unfortunately this crash still seems to be appearing on canary client version 4377. A full crash dump is below. Let me know if you need more of them.

https://drive.google.com/file/d/1uyX3msPh7G_cb_aZNLG4J2jitUMxjPKj/view?usp=sharing

1 Like

Well, going to need a repro then.

Dump seems eerily similar to the other reported issue: ClientRPC native invocation is lacking SEH · Issue #798 · citizenfx/fivem · GitHub.

Hopefully this is some random race condition somewhere and sanitizing object->GetGameObject() in ObjectToEntity would be sufficient. The first deference of the GetGameObject pointer would happen at +0x18 (rage::fwExtensionList::Get), which correlates to this dump and stack trace on github.

I can look into this, I suppose.

(Worst case scenario, SEH the entire native execution scope rather than just the call handler?)

Changing the conditional in ObjectToEntity to if (object && object->GetGameObject()) is probably sufficient given how ‘conditional’ callbacks work.

However, if ClientRPC is to be expanded upon, should we consider improving the failure logic in fivem/ClientRPC.cpp at master · citizenfx/fivem · GitHub ? For example, after X units (seconds?) of failed conditions, send a failure message/error-code to the server that can be logged, re-broadcasted, etc.