Random heap corruption (regression?) on servers

Semi-related (but not really… was looking through the uses of HttpResponse).

Should ResourceHttpComponent be ending or closing the HttpResponse if there is no handler associated with a resource?

In theory, this would be fine once all references vanish… but this is indeed a little bit weird-looking.

There was another thread running about this in parallel, this has been closed and is redirected here now:

(yes, the other thread is technically older but this one had more recent activity and is the one that usually ends up found instead)

We found out that we are getting rate limits everytime before crashes. Also i think it’s related to the internet, cuz why it would hand rdp before crash in one server, why we don’t have logs before crash, why there is always network hitch etc. Here are some screenshots of console before crashes




Since an earlier investigation as recent as 2023-03-08T17:39:00Z resulted in claims that the heap corruption exists ‘as low as 5914’, and theoretically ‘5848’ was still fine, there’s a new theory about what might be going on here:

As another experiment, I’ve just pushed tweak(net/http-server): flag to remove EASTL usage · citizenfx/fivem@6fa9f9f · GitHub (build 6314), which should at least behave differently here (and might also finally show the original corruption in cases with a memory debugger attached, so if the issue still occurs there I’ll probably throw an ASan build out there again).


tl;dr

(tl;dr: try again with 6314, if it still fails do upload a full dump and if it’s the same failure and it needs more info still there’ll be an ASan build to try with again too that’ll hopefully catch it unlike last time)

This seems to be a print from this ‘SekulBanSyste’ resource. What does this resource do that makes it send requests that ‘get rate limited’, and when do these get tripped initially?

That behavior also seems to add up with some sort of deliberate attack again, by the way.

We are now on 5855 and have crashes all time

It is discord logs when someone is joining

After having like 30 crashes in a row after starts.

We have disabled every single http request, making only exeption for TXAdmin. We are running still,

I will keep it disabled for 2 days or more to confirm that problem is gone (we had have at least 5 crashes daily)

function PerformHttpRequest(url, cb, method, data, headers, options)
    if GetCurrentResourceName() == 'monitor' then
        local followLocation = true
        
        if options and options.followLocation ~= nil then
            followLocation = options.followLocation
        end
    
        local t = {
            url = url,
            method = method or 'GET',
            data = data or '',
            headers = headers or {},
            followLocation = followLocation
        }

        local id = PerformHttpRequestInternalEx(t)

        if id ~= -1 then
            httpDispatch[id] = cb
        else
            cb(0, nil, {}, 'Failure handling HTTP request')
        end
    else
        cb(0, nil, {}, 'Failure handling HTTP request')
    end
end

Paste this in scheduler to disable every httprequest

Third time’s the charm, christ.

Since apparently the D word is a very bad word (even though it was already used in this thread) I am going to censor this url.

https://verybadword.com/developers/docs/topics/rate-limits#exceeding-a-rate-limit-example-exceeded-resource-rate-limit-response

1 Like

What else we can use to have logs?

Although true, that is not relevant for this thread.
So let’s keep the subject centered on the crashes and solutions, shall we?!

:thinking: Not sure how you believe it’s irrelevant when commenting on a question already put forth. We have a bad logging system being triggered in every resource at high enough frequencies to lead to rate limiting. They may be repeating those http requests until it goes through, and at high enough player count / joining players there might be too many concurrent requests, leading to server hitches.

Players with poor connections may lose connection, or others may try to reconnect to resolve “lag”; more players in queue leads to more requests and other functions that may cause unexpected load on the system. Maybe I’m reaching, but the current assumption is

That behavior also seems to add up with some sort of deliberate attack again, by the way.

I’ve seen plenty of hitches caused by joining and connecting events; it could be deliberate or just a mistake triggering an underlying issue. Of course that’s just based on the limited information (screenshots and some rate limits) and looking at it as a scripting issue.

1 Like

Neither the HTTP requests nor the resource code were the subject of the question here, except to ascertain what these logs would be triggered by. Since it’s already answered as ‘players connecting’, that part is solved already.

What does this resource do that makes it send requests that ‘get rate limited’

“Players connecting” doesn’t seem like a what, rather the answer to “when do these get tripped”.

The what being “sending an excessive number of requests that exceed the limit of a Discord webhook”, hence my message about not using it as a logging service and linking to their documentation. However I’ll just stay out of it as my input is always looked down on.

1 Like

This isn’t the intent at all, and if anything I do comes across as such I apologize for that.

However, since the actual issue at hand here involves users being confused already, adding more noise that isn’t necessarily related to the issue only complicates solving the issue for everyone involved.


Having no HTTP requests didn’t help for a long…
I will post crash dumps

After changing to 6314 i have 30 crashes in a row, crash after crash.

Here are all the dumps generated:
https://drive.google.com/drive/folders/1rDSw0hELgppn_XrwDiKecgJXMsT4pczx?usp=sharing

Build 6318 contains another attempt at fixing/mitigating this issue: fix(net/http-server): some Http2Server thread affinity risks · citizenfx/fivem@bbec02f · GitHub

4 Likes

Will update tonight and let you know if it fixes it on my side.