I don't always get a frame

MilkDrink · 2022-03-04T16:46:00.542Z

Hi guys, how can I find out what is causing these skips in clients?

The reason I started this here is because I think these are server add-ons or some streamed addon stuff might be the culprit…
I just can’t figure out exactly what it is.
I’d like to find out how I can determine what’s causing me to not get frames for x amount of time, thanks for your help!

nta · 2022-03-04T17:47:36.422Z

‘Dropped frames’ wouldn’t make sense in this context, these are rather long frames, and neither ‘server add-ons’ or ‘streamed content’ would cause this, since streamed content is loaded entirely asynchronously. That being said, there’s some weird behavior in GPU drivers at times that makes some assets with large amounts of textures or vertices/polys take a while in the GPU driver (commonly the case for NVIDIA), blocking the game from rendering anyway.

This is also often seen on systems with low amounts of VRAM but with ‘extended texture budget’ set higher than the default with a lot of software running in the background, so the game will allocate a large amount of cached data → GPU page faults will occur which would also block.

Recording and uploading a .etl file from such slowdowns would make it easier to help, or if you know how to analyze CPU waits in ETW traces yourself (note, this is not the same as the common resource time guide) that could also work.

MilkDrink · 2022-03-07T12:41:29.307Z

Thanks for the info!

I sent you the trace in a private message.

I tried to investigate, but I found out that this phenomenon does not only occur when there are many vehicles in a certain area, but “enough” when there are only 15-20 people in a small place. Most of the time we use basic cloths with few addons and those with little texture and few polygon numbers.

I tried spawning several cars with a large number of polygons in one place and going back from one end of the track to the other to fill in the models, however, this didn’t cause the “long frame” just less frames.

I experience these “long frames” when there are network players in a particular area I enter.

warboyo · 2022-03-08T11:16:33.359Z

Your stutters, is it when approaching populated areas mainly places like legion and surrounding areas there?

I seem to have a similar issue and have even taken out the custom cars at one point to no avail. I kinda get the impression its when you enter that populated area or at least an area with players and it creates the physical player, this is were i see the stutter/hang for a second or 3, i say see…i was just watching the console to see what activity was occurring and when the list of creating the physical players appeared (about 25 of them) i encountered the stutter

MilkDrink · 2022-03-08T11:40:29.669Z

It is not limited to areas, but rather to places where there are usually several players in a small place, or several large groups in a given area.

You should also try to record and submit an ETW trace because more measurements may hopefully better pinpoint the cause of the problem.

Random ASCII - tech blog of Bruce Dawson – 14 Apr 15

UIforETW – Windows Performance Made Easier

Event Tracing for Windows (ETW) aka xperf is an amazing tool for investigating the performance of Windows machines – I’ve blogged about it many times and it’s helped me find some amazin…

warboyo · 2022-03-08T13:45:20.550Z

Yes, the more populated areas or hub points if you will is what i was referring to.

There was a mini event going on last night at Yellow Jack on my server, about 35 people were in there and the stutter occurred on approach to the general location.

We have removed cars, removed clothing, mlo’s, scripts you name it we removed it and the issue still stands. We have took out a lot of high poly cars (over 300k polys) and are now looking to keep them at lowest <200k, we have lowered the textures as well and also done the same with clothing.

Will run ETW tonight after work when the server is full and get a recording

nta · 2022-03-09T09:21:00.624Z

MilkDrink:

I sent you the trace in a private message.

I just happened to have time to check it and take a quick look, and it seems to confirm my theory above that this is a GPU driver hang/being slow, possibly as a result of exceeding the amount of VRAM available on the system. AMD sadly doesn’t provide GPU driver debug symbols either, so I can’t really deduce much more than that some memory management call is blocking on work performed in a background thread, which is a bit less direct behavior than NVIDIA does here, and even more difficult to detect via code.

Does the user who recorded this trace have ‘Extended Texture Budget’ set to the default of 0, or is it set to any higher value? If it’s set too high, and models that are being loaded in (as an effect of entering a new area, in this case) are big, they may be causing a lot of memory management work for the GPU, leading to a lot of swapping between system memory and GPU memory.

Since this is a GPU with 4 GB of video memory, any higher setting than the default may be problematic (default of no extended budget amounts to a limit of 3 GB for the game, which should be fine assuming any background processes don’t have some excessive allocations. Electron/CEF apps like Steam or Discord are often a culprit for that however).

If ‘Extended Texture Budget’ is set above the default to ‘fix’ an issue with the world failing to load, I think it’s time to actually reduce the memory size of commonly-used assets on the server, as then there is no other way this will fit on a 4 GB GPU. This setting was added as a compatibility toggle to fall back to old (broken) behavior people relied on before where the limit was ignored if using ‘Normal’ texture quality, it is still incorrect to configure such this way and server content should be tested to work fine with the default setting.

PlutoIsAPlanet · 2022-03-09T13:07:39.317Z

We’ve also noticed a large amount of these “stutters” when having large amounts of other players entering the players scope, we’ve also tried with custom content disabled and still get issues. Our script time is also very low (0.50ms) as we have switched our code base to use game events rather than loops so I don’t think its a script issue.

I have tried playing with these config options to see if there’s any change but no help.

set onesync_distanceCulling true
set onesync_distanceCullVehicles true
#set onesync_forceMigration true
set onesync_enableInfinity true
#set onesync_enableBeyond true
set onesync_workaround763185 true
set onesync_population true

A lot of these stutter reports seem to come mostly from AMD (CPU + GPU) users, including myself. I’ve had a look at a ETW trace and can’t see anything note worthy. There was a JS “tick” that was quite frequent but that’s been removed, with no help. I’m more than happy to PM a trace.

Would tweaking SetPlayerCullingRadius help at all? The only other work around I can see happening is putting these populated zones into a routing bucket.

nta · 2022-03-09T13:51:52.016Z

nta:

Does the user who recorded this trace have ‘Extended Texture Budget’ set to the default of 0, or is it set to any higher value?

To you, too.

Posting a ‘me too!’ report with a lot of irrelevant information (variables that have been long since removed, wild guesses that contradict the posts above, wishes of unrelated ‘workarounds’) while not answering any more important questions is very much counterproductive.

If this continues, I’ll go and leave you folks be and do something less irritating with my time than trying to pry info from people on these forums.

PlutoIsAPlanet:

I’ve had a look at a ETW trace and can’t see anything note worthy. There was a JS “tick” that was quite frequent but that’s been removed, with no help.

As to this, I’ll repeat again:

nta:

or if you know how to analyze CPU waits in ETW traces yourself (note, this is not the same as the common resource time guide)

The ‘resource time’ guide has no relation to actual wait analysis other than both being stored in the same file and being viewable using the same tool.

MilkDrink · 2022-03-09T16:29:01.755Z

Thank you for the time you have spent on this topic!
I looked into the things you described and took a closer look at my supplemental content and unfortunately I found that there are many that only have a single lod level. So I guess these models cause problems when loading and later on they cause less fps from a distance because they don’t have the low level.
Hopefully if these are fixed/removed these “stutters” will be reduced.

warboyo · 2022-03-10T21:22:44.399Z

I did have my texture budget increased however after reading this thread last night i lowered it to 0 and my stutters have dropped considerably, i also got others to try and they to state the same.

We had the budget up a bit as admittingly we had clothing that were high in poly and texture size but we have optimised all the clothing and have done a lot of work on the cars as well, currently going through them to check over the LOD’s.

nta · 2022-03-11T18:51:36.484Z

@MilkDrink I’m guessing you marked my post as a solution since it indeed happened to be the budget setting?

MilkDrink · 2022-03-11T21:59:23.768Z

What you wrote overall and this includes minimizing the budget setting is partly the solution to avoid major “stutters”, although I don’t use it anymore, only when there is a major event.
That is-if you follow the basic game limits and use normal models (lod levels, reduced texture) you will mitigate the stuttering as described above.

However, the main problem after prolonged testing is a base game bug.
We use only freemode peds on our server, however creating female and male freemode peds takes more time than other models, I guess because these two models are the ones that can be customized with great detail. I guess it’s again a game limit and not prepared to create 20-30 peds in one frame.

I have tried with other non FIVEM clients and there I have the same problem with CREATE_PED native.
If always call create_ped native in a new thread or not try to create all 30 peds in one frame, it doesn’t cause any or very minimal stuttering.
If you are moving you can load up to 20-30 ped at a time into a frame in the game pool so it will start creating them in that frame and cause stuttering.

Create a server side with 30 freemode ped’s in one place and when you enter the area at least for me it loads into one frame and I get the long frame.
The same can be achieved on client side if you create the peds in a for loop, however if you start them in a new thread or create only one in each frame then I don’t have a long frame.

Unfortunately I can’t contribute (PR) to solving this phenomenon, I only have theories and ideas, so I decided that I don’t want to waste your time and to keep this topic from going in another direction, so I tried to close it.

nta · 2022-03-12T08:44:45.597Z

MilkDrink:

However, the main problem after prolonged testing is a base game bug.
We use only freemode peds on our server, however creating female and male freemode peds takes more time than other models, I guess because these two models are the ones that can be customized with great detail. I guess it’s again a game limit and not prepared to create 20-30 peds in one frame.

Right. A quick mitigation for this would be throttling the game applying CPlayerAppearanceDataNode to not apply this for ‘new’ nodes too often.

manzarek · 2022-03-17T18:43:43.014Z

It would be nice to implement the mitigation.
When cruising around in a high population server you will hit those large frames pretty often.

nta · 2022-03-17T18:47:19.274Z

manzarek:

It would be nice to implement the mitigation.

Having an easy enough method (server/time/place?) to test would be helpful, especially as this would be hard to ‘fake’ with non-player peds. Without any means for validation this is unlikely to happen any time soon.

manzarek · 2022-03-17T19:08:23.297Z

This can be reproduced with non player peds

RegisterCommand("mp_f_freemode_01", function(source, args, rawCommand)
    local model = `mp_f_freemode_01`
    local coords = GetEntityCoords(GetPlayerPed(source))
    for i=1, 60 do
          CreatePed(4, model,coords, 3374176, false, true)
        end
end)

Or you meant it will be hard to develop the mitigation without real players?

nta · 2022-03-17T20:12:07.694Z

manzarek:

This can be reproduced with non player peds

Reproduced doesn’t mean the suggested mitigation would apply or be testable that way. Non-player peds are using different data nodes for such.

MilkDrink · 2022-03-18T09:47:49.875Z

Okay, but how can we move forward towards a solution?

At the moment we have a growing server, that’s why we switched to infinity, but a lot of people are complaining about this phenomenon, especially players with low performance computers, some of them have left our server and prefer to play on a small server with 20-30 players because they can move around with a car, not just stand still.

To solve this, I think we can get “real” players together on a test server or even provide a test server.

Or please tell us what is needed to solve this problem.

nta · 2022-03-18T09:58:01.788Z

MilkDrink:

Okay, but how can we move forward towards a solution?

Not sure at this time. I’m still not actively working on the project, @frenzy-renegade - you got any clue?

One thing that confuses me is that if this thing is so ‘easy to reproduce’, why aren’t there way more people mentioning this as being a problem? Might there be some other environmental factor involved as well, or are people somehow really forgetful on trying to report any concerns they have?

(similarly, the CREATE_PED repro scenario may be triggering a different slow code path compared to actual players, so that’s even weirder)

EDIT: I was about to actually try to look into the repro when I received an unwanted and explicitly unasked for DM from a person reading this topic adding some additional stressors and triggers to this concern.

All in all, I’m not going to be able to look into it today as now I’ve got massive pain all over my body from usual stress symptoms, and I don’t know if I’ll bother any later day as the particular ‘trigger’ mentioned added yet another emotional burden to this report that makes me highly unlikely to touch it again.

~~I also don’t know if we have any other people that can look into this class of issue working for the project, so this might end up being a ‘scratch your own itch’-style issue.~~ It’s been delegated, I suspect.

Again, the first paragraph in this message very much applies, and do not DM me trying to get your question, issue, or other concern looked into any faster. It’ll not help anyone, and at best will only get me to be annoyed, at worst will get me to ignore your issue entirely as it may be tied to emotional triggers (like what was sent in DMs here).

EDIT2: I’ve amended my forum profile information to clarify my current role.