[Help] Network Thread Hitches spamming in console -> All Players time out. ETL file attached

JordanZABOT4411 · 2023-08-02T20:10:34.930Z

Hello peeps, I’m having an issue where almost every night, if not every other night the console will start spamming network thread hitch and all players will time out. Sometimes a few sync thread hitches in there, but 99% Network thread hitch. I have personally gone over every single resource in the server (minus the escrowed ones we have) and I have also run neteventlog with OBS recording and did not see any large events coming over. I have also done wireshark packet captures too.

I’m hoping someone would be kind enough to analyze this ETL file that I captured during the network thread hitches and see if anything stands out? I’ve been trying to fix this for two month.

https://cdn.discordapp.com/attachments/1100261422216249364/1136131666956533884/2023-08-01_19-59-07_admin.etl

Only other thing I can think of is we have been with our current DDoS Provider for almost 3 years, we have contacted them to see if we are being attacked, and they have told us numerous times now that they do not believe it’s an attack, and they said it could potentially be a CPU issue. We’re also using a proxy + caching proxy through the same DDoS provider, again that has been fine forever.

I have also used the profiler and there is not a single hitch visible on those results, have done it countless times.

JordanZABOT4411 · 2023-08-03T17:17:29.281Z

It would be really nice to get some help here. If there’s information that I’ve left out that would help, let me know and I can provide it.

JordanZABOT4411 · 2023-08-04T15:57:25.013Z

Cool. Thanks for the help FiveM

AvarianKnight · 2023-08-04T16:11:40.576Z

Guess you missed the part where people who do support are volunteers. Fix your entitlement.

JordanZABOT4411 · 2023-08-04T17:14:52.587Z

I get it, you’re volunteers, I too help people myself all the time, discord, forums, etc. It’s extremely frustrating that I have asked for help in multiple venues and this problem is so infuriating, because it seems that there’s no way to debug the issue myself, instead I have to rely on the forums, discord, cfx tickets and hope that someone has even a hint to what my problem is.

AvarianKnight · 2023-08-04T18:38:21.851Z

Likely because there’s only a small minority who know how to read the etl traces properly.

Please note I am not one of these people and what I’m going to say is an assumption.

When your FXServer spikes you have a long tick in the V8 runtime (node) so you might try looking there.

This would also make sense as to why it doesn’t show in the profiler as v8 doesn’t get profiled iirc

JordanZABOT4411 · 2023-08-04T18:56:03.985Z

AvarianKnight:

s v8 doesn’t get profiled iirc

Thank you, I will look into it. If anyone else has any ideas, please feel free.

JordanZABOT4411 · 2023-08-05T02:41:43.166Z

Actaully, funny enough, I am looking at any possible issues with 3dme, which is one of your scripts actually. Specifically, we changed it a bit. Do you think any of these changes would cause an issue? I’m not 100% on js stuff, but this is what I did:

setImmediate(() => {
	if (GetConvarInt('3dme_enableProximity', 0) && !GetConvarInt('voice_syncData', 0)) {
		console.log('[^1ERROR^7] You have `^13dme_enableProximity^7` enabled but dont have `^1voice_syncData^7` enabled in pma-voice!')
	}
})

and also this bit we changed:

RegisterCommand('me', (source, args) => {
	const checkmsg = `/me ${args.join(' ')}`
	const message = `* ${args.join(' ')} *`
	exports.DiscordChatLogs.createLog(source, GetPlayerName(source), ["/me"], "/me " + message)
	// check if the message contains any blacklisted words
	if (!exports.DiscordChatRoles.msrpBlacklistCheck(source, checkmsg)) {
		// make sure they're not on cooldown (they could try spamming the command)
		if (!meCooldown[source] || meCooldown[source] < GetGameTimer()) {
			meCooldown[source] = GetConvarInt('3dme_cooldown', 1500) + GetGameTimer()

			let proximity

			// enable support for pma-voice, if the player has toggled it.
			if (GetConvarInt('3dme_enableProximity', 0) && GetConvarInt('voice_syncData', 0)) {
				proximity = Player(source).state.proximity.distance * (GetConvar('voice_useNativeAudio', 'false') === 'true' && 3 || 1)
				if (GetConvarInt('3dme_enableProximity', 0) && Math.floor(proximity) > GetConvarInt('3dme_maxRange', 25)) return emit('3dme:voiceRangeExploit', source)

			} else {
				proximity = GetConvarInt('3dme_maxRange', 25)
			}
			
			// useful for logging
			emit('3dme:sent3dme', source, message)

			const plyCoords = GetEntityCoords(GetPlayerPed(source))
			const players = getPlayers()
			players.forEach((plySrc) => {
				const tgtCoords = GetEntityCoords(GetPlayerPed(plySrc))
				if (getDistance(plyCoords, tgtCoords) < proximity) {
					emitNet('3dme:show3dme', plySrc, source, message)
				}
			})
		}
	} else {
		// useful for moderation/anticheat
		emit('3dme:blacklistedWord', source, message)
	}
})

AvarianKnight · 2023-08-05T21:07:28.479Z

Hey sorry, read this but forgot to reply, i doubt it would cause any issues

JordanZABOT4411 · 2023-08-08T02:02:16.609Z

Is it possible that we have a lack of CPU power and that eventually leads to the problem? We’ve had this server box for quite a while, but is that a possibility? On one of the ETL traces, I saw that one core was being utilized more than others, but the CPU never maxed out or anything like that.

JordanZABOT4411 · 2023-09-16T05:02:45.888Z

Anyone got any ideas at all?