Cfx.re Monthly Update - March 2022 edition

Cfx.re Announcements
1

Hi everyone!

Time really is flying by - we’re already half way into March! While the last month hasn’t seen anything terribly exciting that we’re ready to talk about yet, we still want to share a few things with you!

Beware of malicious software

Before we get into the rest of the update, we want to share some words around safety with you.

As of late we’ve seen a surge in targeted abuse, spreading malicious software by claiming it will ‘fix FiveM/RedM issues’.

This is not legitimate software, and you should never run .exe files from people you don’t trust. Cfx.re staff will never ask you to run untrusted executables, and we rarely, if ever, contact you in a Discord DM without prior notice. If we do contact you, or if you’re not sure if someone is impersonating us, it is easy to verify that we are in fact real staff by checking our roles in the Cfx.re Project Hub on Discord.

If you receive a message from anyone who might be impersonating Cfx.re staff, block the user, report the content to Discord, and do not reply to them.

To re-iterate: ignore and report DMs asking you to run untrusted software, and stick to the public channels on Discord when getting or providing help.

Stay safe.

Usage of leaked resources

Every so often, we get a request from people asking us why their server was “blocked for stolen scripts”, where we in fact haven’t blocked their server.

Indeed, it turns out that certain server assets have been circulating that contain a backdoor, often used by the supposed author of these assets to ‘destroy’ a server, corrupting its scripts and adding a message saying the server was blocked.

This seems to mainly circle around a set of assets sold from unauthorized stores, then “cracked” and repackaged in “server bundles” sold on other stores purporting to be a “FiveM script marketplace/store” or similar.

If you’re looking to set up a preconfigured server, we’d suggest you stay away from these stores or other “super script bundles”, and instead look for the recipe options in txAdmin, or download/buy compatible scripts yourself from the #development:releases section.

FiveM

Most of our work on FiveM for the past month has been focused on fixing various issues within the game. Give the changelog a peek!

  • Client security improvements and abuse mitigation
    • FiveM: Refactor ‘PointerArgumentSafety’ to be part of the build process, and marks ‘Any*’ arguments as an explicit no-go.
    • Prevent abuse of non-msgpack arguments passed to ResultAsObject and a direct InvokeNative call being able to return an arbitrary string as a result.
    • Reset result state if it ‘looks like’ game code didn’t actually touch the result.
    • Prevent DllImport from non-platform modules in Mono.
  • Replace the deprecated frontend.chrome-dev.tools URL for profiler.
  • Fix a few C# scripting bugs, such as enum not being correctly serialized, and a server-side NullReferenceException issue.
  • Fix V8 (JS) stack overflows crashing the game instead of raising an error.
  • Fix game build switching for second-client (CL2) mode
  • Fix LocalPlayer state bag breaking when a client was rejoining a server after being disconnected, but getting assigned the same player ID.
  • Anticheat changes, including the following:
    • Fixed a compatibility issue with Windows 7 (closing instantly on launch)
    • FiveM: Reimplemented mitigation of ‘SLOD’ peds so that it works on builds other than 1604.
    • Note that not all anticheat changes are included in the changelog.
  • Rockstar integration fixes:
    • Most cases of ‘Timed out waiting for ROS/MTL’, mainly on systems with dual/quadcore CPUs should be gone now.
    • We no longer delete the cached Rockstar account when services are down, so no more unexpected Rockstar login dialogs.
  • Some fixes in crash reporting:
    • FiveM: Fixed a regression introduced in the last release where ‘pool full’ errors instead showed up as a fmt::v8::format_error exception.
    • Support ‘symbolication’ (friendly names, used in server quit messages and advanced error dialogs) for Windows 7/8.1.
    • Fixed some cases where the crash dialog itself would crash due to a timing issue.
  • FiveM: Asset streaming reliability improvements:
    • Reduce download priority for assets that are no longer relevant, such as when moving rapidly across the map.
    • Some fixes for hanging when disconnecting from a server, before the main menu gets a chance to show: now we rely less on the streaming engine to succeed for a frame or two.
  • FiveM: Prevent custom key binds breaking with a lot of duplicate RegisterKeyMapping calls
  • FiveM: Fix a rare FxDK map compilation issue, and fix key input in FxDK.
  • FiveM: A few regression fixes for game build 2545:
    • Loading story mode when having last used 2545 no longer crashes.
    • ‘NetworkEntityDamage’ game events now contain actual data.
  • Misc fixes:
    • Mitigations for a relatively rare game hang in Mumble client code.
    • When using the optimized ‘OAL’ mode for the Lua runtime, vector results now work correctly.
    • FxDK: No more duplicate entries in the project explorer when renaming an entry.

Documentation

Earlier this year we onboarded a new team member who is helping us clean up our documentation efforts. We have already pushed some great new documentation, and more is coming! The repository is now receiving more love and care, and we are aiming to have all pull requests addressed by the end of the month.

Anti-cheat

We’ve recently pushed out some changes to the anti-cheat that should help us get rid of even more cheaters and make the games more enjoyable for everyone.

Said changes led to more than 5,000 users of paid cheats being banned from our platform for 300 days. :tada:

We mainly rely on community submissions to get our hands on working cheats, paid or free. (To preserve our anonymity as well as avoiding any sort of account flagging that would prevent us from being able to reverse-engineer those cheats)

If you want to provide us with a known working and undetected cheat, please send an email to abuse@fivem.net with the following information:

  • Archive containing the cheat executables
  • Name of the cheat
  • Where did you get it (Discord server, website, Youtube video, …)
  • NEVER USED license or account + instructions on how to activate the license (for paid cheats)

RedM

While RedM is more of a niche game, we’re happy to say that we’re observing a slow but steady growth! RedM is now nearing 200,000 unique players and peaked at over 3000 concurrent players over 450 servers :tada:

Have any ideas on how we could improve the RedM experience? Have cool any projects you want to show off? Reach out and let us know!

Website update

We’ve updated the RedM site to align it with our other web-facing content - you’ll notice it’s now much more similar to the FiveM site in design. We’ve kept it simple for the time being as we’re working on expanding both of the core websites. Make sure to check it out!

Platform Stability

Downtime for any game is painful for everyone involved - both us at Cfx.re, and of course, the players. While we have already been able to make great improvements, we’re still not done. Some weeks ago we kicked our projects to improve stability into higher gear, and we are going to continue improving it for the foreseeable future.

Establishing more resilient infrastructure

February saw us onboarding a new team member to help improve the core infrastructure that runs our services, including a full refresh of the cluster powering it all. Currently, we are testing different providers to figure out which one we would like to go with, and we are hoping to conclude this testing soon. After we determine the primary provider we will start working out a timeline to share with all of you. Hopefully, this should be ready in the next monthly update.

Another big part of increasing the resiliency of our infrastructure is simply better monitoring and alerting. We are making tweaks to some of our alerts to be able to catch some issues faster, and to be able to catch issues we haven’t seen previously.

In the end, we are aiming to have an infrastructure that can more reliably take a hit and still keep going without anyone noticing. Our most glaring issue right now is DNS, which often leads to a ‘trail’ effect of what would otherwise be an outage no one would have noticed. Anyone on heavily cached DNS resolvers, often those ran by consumer ISPs, suffer for longer than others.

Outages

We’ve had a number of outages already this year, and we understand that this is incredibly frustrating for everyone, including us. To touch more on what was mentioned in the previous point, our most glaring point of failure is currently DNS.

Currently, our infrastructure has its ingress through Cloudflare which is working great for us - and one of the standard ‘big name’ ingresses in the Kubernetes space behind it. However, the flimsy part here is that our infrastructure is currently not homogeneous, is spread across multiple providers in an attempt to mitigate single-homed downtime, and has grown organically with the platform. With this in mind, we currently use a component named external-dns to handle our DNS - and it has somewhat strange handling of DNS records for Cloudflare.

Whenever one of our ingress controllers suffers a minor outage (be it a few seconds or just a few minutes), the DNS is removed from Cloudflare and re-added. If we are unlucky, this change is cached by ISPs around the world and you get the 6: couldn't resolve host: x.fivem.net that some have faced in recent times. This fixes itself automatically within an hour or so, but is still not ideal.

We’re working on better solutions that we will deploy with our new production infrastructure in the future, and we are hoping to maybe even deploy it on the current cluster if we can find a safe way to do it.

We hear you. Outages are incredibly annoying, and we want everyone to be able to enjoy the experiences on the platform. More updates pertaining to outages and the infrastructure will be shared as time passes.

txAdmin

Last month we released the v4.13 update, here are some highlights from the changelog:

  • Core: Improved web performance (over 30%!) and resiliency against DoS;
  • Core: Added the reason for the playerDropped events in server log;
  • Menu: Added the txAdmin-menuAnnounceNotiPos convar to be able to control the announcement position;
  • And many bug fixes: particle effects on wrong players, invincibility during NoClip, not being able to revoke bans (broken IDs) and more!
72 Likes
2

Thanks for the info and hard work!

9 Likes
3

This is awesome!

2 Likes
4

Great work! Good luck on improving reliability!

3 Likes
5

Good comms and the docs are finally being prioritized

1 Like
6

Thanks for the detailed information! Truly appreciate all the hard work you guys do on the backend!!

1 Like
8

wow RedM was mentioned me reading :grinning:

me after reading :face_with_raised_eyebrow: :thinking:

why even mention RedM? a web page? half the buttons on the top that leads you to fivem stuff…
where are the docs just like fivem has?
even in discord where are the channels for releases? just like fivem has, a showcase a, Bazar channel?
how long was RedM launched? there’s little to no progress or is it a lack of interest?
there are not many contributing yes, and yes what i mention is nothing to worry/care about but damnnnn show some interest at least, little by little we get there ofc, but there’s not even a little being shown or given.

4 Likes
9

Sadly I agree with you, but at least this is a step in the right direction. I do feel like RedM is massively neglected which has significantly stunted its growth time and time again. Gotta hand it to the servers that keep going despite the regular drawbacks though! Fortunately for most of the users, they’ll at least see a download button now. :smiley:

1 Like
10

yeah after how many years late? this is not a step in the right direction in my point of view. if it takes years to make a web page god knows how long we will have other things working.

1 Like
11

I agree and disagree. For example, the community at QBCore has been a major driving force in improving and expanding RedM and its available resources and content. Seeing as that’s mostly open source, it has lead other developers, such as my team and I, to start actively working on the project too. The website at least was a step in the right direction, as before, I’d constantly get the assumption from our users that it was a landing page and the project wasn’t actually launched or running.

12

that has nothing to do with cfx.

1 Like
13

You are doing an awesome job as well as the whole community does!

I’m one of the developers of the german LooneyLoops Server and we are one of the biggest, if not the biggest, RedM Server. We would really love to see OneSync evolving and getting as good as it is with FiveM. We are aiming for roughly 500 players on our server at prime time, but OneSync can’t handle it right now for RedM. At the moment we are close to 300 players at prime time and growing.

Thank you for your work and effort!

14

But it does, as it shows that CFX doesn’t really have to do much more to attract the population for the project. So the website change is a step in the right direction.

1 Like
15

“My admins keep abusing the permissions I myself granted to them and they use bad words on their messages”
“please remove txadmin”

I will not waste more time with this subject, as you seem to not understand that txAdmin is used by over 15 thousand servers making it deeply unwise to apply every single one of your “feature requests”.
I’ll just leave here my actual reply (which you decided to just ignore) to your “concerns” where I explained my decisions:

5 Likes
16

Well you need not worry, as that number is no doubt going to start dropping if things like what’s happening to us and several other communities continues to happen.

Also, seeing as Douth is technically a representative of yours, his answer matters too.

Too bad you won’t waste time on features that protect the community from abuse. Your response proves my initial point entirely, so thank you for that.

1 Like
17

He’s replying now, so I look forward to his response too. Again, another official representative of txAdmin.

1 Like
18

Howdy,

This response is in to your " txAdmin" section. FXServer comes pre-installed with txAdmin because it’s approved by Cfx members due to the easy accessibility, great web design, prebuilt features, and the general amount it has contributed to the FiveM community. We’ve tried to be direct with you
[chrislenga] in the past, but to no avail. Like you stated in your initial message it’s the server manager or owner’s responsibility to handle permission-based information processing (r; “They cited that its my problem for not vetting staff well enough, which sure, I guess they’re true to an extent, but my team can only do so much.”). We, txAdmin, are not responsible for the wellbeing of FXServers, their admins or what permission based system you/they use. No matter what ‘program’ or ‘web manager’ you use, someone will always be there to ruin the fun.

Now with this being said, it’s not that the txAdmin ‘team’ hasn’t taken significant steps towards preventing or completely stopping administrative-based abuse. For example, more permission nodes were added so specific members allowed into the panel were restricted to X, Y or even Z. We also provide different ‘panel access’ locations to prevent users from seeing console, etc - like most quality monitoring web panels.

Other Steps Towards Anti-Abuse

  • Prevented abuse by adding PTFX to NoClip.
  • Since txAdmin version v1 we have added admin-action logging abilities.
  • Created a transparent system where every admin can see the server log.
  • Added a advanced permission system, which progressively got more permissions added.

Like Tab said we provide a open source service to over 15,000 servers at any given time. This issue has been relatively isolated to your community, in the capacity of being such a ‘big deal’. I’d like to mention once again that txAdmin is a open source project. You, along with your own development team, are at free will to create pull request for these much “needed features”.

Refrence; GitHub Issue Link (#588)

Douth

19

Well I must say, I like this response more than your “its your fault not ours” response back on Github.

We as in the server owners are forced to have the monitor resource installed with the artifacts. Sure, we can delete it, and we have a python script to remove the monitor resource entirely on artifact updates, which we have shared with a few other server owners. Something that is actively being requested more and more as it becomes widely known about as many we’ve talked to are confused about how not to run txAdmin with their server.

You said the following which I will quote.

We, txAdmin, are not responsible for the wellbeing of FXServers, their admins or what permission based system you/they use. No matter what ‘program’ or ‘web manager’ you use, someone will always be there to ruin the fun.

So you provide a tool and resource that’s included in everyone’s server, and the average user doesn’t know how to disable it, so they use it out of ease because its there, but if something goes wrong, its their fault? Is there any license agreements upon downloading the artifacts or prior to setting up txAdmin during installation of artifacts that state this?

You also mention we could easily create a pull request with features, yet, there’s some pull requests that are months old, so that’s a statement of deflection on your part. In fact, looking at the closed ones, it almost seems like there’s more closed and rejected, than there are closed and accepted. Strange.

I’ll quote you on another interesting response.

Like Tab said we provide a open source service to over 15,000 servers at any given time . This issue has been relatively isolated to your community, in the capacity of being such a ‘big deal’. I’d like to mention once again that txAdmin is a open source project. You, along with your own development team, are at free will to create pull request for these much “needed features”.

As someone that owns three separate registered businesses in the US, I find your “take no responsibility” stance rather troubling, especially since your tool is provided to every single server artifact if I’m not mistaken.

Since you bring up things like responsibility, during my next meeting with legal, perhaps I’ll have them do a little research on this situation and get their feedback on the subject. Hopefully by then since we meet once a month, I’ll have some data to go along with that conversation.

I’m especially curious over the liability aspect when it comes to brand damage and damages in general. There’s been some pretty huge cases regarding things like brand damage caused by external factors, specifically services used to carry out situations such as sabotage.

I am certainly delighted that both of you took time to reply to this, and I very much look forward to seeing what the future holds regarding this.

20

I was previously going to respond to your reply, but the bottom mentions legal concerns. Due to this factor, I will unfortunately be ending this conversation here. The severity of these claims means I will need to CC Cfx members and txAdmin maintainers. Have a good one.

CC:

21

I happened to stumble across this whole message exchange and I’m really not in the mood for this so I’ll probably be too hostile, but here - see these as personal opinions at best as I try to stay out of any actual executive action in these regards:

What would you consider ‘proper enforcement’ here?

  • Not giving a grace period: not ‘proper’, people start community raids and attacks overloading both our support staff and volunteer community moderators.
  • Giving a grace period: not ‘proper’ because of you considering it ‘not properly enforced’, also much higher conversion rates over time.

If your concern is something else, however, it’d be helpful to explicitly state that.

Patreon has a 7% fee for new accounts, plus some payment processor/conversion fees. This is more than the Tebex fee (which makes up less than half of the 15% advertised), and more than the payment processor fees, as well.

Other than that, this specific argument is a bit of ‘if nobody gets caught, why should I not abuse either?’, which is generally a non-argument: the answer should be because not abusing is the right thing to do in any case, in any context.

Migrating existing subscriptions across payment processors is a near-impossible operation, and doing so just to placate some people who are going to use it as a ‘justification’ to not read what they can and can not do is wrong.

We’re planning on rewriting the ToS page to be more readable and show examples of what is/isn’t accepted sometime in the near future, however. Think something like the Xbox Community Standards in readability.

The particular ‘project’ mentioned seems to just rely on brand-awareness of ‘look, this is a proper defense!’ to convince ill-informed people that ‘their’ project is ‘better’.

The open (EOS) version of EAC doesn’t actually have any game-specific detections, all it does is some handle hardening and slightly more enforced HWID bans without requiring game developers to implement these.

Looking at cheating forums, there’s actually more free public cheats for the project you’re naming than for FiveM.

Given it’s game-specific, and we use some novel methods for HWID bans - one of the main features of these ‘kernel-mode anticheats’ - that aren’t generally supported by popular ‘spoofer’ drivers, I’d say pretty much on par.

These files are actually not downloaded from any servers we control and are rather from a P2P environment, in part due to bandwidth costs, in part due to these being more complete sets of game files than FiveM’s (which are delta patches), which may be questionable legally.

It is confusing that the UI looks the same as it does on FiveM, however, yes, and it may lead to the expectation that this is downloaded from some shared servers.

If you don’t launch in ‘monitor mode’ (i.e. if passing a +exec argument), ‘txAdmin’ does not run at all.

However, I personally do agree the way Tabarra et al. are handling the relationship between the main project and their ‘sub-component’ has a lot of room for improvement, since we also often have to correct some perceived odd choices performed by them.

I’m surprised a paragraph like the following wasn’t included:

In addition to that, we have been working on top fixes for RedM issues (including ones related to OneSync), but due to current world events affecting our main RedM developer, not much has happened for the past month.

Also, RedM still runs on a shared codebase with other Cfx.re platform modifications, so many improvements in FiveM also apply there.

This is the one of the two parts about ‘txAdmin’ here that I’ll reply to, but I feel it having its own ‘official representatives’ is somewhat… problematic.

The other is that a missing access control feature can not remotely be seen as anyone’s ‘fault’, and blaming anyone here is counterproductive: both blaming @chrislenga for ‘giving someone permissions to kick people with any reason’ - people may violate trust unpredictably and take a complete 180, and blaming the txAdmin maintainers for not instantly prioritizing a feature to mitigate any abusive user input, do not help this discussion at all.

I don’t know if any other server owners had issues with staff writing inappropriate text in ‘ban reason’ fields, but if not, I can see why this feature wasn’t prioritized, and I can also see why @chrislenga wants to prevent this kind of thing from happening again, as that is a typical human response to sudden immoral behavior on the part of any individual, especially one that was considered trustworthy before.

The escalation about ‘I’ll talk to my lawyers!’ below is rather silly and even more so counterproductive. If you want to take legal action against anyone, why not do so against your abusive ex-staff member? :stuck_out_tongue:

7 Likes