Kashacters - major design flaw fix

FiveM Resource Development & Modding Discussion
1

I’ve been getting a FiveM server up and running starting with zero experience with Lua and having written only a car spawner in c# prior. It’s been a steep learning curve and we’ve been plagued with issues, primarily with fixing other people’s code. Some of the resources we’ve used are tedious but work and elements of others are just down right garbage. I accept that we are using resources written by a community of “developers” who have no experience and this is their way into coding, but seriously it amazes me what is left and is never corrected. I thought I’d throw some experience into the community to try to make a positive difference. Ok. It trying to offend anyone I’m just exhausted.

Any RP system is going to have a backend database to store player character progress. There’ll be a base table with an identifier linking all child tables together. Each resource slaps on some new functionality and you sometimes need to run sql that comes with it to create a new table. We had to do quite a bit of work to get kashacters running, even though the one we finally integrated was close to being perfect, was still missing some things. Anyway we ran the query to get the list of tables with identifier and owner columns and stuck that in the Lua table at the top of the server Lua file.I personally hate that step and found after a couple of months of dev that we hadn’t added more recent table names to that list so some info was being shared across characters. We have been fighting all sorts of performance issues, doing things like avoid repetition to be DRY and checking our thread loops timeouts aren’t killing things, moved some ESX code from server to main removeing recursive calls using set timeout calls plus a few other things. One thing I’d never really gotten around to was performance optimising the database. Now the average dude who has minimal coding experience knows very little about database theory but I’ve found a major architectural flaw in kashacters. That f’ing list of tables that is set so that when you switch characters all related assets are switched out…!!! Groan… So, let’s run a separate update statements PER table that has the steam/license code. I have 15 of these inclusive of the users table so to switch to another character a total of 30 individual update statements need to be executed, 15 to update license: to Char1: plus another 15 to change from Char2: to license:, PLUS one more to update the lastcharcter.charid, 31 individual update statements! The most expensive thing you can do is running multiple sql statements and 5 is insane in a single server callback. Heck with trewhud we found a server call to pull back stats that were already in memory for three values hitched our server to death even without the overhead of a database server to connect to.

So, what I’ve done with my system is eliminated the kashacters list of identifier tables and updated the functions to NOT loop and just run a single update statement for the identifier only on the users table. I have also assigned actual relationships between tables. MySQL is a “relational” database management system. Basic course / tutorials into database management cover how to set a relationship between tables where you pick a primary key on your root table and use that as a foreign key in child tables. A foreign key relationship is known as a “constraint” and has options on what to do if the key on the root table fails, known as UPDATE and DELETE conditions where you get to choose what to do with the foreign key in the child table. If you set them to “CASCADE” then the DBMS knows to automatically change the value in the child tables to the new value in the root table, if the root is deleted then the child tables are automatically deleted by the DBMS. Having this built into the database schema is the most efficient way to achieve these updates and you get it for free when you run a single update statement setting the identifier column if the relationship is set. Yet kashacters runs 15…:roll_eyes:

I understand that with kashacters they’ve provided a simple way to achieve this and it appears to be a good idea given that every ESX schema out their is different as each table comes bundled with resources and they don’t know what your schema already looks like. But it’s fundamentally wrong. They should have provided the information schema query to get a list of tables with identifier and owner columns, the should have pointed people to articles on how to correctly setup referential integrity with cascading update and deletes. Lua is a scripting language and binds to other libraries to interoperate with systems and is very slow compared with MySQL making decisions to do something internally. Everyone running kashacters, please do what I’ve done and remove another atrocious inefficiency from your systems.replace the Lua updates loops with relationships from the users.identifier column.

If you want to go a step further, you can reduce the number of calls from 3 (update license to char1, another update from char2 to license and the last character update) to 1. You can create a “stored procedure” on the database where you put the 3 update statements. The sproc will receive as arguments the license ( after the : ) and number of the tile clicked in the kashacters character selector UI.

The sproc will get the lastcharacter char internally (actually could be included as a sub query on an insert statement making it a set operation and even more efficient) then do the switching around server side using MySQL scripting code to setup the 2 update statements. This means from Lua you can simply make a single database call. Code run dbms server side is always extremely efficient, the biggest lag point being the connection to the database and sending queries through. A single connection is the most ideal situation you could achieve.

Whilst I’ve learned a lot about FiveM development I’m incredibly discouraged at the state of the open source modding community code. Being a developer shouldn’t be about hacking crap together as quickly as possible and throwing it out there just because it works then drive everyone else around the bend. I know people mean well but it’s so frustrating for people consuming this mess.

I hope this helps at least one server out there to clean up their system. By going through the exercise of stepping through every table it became an auditing process and I found a lot is mismatches with related column lengths and found places I could add indexes, such as vehicle plates to help add some more efficiencies.

1 Like
2

I completely agree with that. However, I think that their decision was made since most ESX resources are already flawed, and knowing that the average FiveM server owner barely knows what’s a database. And to be honest, since these updates are happening only once, when a player switches characters, and not continuously in some loop, I don’t think the performance impact is huge. I’d like to see some benchmark here, to compare the original kashacters and your improved version.

3

You’ll soon realize that more than the majority of resources released on here are flawed in more way than 1. And from what I’ve seen from these paid resources, not much of a difference there either.

When I first started working on my project over 3 years ago, it was even worse then. Imagine not knowing how to program at all and then trying to learn off of peoples broken examples. It can be fucking rough. But either way, with posts like yours, does overall help the quality of the community. However, I’d recommend mentioning this fix for people in the actual thread so people utilizing that resource and see the improvements. So, anyone actually interested in improving their resources can do so.

4

My concern is that if you do a restart then have 50 people coming on and switching characters, there’s likely to be some hitches. I agree though it’s not a massive performance boost, but the way it’s done just makes my skin crawl. Take this as an exmple, we just had two people in a single household using different computers, with different rockstar accounts and different steam accounts. With cascading I could have just done a single update at the end wha I was transferring characters around, but alas. In doing the migration to use proper relationships I found if I left joined from child tables to users, there were null users.identifier records on a few tables. The indexes I’ve added and now the indexing on the fks will help queery performance overall as well.
One other thing to note, I had tables where the identifier had different character collation and even the db engine was set differently. All cleaned up now :slight_smile:

1 Like
5

I have some more work to do. My test server is now migrated with a snapshot of the prod system synced into it. I’m finding it all works perfectly… ahem, well on the database level it all works perfectly. But the issue I’m facing now is that when I switch characters I see the inventory from the character I was just logged as. The reason is that esx_addoninventory preloads everything from the store tables, then handles esx:playerLoaded which pulls from the in memory dictionary (lua table) based on the identifier value. Problem is that whilst the first character has been switched to Char1:xxxx, internally addon inventory hasn’t been triggered to refresh that. And for that matter another performance gain would be to limit the call in addon inventory server main.lua in the MySQL.ready(…) handler to limit what’s loaded with WHERE identifier LIKE 'license:%' (or 'steam:%') so there’s less in memory.
I actually successfully bypassed the addoninventory calls altogether where triggers 4 events with this:

SELECT aad.account_name 'name', aad.money 'count', NULL 'label'
FROM `users` u
    INNER JOIN addon_account_data aad
    ON u.identifier = aad.owner
WHERE identifier = @identifier1
    AND aad.account_name IN ( 'property_money', 'property_black_money' )
    AND aad.money > 0

UNION

SELECT aii.name, aii.count, i.label
FROM `users` u
    INNER JOIN addon_inventory_items aii
    ON u.identifier = aii.owner
    INNER JOIN items i
    ON aii.name = i.name
WHERE identifier = @identifier2
    AND aii.count > 0

We’re not using standard weapons as dutch players inventory hud uses items by weapon name so we didn’t need the 4th call to datastore. But what you get with that query is a single query to get three “accounts” worth of information in one go. I just massage the result into the same shape as what was already returning. So, the best bit about that is you can re-imburse items to properties and the player doesn’t have to restart. Downside is that it’s still all out of whack with the addon inventory system, that’s all buried into the ESX player object.

Anyway, once I’ve got a full working solution I’ll document the process with code snippets.

1 Like
6

With all due respect, and honesty here - I didn’t read your whole post but I don’t disagree that there are flawed resources. Have you considered PR’ing or forking and creating your own version? I like this train of though, identify a problem, explain it in detail (it seems) then work on a fix as a group / collaborative effort.

1 Like
7

I tried forking and PRing back to Dutch Players inventory hud but that messed some things up. Whilst they are “approving” PRs they’re not confirming integration works, as it messed up something. Maturity in processes seem to be very weak. My day job is working on tooling for “devops modernisation” work to support 20+ teams of developers working on many projects and adhering to coding standards and aligning with auditing requirements by independent bodies, so … I get my standards are shot well near the stars. I just left it and moved on as I had a server to stand up with a community of players banging on our doors. Now it’s fairly stable I’m fixing these finer details and trying to apply good standards and replace all the guts that are just plain wrong. I have form beliefs on how things should be (all devs are) but understand that my thought process is different to everyone else’s. My posts where I ahve something to offer will be aimed at helping people to think about problems rather getting people to do it the way I do verbatim.

My original goal after being jaded playing on an RP server was to write my own RP framework for NFive, which is doing what ESX 2 is aiming for with framework plugins rather than other resources integrated through FiveM events, and written in C#. I wouldn’t even consider sticking with ESX in the long run. My plan is to go with nice language typing with C# and other glorious APIs like LINQ and Entity Framework. Will be more of an experiment to see if I can get a C# based RP framework to run lean with low ms times in the resmon. The lack of isolated automated unit testing is really tedious as well which makes dev for the game so damn slow.

I’ve also been thinking a bit about create isolated subsystems that are run OUT of the FiveM process. FiveM is not great with multi core load spreading, so it would be good to offload some work to other system services with netwrok connections to do heavy lifting where possible. Dunno if it will work yet, would need to be able to access ports on other applications and might need to import some LUA pakcages to do it. I think it’s possible… anyone?

I will stick with some LUA resources that are independent. I think LUA is a great scripting language on many levels but has been overused like crazy for game modding.

8

I think the majority of our expectations are also up there with you but unfortunately… doesn’t work that way as you can see.

This seems over my head to be honest but be cautious as to how those systems interact with FiveM as they may become detected by the anticheat in thinking that what you’re doing is malicious. Don’t quote me on that as I’m not 100% sure as to the exact detection methods used.

9

It would be my FiveM resource calling out to the separate processes, not the systems reaching into the resource. I hope it will work. I guess another option is that if I use c# then I can just link I could use http client or maybe grpc (if that’s available in .net 4.5) or similar.

10

I’ve done a similar thing but for other purposes and it worked for me.

1 Like