[Free] oxmysql - asynchronous MySQL wrapper

There is possibility to use named placeholders from node-mysql2 docs.

You can use named placeholders for parameters by setting namedPlaceholders config value.
Named placeholders are converted to unnamed ? on the client

Currently this config is enabled by default and is defined by syntax :param instead of ghmatti @param.

So final query using named placeholders will look like:

exports.oxmysql:fetch('SELECT * FROM `jobgrades` WHERE job = :job LIMIT 1', { job = k }, ...

If you would like to preserve syntax of ghmatti @param you would need to code your own wrapper replacing : with @ in query and appending @ to keys in params table.

4 Likes

This is awesome !
Thank you for sharing and explaining !

Nice work!!! :ox: on top

1 Like

Just tested the script and i can only say
WOW
it just left mysql-async and ghmattimysql in the dust !!

I looped it for 10 iterations
Default ESX server with no extra resources
Waited 2 seconds between each iteration and…

it is ~52% faster than mysql-async (on average)
and ~55% faster than ghmattimysql (on average)

(maybe my math is wrong, but you can see the numbers below)

and for anybody that want to check the performance, this is my little messed up code, probably not perfect, but it get the job done

local iteration = 10

CreateThread(function()
    print('\n^2Benchmark started : ^0')
    local oxmysqlAverage = 0.0
    local ghmattimysql = 0.0
    local mysql_async = 0.0

    for i=1, iteration do
        print('^4--------Iteration #'..i..'--------^0')

        Wait(2000)

        local startTimeOxmysql = os.clock()
        local result = exports.oxmysql:fetchSync('SELECT * FROM `users`', {})
        local deltaTimeOxmysql = os.clock() - startTimeOxmysql
        print('oxmysql Elapsed Time :     ', deltaTimeOxmysql)
        oxmysqlAverage = oxmysqlAverage + deltaTimeOxmysql

        Wait(2000)

        local startTimeGhmattimysql = os.clock()
        local result = exports.ghmattimysql:executeSync("SELECT * FROM `users`", {})
        local deltaTimeGhmattimysql = os.clock() - startTimeGhmattimysql
        print('ghmattimysql Elapsed Time : ', deltaTimeGhmattimysql)
        ghmattimysql = ghmattimysql + deltaTimeGhmattimysql

        Wait(2000)

        local startTimeMysql = os.clock()
        local result = MySQL.Sync.fetchAll('SELECT * FROM `users`', {})
        local deltaTimeMysql = os.clock() - startTimeMysql
        print('mysql-async Elapsed Time : ', deltaTimeMysql)
        mysql_async = mysql_async + deltaTimeMysql

    end
    print('^0-----------------------------')
    print('^4Averages For '..iteration..' Iterations :^0')
    print('^2oxmysql : ', oxmysqlAverage / iteration)
    print('^3mysql-async : ', mysql_async / iteration)
    print('^1ghmattimysql : ', ghmattimysql / iteration)
end)

This is gonna be game changing for a lot of servers with high SQL usage !

5 Likes

This isn’t really a good benchmark as you’re comparing a resource that waits at least a tick to a resource that doesn’t. Waiting doesnt really make scripts slower. The biggest “performance” gain you’re showing is you don’t need to wait 50ms for a result - which should never really be an issue.

The real performance gain from my own testing is around ~25%, which is still very significant, but not 25x faster that your benchmark could lead one to belive.

Citiz

en.CreateThread(function()
    while true do
        Wait(0)

        -- for i = 1, 200 do
        --     MySQL.Async.fetchScalar('SELECT COUNT(identifier) FROM users')
        -- end
        -- --> ~15.8ms

        for i = 1, 200 do
            exports.oxmysql:scalar('SELECT COUNT(identifier) FROM users', {}, function(result)
            end)
        end -- --> ~12ms
    end
end)

fivem-mysql-async does single scalar fetch in around ~0.08ms
oxmysql does single scalar fetch in around ~0.06ms

Large performance increases could be achieved with adding indices for columns that you’re filtering by, especially when your tables grow larger in size.

I found a bug @Dunak when you want to connect to a database without password using connect string of mysql://root:@127.0.0.1/lsrp?charset=utf8mb4, where it just crashes with TypeError: Cannot read property '0' of undefined

I’d love if the library would support passwordless database access (local machine), but I just created another user with a password :slight_smile: If it’s not possible to support, validating the input and giving user-friendly error message would be second best solution. Just didnt know how connect strings work :slight_smile:

Overall, great release :slight_smile:

1 Like

It’s not a bug your string formatting is wrong.
It should be:
mysql://root@127.0.0.1/lsrp?charset=utf8mb4
and not:
mysql://root:@127.0.0.1/lsrp?charset=utf8mb4

Oh thanks, learned something new today :slight_smile: Will cross it out in original

2 Likes
[      script:oxmysql] Error: Can't create more than max_prepared_stmt_count statements (current value: 16382)

Any idea why this happens and what to do when it happens?

1 Like

This seems good, Can’t wait to test this out.

Great job.

That’s a lot of prepared statements to have gone through, but I could see it happening on a large server under certain circumstances. Is this during a benchmarking test, or actual use?

Are you using string-concatenation in a lot of queries (vs prepared statements), and what’s your player base like?

CreateThread(function()
	local query = 'SELECT * FROM users WHERE lastname = '
	for i=1, 16400 do
		exports.oxmysql:scalar(query..i, {}, function()
			print(i, 'async')
		end)
	end
end)

Each query sent through has its own entry in the cache, which makes future lookup of that statement quicker. SELECT * FROM users WHERE lastname = i

Unlike the following, which stores a single query SELECT * FROM users WHERE lastname = ?

CreateThread(function()
	local query = 'SELECT * FROM users WHERE lastname = ?'
	for i=1, 16400 do
		exports.oxmysql:scalar(query, {i}, function()
			print(i, 'async')
		end)
	end
end)

This is through actual use on a large server of around 170 players. It’s an RP server so there’s quite a lot of SQL queries that are put through. The difference is actually noticeably different.

I do use a lot of string concatenation, especially with my inventory. I should probably definitely look into fixing this.

Great release! I made a simple “wrapper” to make it work like old mysql-async.

MySQL.lua

MySQL = {
    Async = {},
    Sync = {},
}

local function safeParameters(params)
    local new_param = {}
    if nil == params then
        return {[''] = ''}
    end

    assert(type(params) == "table", "A table is expected")
    assert(params[1] == nil, "Parameters should not be an array, but a map (key / value pair) instead")

    if next(params) == nil then
        return {[''] = ''}
    end

    for k, v in pairs(params) do
        new_param[k:gsub("@", "")] = v
    end

    return new_param
end

---
-- Execute a query with no result required, sync version
--
-- @param query
-- @param params
--
-- @return int Number of rows updated
--
function MySQL.Sync.execute(query, params)
    assert(type(query) == "string", "The SQL Query must be a string")
    query = query:gsub("@", ":")

    local p = promise.new()
    exports['oxmysql']:execute(query, safeParameters(params), function(result)
        p:resolve(result)
    end)
    return Citizen.Await(p)
end
---
-- Execute a query and fetch all results in an sync way
--
-- @param query
-- @param params
--
-- @return table Query results
--
function MySQL.Sync.fetchAll(query, params)
    assert(type(query) == "string", "The SQL Query must be a string")
    query = query:gsub("@", ":")

    local p = promise.new()
    exports['oxmysql']:fetch(query, safeParameters(params), function(result)
        p:resolve(result)
    end)
    return Citizen.Await(p)
end

---
-- Execute a query and fetch the first column of the first row, sync version
-- Useful for count function by example
--
-- @param query
-- @param params
--
-- @return mixed Value of the first column in the first row
--
function MySQL.Sync.fetchScalar(query, params)
    assert(type(query) == "string", "The SQL Query must be a string")
    query = query:gsub("@", ":")

    local p = promise.new()
    exports['oxmysql']:scalar(query, safeParameters(params), function(result)
        p:resolve(result)
    end)
    return Citizen.Await(p)
end

---
-- Execute a query and retrieve the last id insert, sync version
--
-- @param query
-- @param params
--
-- @return mixed Value of the last insert id
--
function MySQL.Sync.insert(query, params)
    assert(type(query) == "string", "The SQL Query must be a string")
    query = query:gsub("@", ":")

    local p = promise.new()
    exports['oxmysql']:insert(query, safeParameters(params), function(result)
        p:resolve(result)
    end)
    return Citizen.Await(p)
end

---
-- Execute a List of querys and returns bool true when all are executed successfully
--
-- @param querys
-- @param params
--
-- @return bool if the transaction was successful
--
-- function MySQL.Sync.transaction(querys, params)
--     local res = 0
--     local finishedQuery = false
--     exports['mysql-async']:mysql_transaction(query, params, function (result)
--         res = result
--         finishedQuery = true
--     end)
--     repeat Citizen.Wait(0) until finishedQuery == true
--     return res
-- end

---
-- Execute a query with no result required, async version
--
-- @param query
-- @param params
-- @param func(int)
--
function MySQL.Async.execute(query, params, func)
    assert(type(query) == "string", "The SQL Query must be a string")
    query = query:gsub("@", ":")
    exports['oxmysql']:execute(query, safeParameters(params), func)
end

---
-- Execute a query and fetch all results in an async way
--
-- @param query
-- @param params
-- @param func(table)
--
function MySQL.Async.fetchAll(query, params, func)
    assert(type(query) == "string", "The SQL Query must be a string")
    query = query:gsub("@", ":")

    exports['oxmysql']:fetch(query, safeParameters(params), func)
end

---
-- Execute a query and fetch the first column of the first row, async version
-- Useful for count function by example
--
-- @param query
-- @param params
-- @param func(mixed)
--
function MySQL.Async.fetchScalar(query, params, func)
    assert(type(query) == "string", "The SQL Query must be a string")
    query = query:gsub("@", ":")

    exports['oxmysql']:fetch(query, safeParameters(params), func)
end

---
-- Execute a query and retrieve the last id insert, async version
--
-- @param query
-- @param params
-- @param func(string)
--
function MySQL.Async.insert(query, params, func)
    assert(type(query) == "string", "The SQL Query must be a string")
    query = query:gsub("@", ":")

    exports['oxmysql']:insert(query, safeParameters(params), func)
end

---
-- Execute a List of querys and returns bool true when all are executed successfully
--
-- @param querys
-- @param params
-- @param func(bool)
--
-- function MySQL.Async.transaction(querys, params, func)
--     return exports['mysql-async']:mysql_transaction(querys, params, func)
-- end

function MySQL.ready (callback)
    Citizen.CreateThread(function ()
        -- add some more error handling
        while GetResourceState('mysql-async') ~= 'started' do
            Citizen.Wait(0)
        end

        callback()
    end)
end

There is one problem, I didn’t know how to hook transactions. Maybe that code gonna help someone.

3 Likes

You’re just wasting memory by loading imported files into your resources, and they aren’t 1:1. It’s been close to 2 years since mysqljs was updated (disregarding dependencies), and there are some breaking changes between the old package and node-mysql2.

MySQL.ready is pointless, you don’t need to assert the queries as strings, and safeParameters is unnecessary since it’s already handled. As for transactions - they haven’t been aded yet since there’s some other minor issues we want to look into.

Yeah, if you’re able to stop using string concatenation then I recommend it. In the meantime you may want to increase the value of MAX_PREPARED_STMT_COUNT.

For your queries, find anything that is being sent as a string and send it as parameters instead - any value that can be set is fair game for ? or :var. There’s a lot of useless queries that select a field just to check if it exists before either inserting or updating - fix those if you can. If updates are infrequent then you can make do with REPLACE INTO, but for the best performance you want something like

exports.oxmysql:executeSync('INSERT INTO ox_inventory (name, data, owner) VALUES (:name, :data, :owner) ON DUPLICATE KEY UPDATE data = :data', {
	['name'] = inv.id,
	['data'] = inventory,
	['owner'] = inv.owner
})

Perhaps we could try catching the error and resetting the prepared statements as a fallback :thinking:

Dannnggg

Awesome, I’ve migrated to this fully, but there’s some errors like TypeError: Bind parameters must not contain undefined. To pass SQL NULL specify JS null which I know stem from the incorrect {} in the queries. Could there be a way to have all queries print out or maybe a way to capture this?

Dope ! thx

1 Like

I will add mysql_debug convar to next update.

Node-mysql2 doesn´t accept undefined params and lua nil is converting to js undefined. So until I fix that you cannot send nil param from lua.

1 Like

New update 1.1.0

  • Experimental backward compatibility for ghmatti/mysql-async like syntax
exports.oxmysql.single('SELECT * FROM users WHERE identifier = @identifier, {
  ['@identifier'] = xPlayer.identifier
}
  • Added support for mysql_debug true/false convar - this will log both query and parameters with execution time into console.

  • Handle connection error immediately on resource start-up

4 Likes

is that possible to add debug in the file and one sql ready export to compatibility mysql-async?