First of all, blame yourself. I’m not saying this to make you feel bad, but always try to put the blame on yourself first.
Why? Because 90% of issues regarding low performance is due to badly written scripts. Make use of resmon 1 - type it in your F8 console - and monitor the latency of all your scripts. Is something higher than 0.20ms throughout your gameplay(that’s quite lenient if I must say so)? Note it down and optimize the script. You can also use profiler. [How-to] Use the profiler command to identify problematic scripts
Got a lot of custom vehicles and/or maps? Without unloading the models if they aren’t needed anymore you can get issues.
If you want a comparison…I run a 32 slot non-OneSync server that is nearly always full and players have little to no FPS drops compared to GTAO/empty FiveM server. Running on a measly $20 VPS with 2vCPU (only 50% usage) and 8GB RAM.