Issue with nginx proxy setup

Hello, I tried looking for this issue already but wasn’t very successful so I have no other choice but to make this thread.

I followed Proxy Setup - Cfx.re Docs through completely and added the server to my nginx load balancer.

Now, when I start the server and I go to play.example.com, the cfx link does not resolve but if I go directly to the servers IP:30120 (skipping nginx load balancer), I get redirected to the cfx link but only in http, https gives me a self signed error for citzenfx.tls.donottrust or something like that.

If I go to play.example.com/info.json or any other json, it works.
Also, my srv/files folder for cache is empty on the nginx load balancer.
As expected after a few minutes of running the server I get
Server list query returned an error: System.Net.Http.HttpRequestException: Response status code does not indicate success: 521 (). <- System.Exception: Could not query via https://play.example.com/ - check if your sv_listingHostOverride is correct

My host override
set sv_listingHostOverride "play.example.com"

my proxy vhost

proxy_cache_path /srv/cache levels=1:2 keys_zone=assets:48m max_size=10g ;
upstream backend {
        server INTERNAL.IP:30120;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name play.example.com;

    ssl_certificate /etc/nginx/ssl-certs/name.pem;
    ssl_certificate_key /etc/nginx/ssl-certs/name.key;

    access_log /var/log/nginx/reverse-access.log;
    error_log /var/log/nginx/reverse-error.log;

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_pass_request_headers on;
        proxy_http_version 1.1;
        proxy_pass http://backend;
    }
    location /files/ {
        proxy_pass http://backend$request_uri;
        add_header X-Cache-Status $upstream_cache_status;
        proxy_cache_lock on;
        proxy_cache assets;
        proxy_cache_valid 1y;
        proxy_cache_key $request_uri$is_args$args;
        proxy_cache_revalidate on;
        proxy_cache_min_uses 1;

        }
}

nginx.conf

stream {

    upstream backend {
        server INTERNAL.IP:30120;
    }
    server {
                listen 30120;
                proxy_pass backend;
        }
        server {
                listen 30120 udp reuseport;
                proxy_pass backend;
        }
}
http {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ssl_protocols TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        gzip on;

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

My endpoint is set to the internal IP of the server the FiveM server is running on
set sv_endpoints "INTERNAL.IP:30120"

My nginx is set up to redirect http to https traffic, I’m using my CloudFlare Origin certs for the proxy vhost as everything on my infrastructure is running behind CloudFlare.

I tried disabling my auto https rule in nginx and adding a :80 listener in the proxy config but it didn’t help at all either. Also tried chaning proxy_pass-es to https instead of http, same result.

I can start a connection to the server using connect https://play.example.com/ but it hangs on Downloading resource manifest and then aborts connection. I believe this is due to my server cache folder on nginx not populating?

If anyone could share some insight on what I might be doing wrong I’d greatly appreciate it.
Thanks.

1 Like

I managed to get the server working.
Only thing now is that my users.cfx.re link is not resolving and it’s saying This server is not available from this endpoint. but my play.example.com link works and my txadmin reverse proxy works.

What I did:

  1. New nginx VM with its own external IP, seperated from the www nginx
  2. Adjusted .conf files according to the proxy guide
  3. Added reverse proxy for txadmin

:warning: Make sure you have your own TLS/SSL certs! I generate my own ones for the load balancers using certbot ONLY for game servers, my web servers use CF.

nginx.conf – add this above html

stream {
        upstream backend {
                server INTERNAL.IP:30120;
        }
        upstream txadmin {
                server INTERNAL.IP:40120;
        }
        server {
                listen 30120;
                proxy_pass backend;
        }
        server {
                listen 30120 udp reuseport;
                proxy_pass backend;
        }
        server {
                listen 40120;
                proxy_pass txadmin;
        }
}

txadmin.conf

upstream txadmin{
     server INTERNAL.IP:40120;
}

server {
    if ($host = admin.example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        listen 80;
        listen [::]:80;
        server_name admin.example.com;
        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $remote_addr;
                proxy_pass_request_headers on;
                proxy_http_version 1.1;
                proxy_pass http://txadmin;
        }


}
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name admin.example.com;
    ssl_certificate /etc/letsencrypt/live/xxxxx/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/xxxxx/privkey.pem; # managed by Certbot
    location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $remote_addr;
                proxy_pass_request_headers on;
                proxy_http_version 1.1;
                proxy_pass http://txadmin;
    }

}

server.conf

upstream backend {
    server INTERNAL.IP:30120;
}

proxy_cache_path /srv/cache levels=1:2 keys_zone=assets:48m max_size=20g inactive=2h;

server {
    if ($host = play.example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        listen 80;
        listen [::]:80;
        server_name play.example.com;
        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $remote_addr;
                proxy_pass_request_headers on;
                proxy_http_version 1.1;
                proxy_pass http://backend;
        }
    location /files/ {
        proxy_pass http://backend$request_uri;
        add_header X-Cache-Status $upstream_cache_status;
        proxy_cache_lock on;
        proxy_cache assets;
        proxy_cache_valid 1y;
        proxy_cache_key $request_uri$is_args$args;
        proxy_cache_revalidate on;
        proxy_cache_min_uses 1;
    }
}
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name play.example.com;
    ssl_certificate /etc/letsencrypt/live/xxxxx/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/xxxxx/privkey.pem; # managed by Certbot
    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        # required to pass auth headers correctly
        proxy_pass_request_headers on;
        # required to not make deferrals close the connection instantly
        proxy_http_version 1.1;
        proxy_pass http://backend;
    }
    # extra block for a caching proxy
    location /files/ {
        proxy_pass http://backend$request_uri;
        add_header X-Cache-Status $upstream_cache_status;
        proxy_cache_lock on;
        proxy_cache assets;
        proxy_cache_valid 1y;
        proxy_cache_key $request_uri$is_args$args;
        proxy_cache_revalidate on;
        proxy_cache_min_uses 1;
    }
}

server.cfg – top of the config

endpoint_add_tcp "0.0.0.0:30120"
endpoint_add_udp "0.0.0.0:30120"
set sv_forceIndirectListing true
set sv_listingHostOverride "play.example.com"
set sv_proxyIPRanges "xx.xx.xx.1/32" #Internal subnet range where your load balancer is
set sv_endpoints "LOAD.BALANCER.IP:30120"
#set sv_listingIpOverride "LOAD.BALANCER.IP"
#LOAD.BALANCER.IP = external ip of your nginx load balancer

Hope this helps anyone that runs into the same issues and if anyone knows why my endpoint for cfx.re is not resolving I’d be very happy to listen!

1 Like

Okay, well, now I’m having more issues.
When I add a new resource fxmanifest.lua, the server does not recognise it but it recoginses any other file change/script addition/etc. Encoding is set to UTF-8 on the Lua file.
It seems that my nginx proxy only worked the initial time I set up, it’s not populating /srv/cache/ or /var/cache/nginx/proxy_temp anymore at all, or maybe I’m looking at the wrong places?
image
image

I’ve read on the forums here that when I add a new resource I need to clear the cache and restart the server, so I did that, still didn’t help.

Creating a new resource (i added a client.lua after this)

fx_version 'cerulean'
game 'gta5'

description 'Wowee this is a test!'
version '1.0.0'


client_scripts {
	'client.lua'
}

lua54 'yes'

The files show up on server FTP.
image

Server gives this message when I refresh ( Could not open resource metadata file - no such file.):
image

But if I was to edit a existing script and add anything to it, the changes would go through without any issues.
I was monitoring wireshark and all my connections are always made to the nginx proxy server, on server connection, on downloading resources and in-game. In no situation does my Real IP go out there nor the files get pulled directly from Real IP cache to the player, so where are the cached resources loading from?

Okay well it didn’t take me long to figure out what I was doing wrong with the resource metadata not loading.

If you are using VS Code and the SFTP addon, make sure to do sudo chmod -R 777 newresource or do this. It will load then.

This helped me the most:

sudo chmod -R o+rwx /path/to/FiveM
sudo chmod -R g+rwx /path/to/FiveM

Final update for this post!

The reason why I was getting this message:
Server list query returned an error: System.Net.Http.HttpRequestException: Response status code does not indicate success: 521 (). <- System.Exception: Could not query via https://play.example.com/ - check if your sv_listingHostOverride is correct
Is because I blocked all requests coming to the server, after properly setting up the firewall settings everything works correctly.

No more issues with proxy, no more issues with the play. subdomain.

In the end, I ended up migrating the FiveM server to a Windows box instead of running it on Linux. You can set up nginx as a service on the Windows box if you want to, but I recommend running a separate nginx server that proxies to your game server.

I’m still using these settings and everything is working.

1 Like

What firewall rules did u use ? Cuz i’ve this issues. I’ve allowed TCP/UDP traffic from any to my ip-server on port 30120.