Issue with Nginx reverse proxy and external voice server

I ran into some issue while trying to connect my players to an external voice server (regular fx server) via some reverse proxy running nginx. Only 16 players were able to connect to the external voice server per proxy (3).
The server was running with enough slot (2048) and without adhesive so it’s not a slots/license issue.

I’m using the default nginx setup from the fivem doc:

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
worker_rlimit_nofile 40000;

events {
        worker_connections 39000;
}

stream {
    log_format basic '$remote_addr [$time_local] '
                     '$protocol $status $bytes_sent $bytes_received '
                     '$session_time';
    access_log off;
    error_log /etc/nginx/errors;
    server {
         listen PORT;
         proxy_pass VOICE_IP:PORT;
    }

    server {
        listen PORT udp reuseport;
        proxy_pass VOICE_IP:PORT;
    }
}

http {
    # Basic Settings
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    # SSL Settings
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
    ssl_prefer_server_ciphers on;

    # Logging Settings
    access_log off;
    error_log off;

    # Gzip Settings
    gzip on;

    # Virtual Host Configs
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

When I use the same nginx configuration to proxy player connections to an fx server running an ‘internal’ voice server, I don’t have this issue.

Looking at nginx error logs, every new clients are just timed out. I didn’t find any revelant logs on the external voice server with mumble_logLevel set to 2.

2022/05/30 01:01:27 [error] 15152#15152: *285759 upstream timed out (110: Connection timed out) while connecting to upstream, client: X.X.X.X, server: 0.0.0.0:XXXX, upstream: "X.X.X.X:XXXX", bytes from/to client:0/0, bytes from/to upstream:0/0

Is there any connection limitation per source ip for the external voice server ? I wasn’t able to forward player’s real ip with this setup so this may be the issue.