Cache proxy failing sporadicly on client (corrupt download?)

FiveM Server Development Server Discussion
1

Hello team! Happy new year :fireworks:

I’m trying to setup nginx (cache proxy) and use the fileserver option using the Cookbook guide. :man_cook:
It’s working great with awesome loading times, except that sporadicly assets is failing to download as showed on this picture:

image
image1913×215 102 KB

  • It repeats over and over, due to FiveM is trying to download the asset when in the area.
  • It’s random which asset it happens for
  • If I clear the nginx cache it loads instantly whenever this errors occurs.
  • I’ve tried both with and without SSL, on both ports 80, 443
  • I’ve tried both with and without http2

Can someone point me in a direction?

Here’s is my nginx config:

proxy_cache_path /srv/cache levels=1:2 keys_zone=assets:48m max_size=10g ;
log_format asset '$remote_addr - [$time_local] "$request" $status $body_bytes_sent $upstream_cache_status';

upstream backend {
    server SERVER_IP:SERVER_PORT;
}

server {
    #listen 80 http2;
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name filecache.socialrp.dk;

    #ssl on;
    ssl_certificate /var/www/certs/proxy.pem;
    ssl_certificate_key /var/www/certs/proxy.key;


    access_log /var/log/nginx/filecache-access.log;
    error_log /var/log/nginx/filecache-error.log;

    location /files/ {
        proxy_pass http://backend$request_uri;
        add_header X-Cache-Status $upstream_cache_status;
        proxy_cache_lock on;
        proxy_cache assets;
        proxy_cache_valid 1y;
        proxy_cache_key $request_uri$is_args$args;
        proxy_cache_revalidate on;
        proxy_cache_min_uses 1;
    }

}

Server vars:

fileserver_add ".*" "https://WEBSITE_URL_DOT_COM/files"
adhesive_cdnKey "some_degenerate_passphrAzZZe"
1 Like
2

Seems as if something is caching with a weird key. Anyone else experienced this?

3

We’re also experienced this, It’s seems to be a random assets (and random client too).

It’s appear to have this behavior when we’re updated to FXServer build 5053 (between 4527 - 5053. My apologies that can’t point any exact artifact, We’re jumping to recommend only.)

4

We are currently on FX 5128 though :thinking:

6

Can I do something to provide debug-info? It’s pretty fast/easy to provoke the error.

7

No, a repro would be the only actual way. ‘Debug info’ would require knowing it’ll happen before it happens.

8

Well, then I guess you have the repro setup above.

Repro recipe would be as follows:

  • Setup nginx (have tried both 1.14.0 and 1.16.0) - Debian distro
  • Setup fileserver as shown above
  • Move around to download assets until error occurs, if error doesnt occor delete ‘server-cache-priv’ on client to re-download assets to provoke the error
  • Clear nginx cache and resource assets will download just fine
9

Have you tested this ‘repro setup’ yourself from first principles? (i.e. not having any set of ‘move around’-type assets oneself, or a server with a lot of players to make this race condition or whatever more likely?)

As I don’t have either so I don’t ‘have the repro setup above’.

1 Like
10

It’s tested with one, and two players - and we can repro it almost instantly.

The ‘moving around part’ is just to make the client request different asset downloads, and provoke it faster.

11

That still doesn’t help anyone who is not ‘you’ as nobody else has your set of assets on their server.

As above, I don’t have any set of assets to even test this with.

12

If it’s any help, pre-loading stream assets in loading screen never fails, it is always in the game.

EDIT; The above statement is false, after further testing.
When I was solo-testing this everything seemed fine, when we we’re two joining simoustanely the error happened already in loading-screen while pre-loading assets. This could maybe hint to a race condition? :man_shrugging:

13

… oh you’re missing proxy_cache_lock aren’t you? sigh

14

image
image783×329 8.24 KB

Can you elaborate? :thinking:

15

I even tried switching from Debian to Ubuntu on nginx version 1.18.0, for the lulz, and set it up again. No juice.

16

Also, why are you still ignoring the part where I do not have the ‘assets’ you are referencing in your reproduction steps?

17

I’m not ignoring anything, but I can not share the creators work here.
Do you have a suggestion to how I can provide info then?

Setting up a server for you to join with the assets on and cache proxy enabled obviously - will that be any help?

18

No, that wouldn’t help, since the issue isn’t going to be something induced client side.

I guess you don’t want a fix then, if you’re all like ‘lol i can’t share the creator’s work!!!’, even if without a fix that ‘creator’s work’ is useless.

You could try figuring out if you can induce it with any set of random assets that aren’t some ‘creator’s work’ and link those…?

19

Do you have anything in your NGINX logs? I’ve not had this issue with my setup.

20

Nope, nothing in terms of warnings/errors.

21

I just wanted to report back about this.

The problem we encountered seemed to be network related, as we’ve have moved our FiveM box away from OVH. Though the cache/file-server itself is still at OVH, and now we have no problems, and everything is working perfectly.

This topic can be closed and marked as resolved.