SSL handshake failed with OpenSSL error

FiveM Resource Development Discussion
1

Hey, I am looking for a solution to the networking problems that I have been getting recently, namely Server list query returned an error error logs in the server console:
Error

Most players can’t connect to the server, with a few exceptions, and the custom connection link cannot be reached on HTTPS, only on HTTP (even then it only returns a default Apache page).

In Cloudflare, I have an A DNS record pointed to the VPS’ IPv4 address (partially redacted):

A record
A record1265×404 15.5 KB

As well as an SRV record as follows:
(Removed)

There are some other DNS records for email forwarding and the website, which I can send if needed to solve the issue.

The current SSL/TLS encryption is set to Full. I have tried to add a new origin certificate to the server and use Apache, but with no luck. Then I reverted to nginx which used port 80 before the issues, but that didn’t resolve the issues either. Looking at it, it was never even seemed to have been configured, so I don’t know how it worked in the first place.

The 30120 port seems to be properly setup:
Ports

The server config that is related is as follows (as I mentioned, the subdomain for connections doesn’t work on HTTPS, so adding sv_listingHostOverride won’t work):

endpoint_add_tcp "0.0.0.0:30120"
endpoint_add_udp "0.0.0.0:30120"

Update:
I tried to follow the FiveM proxy setup guide, but with no luck. I also looked at the linked GitHub guide, but was given port bind errors when trying to run nginx and the FiveM server at the same time.

2

I followed this guide and worked for my server (the same without override):

3

As you can see in the images provided, the records are set up like in the guide you provided.

5

So your problem is, an SSL connection tried to connect and it dropped connection, it dropped connection due to an Authentication failure, the Authentication failed because a SSL handshake failed to Authenticate with OpenSSL, the problem is with OpenSSL. The problem in OpenSSL is the TLS process key exchange is/has a bad signature. Here are my suggestions.

  1. Make sure your OpenSSL is configured to port 30120 with the proper ip address, the default port for OpenSSL is 443.
  2. Go online and find out how to get a brand new TLS process key.
  3. Configure your cloudflare to port 30120, and use the TCP method for HTTPS/SSL is based on TCP and not UDP.
  4. So your OpenSSL will provide and 100% secure HTTPS/SSL between your server and clients, Cloudflare would provide extreme network protection to the OpenSSL. Study how to configure your Cloudflare for maximum strength.

Oh and another thing, if you don’t have an set ip address for your server and your using 0.0.0.0, that can also cause issues.