I have the exact same problem using a failover IP address.
On the Keymaster Server page, it shows the IP address of the server and not the failover IP address. with the following error message:
System.Net.Http.HttpRequestException: Connection refused