Reverse Proxy troubleshooting guide

Article ID:360034638071
2 minute readKnowledge base

Symptoms

  • Any error message related to Reverse Proxy configuration, like ERROR! Reverse proxy misconfigured?.

Diagnosis/Treatment

Review the Reverse Proxy configuration

Ensure your Reverse Proxy configuration is providing specifically these headers in all requests to Jenkins:

  • X-Forwarded-Proto which should be either http or https and is only needed if your proxy is being accessed via a different protocol from the protocol Jenkins is serving on. In most cases people use a reverse proxy to offload TLS termination, in that case: the Browser accesses the proxy over https and then the proxy accesses Jenkins over http. Thus for TLS termination at the proxy we need to tell Jenkins that the request was actually over https even though Jenkins received the request on http. So the proxy needs to inject the X-Forwarded-Proto: https header into the requests made by the proxy to Jenkins when the proxy receives a request over https

  • X-Forwarded-Host which should be the original Host header that the proxy responded to. When the proxy receives a request for example-cluster.com, that request will have a Host: example-cluster.com header. Some proxies will be forwarding to a completely different machine under a different hostname…​ and if the machine they are forwarding to is using virtual hosting the proxy may need to supply a different Host: header in the request it makes to the target back-end server. Thus the target back-end server (Jenkins in this case) will see the "wrong" Host header. Jenkins sets the X-Forwarded-Host header to the originally requested Host header value so that it can determine what was the originally requested hostname.

  • X-Forwarded-Port which is to handle the case where the request incoming to the proxy is for a different port from the request incoming to the target server. For example, if the incoming requests are on port 443 but Jenkins is responding to requests on port 8080 thus X-Fowarded-Port to be 443

Validate The URL from Request

From the Jenkins Script Console, run the following script

Jenkins.instance.getRootUrlFromRequest()
▼

The expected output should be the root URL of Jenkins (in case of TLS is implemented then https://example-cluster.com/my-jenkins/- the one used to Browse the Jenkins Web Application). If it disagrees in any way, then the reverse proxy is broken (for instance if it said https://example-cluster.com/my-jenkins/.)

Validate which Headers are Jenkins seeing

From the Jenkins Script Console, run the following script

def req = org.kohsuke.stapler.Stapler.currentRequest
def headerNames = req.getHeaderNames();
while (headerNames.hasMoreElements()) {
  def headerName = headerNames.nextElement();
  if (!headerName.toLowerCase().startsWith("x-forward")) continue
  println(headerName+":");

  def headers = req.getHeaders(headerName);
  while (headers.hasMoreElements()) {
    def headerValue = headers.nextElement();
    println("\t" + headerValue);
  }
}
▼

The expected output would be similar to:

X-Forwarded-Host:
    example-cluster.com
X-Forwarded-Proto:
    https
X-Forwarded-Port:
    443
X-Forwarded-For:
    <THE_IP_ADDRESS_OF_YOUR_BROWSER>
▼

If you are seeing those X-Forwarded-Host, X-Forwarded-Proto and X-Forwarded-Port header values then that suggests that the problem is in the servlet container. If you are not seeing those header values, then you the problem is in the configuration of the reverse proxy.

References