Jenkins - No valid crumb was included in request

Question

Jenkins - 2.263.1(LTS) deployed through tomcat on CentOS-8.2and have Nginx reverse proxy running in-front of Jenkins.

Under Manage Jenkins > Configure Systems - Apply and Save not working, Due to this error, i cannot Apply (or) Save any of my configurations, It always shows below error on browser (Firefox & Chrome).

HTTP Status 403 – Forbidden

Type Status Report

Message No valid crumb was included in the request

Description The server understood the request but refuses to authorize it. Apache Tomcat/9.0.30

Also Jenkins > Manage Jenkins > Configure Global Security - Apply works. But Save not working this too results same above given error.

Systems log error message.

Feb 19, 2021 10:56:05 AM WARNING hudson.security.csrf.CrumbFilter doFilter No valid crumb was included in request for /jenkins/configSubmit by ankit.sahu. Returning 403.

Workaround tried:-

1) Under Configure Global security > CSRF Protection > Enable proxy compatibility( Tick marked Enabled). - Didn't work so disabled with below command.
2) hudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION = true - Even this didn't solve the problem.
3) Installed the Strict Crumb Issuer plugin.
   Enabled this plugin and unchecked Check the session ID from its configuration (Under Jenkins Configure Global Security).
4) Restated the Jenkins.

Even tried by adding below in /apache-tomcat-9.0.30/conf/tomcat-users.xml file.

<?xml version='1.0' encoding='utf-8'?>
<tomcat-users>
    <role rolename="manager-gui"/>
    <role rolename="manager-script"/>
    <role rolename="manager-jmx"/>
    <role rolename="manager-status"/>
    <role rolename="admin-gui"/>
    <role rolename="admin-script"/>
    <user username="user" password="password" roles="manager-gui,manager-script,manager-jmx,manager-status,admin-gui,admin-script"/>
</tomcat-users> 

However still experiencing same problem. I don't know how to fix it, Can someone help me?

Answer 1

You can (temporarily) disable CSRF with below groovy script. Go to Manage Jenkins >> Script Console, then execute the below groovy script.

import jenkins.model.Jenkins

def instance = Jenkins.instance
instance.setCrumbIssuer(null)

Answer 2

The nonces embedded into web output from Jenkins with CSRF protection are based (at least in part as I've read) on values from the requesting client. In addition to making sure your reverse proxy is correctly configured to pass X-Forwarded-For and X-Forwarded-Proto, make sure that Tomcat valve is in place to expose those header values in the servlet request API so Jenkins has access to them.

Add the following to $CATALINA_BASE/conf/server.xml, subordinate to the <Host> element:

<Valve className="org.apache.catalina.valves.RemoteIpValve" remoteIpHeader="x-forwarded-for" protocolHeader="x-forwarded-proto" />

ref: https://www.jenkins.io/doc/book/system-administration/reverse-proxy-configuration-troubleshooting/
ref: https://www.jenkins.io/doc/book/system-administration/reverse-proxy-configuration-with-jenkins/
ref: https://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Remote_IP_Valve