Solr 8 not working when trying to start Solr: java.lang.IllegalStateException: /opt/solr-8.11.1/server/NONE is not a valid keystore?

Question

I am trying to setup Solr 8.11.1 on a new server that is running RHEL 8. I followed all of the steps listed in the Solr guide (https://solr.apache.org/guide/8_11/taking-solr-to-production.html)

I pretty much was able to follow the guide with no issue up until the part where it checks the progress (where I'm supposed to start solr). That's when I try to start Solr and see the error "java.lang.IllegalStateException: /opt/solr-8.11.1/server/NONE is not a valid keystore".

I ran the install script that comes packaged with Solr and which the guide says to run. I kept all the default values. So I'm not sure why I'm getting this issue. I did set "SOLR_SSL_ENABLED=false" in the /etc/default/solr.in.sh file. Yet I still get that error when I try to start solr. Any ideas on why I'm getting it and how to resolve it? Thank you in advance!

==========================================================================

/etc/default/solr.in.sh:

SOLR_SSL_ENABLED=true
SOLR_SSL_KEY_STORE=/opt/solr-8.11.1/server/etc/solr-ssl.keystore.jks
SOLR_SSL_KEY_STORE_PASSWORD=secret
SOLR_SSL_TRUST_STORE=/opt/solr-8.11.1/server/etc/solr-ssl.keystore.jks
SOLR_SSL_TRUST_STORE_PASSWORD=secret
SOLR_SSL_NEED_CLIENT_AUTH=false
SOLR_SSL_WANT_CLIENT_AUTH=false
SOLR_SSL_CHECK_PEER_NAME=true
SOLR_SSL_KEY_STORE_TYPE=JKS
SOLR_SSL_TRUST_STORE_TYPE=JKS

==========================================================================

Solr Log:

2022-02-04 20:52:10.610 INFO (main) [ ] o.e.j.u.log Logging initialized @1276ms to org.eclipse.jetty.util.log.Slf4jLog 2022-02-04 20:52:10.878 INFO (main) [ ] o.e.j.s.Server jetty-9.4.44.v20210927; built: 2021-09-27T23:02:44.612Z; git: 8da83308eeca865e495e53ef315a249d63ba9 332; jvm 1.8.0_312-b07 2022-02-04 20:52:10.898 INFO (main) [ ] o.e.j.d.p.ScanningAppProvider Deployment monitor [file:///opt/solr-8.11.1/server/contexts/] at interval 0 2022-02-04 20:52:11.173 INFO (main) [ ] o.e.j.w.StandardDescriptorProcessor NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet 2022-02-04 20:52:11.185 INFO (main) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0 2022-02-04 20:52:11.185 INFO (main) [ ] o.e.j.s.session No SessionScavenger set, using defaults 2022-02-04 20:52:11.187 INFO (main) [ ] o.e.j.s.session node0 Scavenging every 600000ms 2022-02-04 20:52:11.240 INFO (main) [ ] o.a.s.u.c.SSLConfigurations Setting javax.net.ssl.keyStorePassword 2022-02-04 20:52:11.241 INFO (main) [ ] o.a.s.u.c.SSLConfigurations Setting javax.net.ssl.trustStorePassword 2022-02-04 20:52:11.248 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2022-02-04 20:52:11.252 INFO (main) [ ] o.a.s.s.SolrDispatchFilter ___ _
Welcome to Apache Solr™ version 8.11.1 2022-02-04 20:52:11.252 INFO (main) [ ] o.a.s.s.SolrDispatchFilter / | | | _ Starting in standalone mode on port 8983 2022-02-04 20:52:11.252 INFO (main) [
] o.a.s.s.SolrDispatchFilter _ / _ \ | '| Install dir: /opt/solr 2022-02-04 20:52:11.253 INFO (main) [ ] o.a.s.s.SolrDispatchFilter |/___/|_| Start time: 2022-02-04T20:52:11.253Z 2022-02-04 20:52:11.264 INFO (main) [ ] o.a.s.c.SolrPaths Using system property solr.solr.home: /var/solr/data 2022-02-04 20:52:11.264 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /var/solr/data/solr.xml 2022-02-04 20:52:11.341 INFO (main) [
] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7bedc48a, but no JMX reporters we re configured - adding default JMX reporter. 2022-02-04 20:52:11.671 INFO (main) [ ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelist HostCheckingEnabled=true] 2022-02-04 20:52:11.703 WARN (main) [ ] o.a.s.c.s.i.Http2SolrClient Create Http2SolrClient with HTTP/1.1 transport since Java 8 or lower versions does not support SSL + HTTP/2 2022-02-04 20:52:11.760 ERROR (main) [ ] o.a.s.s.SolrDispatchFilter Could not start Solr. Check solr/home property and the logs 2022-02-04 20:52:11.782 ERROR (main) [ ] o.a.s.c.SolrCore null => org.apache.solr.common.SolrException: Error instantiating shardHandlerFactory class [Http ShardHandlerFactory]: java.lang.IllegalStateException: /opt/solr-8.11.1/server/NONE is not a valid keystore at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:56) org.apache.solr.common.SolrException: Error instantiating shardHandlerFactory class [HttpShardHandlerFactory]: java.lang.IllegalStateException: /opt/solr- 8.11.1/server/NONE is not a valid keystore at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:56) ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83 ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55] at org.apache.solr.core.CoreContainer.load(CoreContainer.java:696) ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50

• janhoy - 2021-12-14 13:50:55] at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:287) ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef3 6b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55] at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:184) ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b5 0 - janhoy - 2021-12-14 13:50:55] at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:140) ~[jetty-servlet-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:731) ~[jetty-servlet-9.4.44.v20210927.jar:9.4.44.v20210927] at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_312] at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) ~[?:1.8.0_312] at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) ~[?:1.8.0_312] at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:755) ~[jetty-servlet-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379) ~[jetty-servlet-9.4.44.v20210927.jar:9.4.44.v20210927 ] at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449) ~[jetty-webapp-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414) ~[jetty-webapp-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:910) ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288) ~[jetty-servlet-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) ~[jetty-webapp-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426) ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.server.Server.start(Server.java:423) ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.server.Server.doStart(Server.java:387) ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$3(XmlConfiguration.java:1907) ~[jetty-xml-9.4.44.v20210927.jar:9.4.44.v20210927] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_312] at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1857) ~[jetty-xml-9.4.44.v20210927.jar:9.4.44.v20210927] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_312] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_312] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_312] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_312] at org.eclipse.jetty.start.Main.invokeMain(Main.java:218) ~[start.jar:9.4.44.v20210927] at org.eclipse.jetty.start.Main.start(Main.java:491) ~[start.jar:9.4.44.v20210927] at org.eclipse.jetty.start.Main.main(Main.java:77) ~[start.jar:9.4.44.v20210927] Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: /opt/solr-8.11.1/server/NONE is not a valid keystore at org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:234) ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83e f36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57] at org.apache.solr.client.solrj.impl.Http2SolrClient.(Http2SolrClient.java:159) ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed12 23c14b50 - janhoy - 2021-12-14 13:50:57] at org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:857) ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36 b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57] at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:319) ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e 83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55] at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:51) ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83 ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55] ... 53 more Caused by: java.lang.IllegalStateException: /opt/solr-8.11.1/server/NONE is not a valid keystore at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:50) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1203) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:322) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:244) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:255) ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] at org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:232) ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83e f36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57] at org.apache.solr.client.solrj.impl.Http2SolrClient.(Http2SolrClient.java:159) ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed12 23c14b50 - janhoy - 2021-12-14 13:50:57] at org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:857) ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36 b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57] at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:319) ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e 83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55] at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:51) ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83 ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55] ... 53 more 2022-02-04 20:52:11.803 INFO (main) [ ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@11bb571c{/solr,file:///opt/solr-8.11.1/server/solr-webap p/webapp/,AVAILABLE}{/opt/solr-8.11.1/server/solr-webapp/webapp} 2022-02-04 20:52:11.926 INFO (main) [ ] o.e.j.s.AbstractConnector Started ServerConnector@383dc82c{SSL, (ssl, http/1.1)}{0.0.0.0:8983} 2022-02-04 20:52:11.926 INFO (main) [ ] o.e.j.s.Server Started @2594ms 2022-02-04 20:53:00.999 INFO (ShutdownMonitor) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@383dc82c{SSL, (ssl, http/1.1)}{0.0.0.0:8983} 2022-02-04 20:53:00.999 INFO (ShutdownMonitor) [ ] o.e.j.s.session node0 Stopped scavenging 2022-02-04 20:53:01.007 INFO (ShutdownMonitor) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.w.WebAppContext@11bb571c{/solr,null,STOPPED}{/opt/solr-8.11.1

Answer 1

So my team and I FINALLY got it working after trying numerous ways. So I thought I'd go ahead and share this info in hopes that the next person will find this info useful to them. First we found out that our RHEL 8.5 system has FIPS enabled. So trying to start Solr without a FIPS compliant keystore just wouldn't work. We kept trying to figure out how to create a FIPS compliant keystore, but were unable to. Then one of our team members found this link https://github.com/kvbutler/solr8-rehl8.5-fips-sip online. This pretty much got us 90% there. We just had to make a few tweaks on our end such as having Solr use Java 11 openjdk instead of Java 8, making a few SELinux changes, and a few other changes to the solr.in.sh file as needed for our setup, but otherwise the guide pretty much got us to our solution. Hope this helps!