Kafka returns "No matching PRIVATE KEY entries in PEM file" when attempting to start using PEM certificates

Question

First of all, I've seen this thread but it's unrelated and having different issue.

I have the following settings fragment in my Kafka properties file:

ssl.keystore.type=PEM
ssl.keystore.key=/path/to/private.key
ssl.keystore.certificate.chain=/path/to/certificate.pem

ssl.truststore.type=PEM
ssl.truststore.certificates=/path/to/ca.pem

ssl.endpoint.identification.algorithm=

Note that ssl.endpoint.identification.algorithm is used because single-server certificate is used for each server in a cluster, therefore I have to bypass SSL hostname verification this way.

When starting Kafka, I am getting the following:

org.apache.kafka.common.KafkaException: org.apache.kafka.common.errors.InvalidConfigurationException: Invalid PEM keystore configs
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
at org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:107)
at kafka.network.Processor.<init>(SocketServer.scala:853)
at kafka.network.SocketServer.newProcessor(SocketServer.scala:442)
at kafka.network.SocketServer.$anonfun$addDataPlaneProcessors$1(SocketServer.scala:299)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:190)
at kafka.network.SocketServer.addDataPlaneProcessors(SocketServer.scala:297)
at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:262)
at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:259)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
at kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:259)
at kafka.network.SocketServer.startup(SocketServer.scala:131)
at kafka.server.KafkaServer.startup(KafkaServer.scala:285)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
Caused by: org.apache.kafka.common.errors.InvalidConfigurationException: Invalid PEM keystore configs
Caused by: org.apache.kafka.common.errors.InvalidConfigurationException: No matching PRIVATE KEY entries in PEM file

Thing is - private.key, certificate.pem and ca.pem are valid files and working with other applications/client libraries. Out of them I used to create keystore/truststore and it works fine. When using keystore/truststore, I've also used these 3 files to connect to Kafka from Python and it worked just fine. I confirm these files are valid and have no issues with other applications.

Also note that private key is PKCS#8 type, which is expected by Kafka:

-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----

What am I doing wrong?

Answer 1

You need to specify PEM file contents when using PEM store types and ssl.keystore.key and ssl.keystore.certificate.chain properties:

security.protocol=SSL
ssl.keystore.type=PEM

ssl.keystore.key=-----BEGIN PRIVATE KEY----- \
................................................................ \
.........................................= \
-----END PRIVATE KEY-----

ssl.keystore.certificate.chain=-----BEGIN CERTIFICATE----- \
................................................................ \
-----END CERTIFICATE-----

Answer 2

The properties for specifying PEM certificates in Kafka are a bit confusing but hope this overview will help. There are two ways to specify certificates in Kafka: as files or as strings (file content).

In both cases you need to specify the following properties:

• ssl.keystore.type=PEM
• ssl.truststore.type=PEM
• ssl.key.password= (empty, if private key is not encrypted)

It is worth reiterating that if the Private Key is encrypted, it must be in the PKCS#8 format for Java to read it.

Providing certificates as files

In this case, you want to specify the following properties:

• ssl.keystore.location=/path/to/file/containing/certificate/chain
• ssl.truststore.location=/path/to/truststore/certificate

The 'trick' here is that the file at ssl.keystore.location must contain the following (and in this exact order):

• your private key
• your signed certificate
• any intermediary CA certificates

Providing certificate content (certs as strings)

If you want to provide the content of the key/certificate/root certificate then you use the following properties:

• ssl.keystore.certificate.chain=-----BEGIN CERTIFICATE----- ...
• ssl.keystore.key=-----BEGIN PRIVATE KEY-----...

Similarly as above the ssl.keystore.certificate.chain should contain the following:

• your signed certificate
• any intermediary CA certificates