'Issuing certificate as Secret does not exist
Below is the describe output for both my clusterissuer and certificate reource. I am brand new to cert-manager so not 100% sure this is set up properly - we need to use http01 validation however we are not using an nginx controller. Right now we only have 2 microservices so the public-facing IP address simply belongs to a k8s service (type loadbalancer) which routes traffic to a pod where an Extensible Service Proxy container sits in front of the container running the application code. Using this set up I haven't been able to get anything beyond the errors below, however as I mentioned I'm brand new to cert-manager & ESP so this could be configured incorrectly...
Name: clusterissuer-dev
Namespace:
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
API Version: cert-manager.io/v1beta1
Kind: ClusterIssuer
Metadata:
Creation Timestamp: 2020-08-07T18:46:29Z
Generation: 1
Resource Version: 4550439
Self Link: /apis/cert-manager.io/v1beta1/clusterissuers/clusterissuer-dev
UID: 65933d87-1893-49af-b90e-172919a18534
Spec:
Acme:
Email: [email protected]
Private Key Secret Ref:
Name: letsencrypt-dev
Server: https://acme-staging-v02.api.letsencrypt.org/directory
Solvers:
http01:
Ingress:
Class: nginx
Status:
Acme:
Last Registered Email: [email protected]
Uri: https://acme-staging-v02.api.letsencrypt.org/acme/acct/15057658
Conditions:
Last Transition Time: 2020-08-07T18:46:30Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events: <none>
Name: test-cert-default-ns
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
API Version: cert-manager.io/v1beta1
Kind: Certificate
Metadata:
Creation Timestamp: 2020-08-10T15:05:31Z
Generation: 2
Resource Version: 5961064
Self Link: /apis/cert-manager.io/v1beta1/namespaces/default/certificates/test-cert-default-ns
UID: 259f62e0-b272-47d6-b70e-dbcb7b4ed21b
Spec:
Dns Names:
dev.test.com
Issuer Ref:
Name: clusterissuer-dev
Secret Name: clusterissuer-dev-tls
Status:
Conditions:
Last Transition Time: 2020-08-10T15:05:31Z
Message: Issuing certificate as Secret does not exist
Reason: DoesNotExist
Status: False
Type: Ready
Last Transition Time: 2020-08-10T15:05:31Z
Message: Issuing certificate as Secret does not exist
Reason: DoesNotExist
Status: True
Type: Issuing
Next Private Key Secret Name: test-cert-default-ns-rrl7j
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Requested 2m51s cert-manager Created new CertificateRequest resource "test-cert-default-ns-c4wxd"
One last item - if I run the command kubectl get certificate -o wide I get the following output.
NAME READY SECRET ISSUER STATUS AGE
test-cert-default-ns False clusterissuer-dev-tls clusterissuer-dev Issuing certificate as Secret does not exist 2d23h
Solution 1:[1]
I had the same issue and I followed the advice given in the comments by @Popopame suggesting to check out the troubleshooting guide of cert-manager to find out how to troubleshoot cert-manager. or [cert-managers troubleshooting guide for acme issues] to find out which part of the acme process breaks the setup.
It seems that often it is the acme-challenge where letsencrypt verifies the domain ownership by requesting a certain code be offered at port 80 at a certain path. For example: http://example.com/.well-known/acme-challenge/M8iYs4tG6gM-B8NHuraXRL31oRtcE4MtUxRFuH8qJmY. Notice the http:// that shows letsencrypt will try to validate domain ownership on port 80 of your desired domain.
So one of the common errors is, that cert-manager could not put the correct challenge in the correct path behind port 80. For example due to a firewall blocking port 80 on a bare metal server or a loadbalancer that only forwards port 443 to the kubernetes cluster and redirects to 443 directly.
Also be aware of the fact, that cert-manager tries to validate the ACME challenge as well so you should configure the firewalls to allow requests coming from your servers as well.
If you have trouble getting your certificate to a different namespace, this would be a good point to start with.
In your specific case I would guess at a problem with the ACME challenge as the CSR (Certificate Signing Request) was created as indicated in the bottom most describe line but nothing else happened.
Solution 2:[2]
1. Setup Using Helm
By far the easiest method I've found was to use helm v3 to install cert-manager. I was able to set it up on a k3s cluster as follows:
$ helm repo add jetstack https://charts.jetstack.io
$ helm repo update
$ helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--version v1.2.0 \
--create-namespace \
--set installCRDs=true
2. Setup ClusterIssuer
Once it's installed you need to create a ClusterIssuer which can then be used when requesting certificates from let's encrypt.
$ more cert-clusterissuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-stg
namespace: cert-manager
spec:
acme:
email: [email protected]
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource that will be used to store the account's private key.
name: le-issuer-acct-key
solvers:
- dns01:
cloudflare:
email: [email protected]
apiTokenSecretRef:
name: cloudflare-api-token-secret
key: api-token
selector:
dnsZones:
- 'mydomdom.org'
- '*.mydomdom.org'
Deploy that, notice it'll get deployed into the same namespaces as cert-manager:
$ kubectl apply -f cert-clusterissuer.yaml
$ kubectl -n cert-manager get clusterissuers
NAME READY AGE
letsencrypt-stg True 53m
3. Setup Cloudflare API Token Secret
Deploy your Cloudflare API token into a secret, again put it into the cert-manager namespace:
$ more cloudflare-api-token.yaml
apiVersion: v1
kind: Secret
metadata:
name: cloudflare-api-token-secret
namespace: cert-manager
type: Opaque
stringData:
api-token: <my cloudflare api token key>
$ kubectl -n cert-manager -f cloudflare-api-token.yaml
4. Create a test Certificate
Now attempt to request the generation of a certificate from let's encrypt:
$ more test-certificate.yaml
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: le-test-mydomdom-org
namespace: cert-manager
spec:
secretName: le-test-mydomdom-org
issuerRef:
name: letsencrypt-stg
kind: ClusterIssuer
commonName: 'le-test.mydomdom.org'
dnsNames:
- "le-test.mydomdom.org"
$ kubectl -n cert-manager apply -f test-certificate.yaml
5. Debugging Certificate Creation
You can then watch the request as it flows through the various stages. I believe the flow is certificates -> certificaterequests -> orders -> challenges.
NOTE: Knowing this general flow was hugely helpful for me in terms of understanding where a request was failing within kubernetes as I was attempting to debug it.
When debugging you'll typically want to do kubectl get -n cert-manager <stage> -A to see a list of all the outstanding resources within that stage. Keep in mind that after a challenge is fulfilled it'll no longer show up within the output of kubectl get -n cert-manager challenges.
Also keep in mind that any DNS entries created to fulfill the challenge stage will typically have their TTL set to ~2min so if you go looking in your Cloudflare UI and do not see them, they likely already timed out and rolled off.
For e.g.:
References
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | |
| Solution 2 | Arnaud Leymet |