Troubleshooting Issuing ACME Certificates
When requesting ACME certificates, cert-manager will create Order and
Challenges to complete the request. As such, there are more resources to
investigate and debug if there is a problem during the process. You can read
more about these resources in the concepts
pages.
Before you start here you should probably take a look at our general troubleshooting guide
1. Troubleshooting (Cluster)Issuers
First of all check if the (Cluster)Issuer you’re using is in a ready state:
$ kubectl get issuer
$ kubectl get clusterissuer
NAME READY AGE
letsencrypt True 38m
letsencrypt-http False 32mIf you see False check the status using kubectl describe. For example:
$ kubectl describe issuer letsencrypt-http
$ kubectl describe clusterissuer letsencrypt-http
Name: letsencrypt
API Version: cert-manager.io/v1
Kind: Issuer
Spec:
Acme:
Email: cert-manager@example.com
Private Key Secret Ref:
Name: letsencrypt
Server: https://acme-staging-v02.api.letsencrypt.org/directory
Status:
Acme:
Conditions:
Message: Failed to update ACME account:400 urn:ietf:params:acme:error:invalidEmail: Unable to update account :: invalid contact domain. Contact emails @example.com are forbidden
Reason: ErrUpdateACMEAccount
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ErrUpdateACMEAccount 101s (x3 over 106s) cert-manager Failed to update ACME account:400 urn:ietf:params:acme:error:invalidEmail: Unable to update account :: invalid contact domain. Contact emails @example.com are forbiddenCommon errors
Failed to update ACME account:400 urn:ietf:params:acme:error:invalidEmail: the email you specified in the Issuer configuration isn’t valid.Error initializing issuer: Failed to register ACME account: secrets "acme-key" already exists: there might be a leftover account from a previous issuer that is no longer valid, you should remove the secret so it can be recreated.
2. Troubleshooting Orders
When we run a describe on the CertificateRequest resource we see that an Order that has
been created:
$ kubectl describe certificaterequest example-com-2745722290
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal OrderCreated 5s cert-manager Created Order resource default/example-com-2745722290-439160286Orders are a request to an ACME instance to issue a certificate.
By running kubectl describe order on a particular order,
information can be gleaned about failures in the process:
$ kubectl describe order example-com-2745722290-439160286
...
Reason:
State: pending
URL: https://acme-v02.api.letsencrypt.org/acme/order/41123272/265506123
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 1m cert-manager Created Challenge resource "example-com-2745722290-439160286-0" for domain "test1.example.com"
Normal Created 1m cert-manager Created Challenge resource "example-com-2745722290-439160286-1" for domain "test2.example.com"Here we can see that cert-manager has created two Challenge resources to verify we control specific domains, a requirements of the ACME order to obtain a signed certificate.
You can then go on to run
kubectl describe challenge example-com-2745722290-439160286-0 to further debug the
progress of the Order.
Once an Order is successful, you should see an event like the following:
$ kubectl describe order example-com-2745722290-439160286
...
Reason:
State: valid
URL: https://acme-v02.api.letsencrypt.org/acme/order/41123272/265506123
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 72s cert-manager Created Challenge resource "example-com-2745722290-439160286-0" for domain "test1.example.com"
Normal Created 72s cert-manager Created Challenge resource "example-com-2745722290-439160286-1" for domain "test2.example.com"
Normal OrderValid 4s cert-manager Order completed successfullyIf the Order is not completing successfully, you can debug the challenges
for the Order by running kubectl describe on the Challenge resource which is described in the following steps.
3. Troubleshooting Challenges
In order to determine why an ACME Order is not being finished, we can debug
using the Challenge resources that cert-manager has created.
In order to determine which Challenge is failing, you can run
kubectl get challenges:
$ kubectl get challenges
...
NAME STATE DOMAIN REASON AGE
example-com-2745722290-4391602865-0 pending example.com Waiting for dns-01 challenge propagation 22sThis shows that the challenge has been presented using the DNS01 solver successfully and now cert-manager is waiting for the ‘self check’ to pass.
You can get more information about the challenge and it’s lifecycle by using kubectl describe:
$ kubectl describe challenge example-com-2745722290-4391602865-0
...
Status:
Presented: true
Processing: true
Reason: Waiting for dns-01 challenge propagation
State: pending
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 19s cert-manager Challenge scheduled for processing
Normal Presented 16s cert-manager Presented challenge using dns-01 challenge mechanismProgress about the state of each challenge will be recorded either as Events
or on the Challenge’s status block (as shown above).
In case of DNS01 you will find any errors from your DNS provider here.
Both HTTP01 and DNS01 go through a “self-check” first before cert-manager presents the challenge to the ACME provider. This is done not to overload the ACME provider with failed challenges due to DNS or loadbalancer propagations. The status of this can be found in the Status block of the describe:
$ kubectl describe challenge
[...]
Status:
Presented: true
Processing: true
Reason: Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://example.com/.well-known/acme-challenge/_fgdLz0i3TFiZW4LBjuhjgd5nTOkaMBhxYmTY': Get "http://example.com/.well-known/acme-challenge/_fgdLz0i3TFiZW4LBjuhjgd5nTOkaMBhxYmTY: remote error: tls: handshake failure
State: pending
[...]In this example our HTTP01 check fails due a network issue. You will also see any errors coming from your DNS provider here.
HTTP01 troubleshooting
First of all check if you can see the challenge URL from the public internet, if this does not work check your Ingress and firewall configuration as well as the service and pod cert-manager created to solve the ACME challenge.
If this does work check if your cluster can see it too. It is important to test this from inside a Pod. If you get a connection error it is suggested to check the cluster’s network configuration.
If you receive a tls: handshake failure, try setting the annotation cert-manager.io/issue-temporary-certificate: "true" on the Ingress or Certificate resource. This will issue a temporary self signed certificate for the ingress controller to use before the actual certificate is issued.
If you still are having issues, there may be an issue with your ingress controller handling multiple resources for the same hostname, in this case, the annotation acme.cert-manager.io/http01-edit-in-place: "true" is likely required.
For example when using GKE with the Google Cloud Loadbalancer it is recommended to set:
cert-manager.io/issue-temporary-certificate: "true"
acme.cert-manager.io/http01-edit-in-place: "true"
This will allow the Google Cloud Loadbalancer to propagate a HTTPS endpoint correctly with a temporary certificate, the http01-edit-in-place part will prevent GKE from assigning a 2nd IP address for the challenge endpoint.
Got 404 status code
If your challenge self-check fails with a 404 not found error. Make sure to check the following:
- you can access the URL from the public internet
- the ACME solver pod is up and running
- use
kubectl describe ingressto check the status of the HTTP01 solver ingress. (unless you useacme.cert-manager.io/http01-edit-in-place, then check the same ingress as your domain)
DNS01 troubleshooting
If you see no error events about your DNS provider you can check the following
Check if you can see the _acme_challenge.domain TXT DNS record from the public internet, or in your DNS provider’s interface.
cert-manager will check if a DNS record has been propagated by querying the cluster’s DNS solver. If you are able to see it from the public internet but not from inside the cluster you might want to change the DNS server for self-check as some cloud providers overwrite DNS internally.
cert-manager identifies the wrong zone for your domain name
cert-manager by default uses SOA (Start of Authority) records to determine which zone name to use at your DNS provider. Some DNS resolvers will filter this information, if this is the case cert-manager cannot determine the zone and it is advised to change the DNS server for DNS01 self-checks.
If you use dnsmasq as your DNS server, this may occur if you use the --filterwin2k flag.
In OpenWRT there is a filterwin2k configuration option.
And in LuCI there is a “Filter useless” option.
By enabling this flag, dnsmasq drops all SOA records.
March 2020 Let’s Encrypt CAA Rechecking Bug
Following the announcement on March 4 Let’s Encrypt will be revoking a number of certificates due to a bug in the way they validate CAA records, we have created a tool to analyse your existing cert-manager managed certificates and compare their serial numbers to the publicised list of revoked certificates. It’s advised that all users of Let’s Encrypt & cert-manager run a check using this tool to ensure they do not experience any invalid certificate errors in clusters. You can find a copy of the checker tool here: https://github.com/jetstack/letsencrypt-caa-bug-checker.