Troubleshooting Issuing ACME Certificates

When requesting ACME certificates, cert-manager will create Order and Challenges to complete the request. As such, there are more resources to investigate and debug if there is a problem during the process. You can read more about these resources in the concepts pages.

Orders

In order to debug why a certificate isn’t being issued, we can first run kubectl describe on the Certificate resource we’re having issues with:

$ kubectl describe certificate example-com
...
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  GeneratedKey  82s   cert-manager  Generated a new private key
  Normal  Requested     81s   cert-manager  Created new CertificateRequest resource "example-com-2745722290"

We can then run another describe on the CertificateRequest resource that has been created:

$ kubectl describe certificaterequest example-com-2745722290
...
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  OrderCreated  5s    cert-manager  Created Order resource default/example-com-2745722290-439160286

We can see here that CertificateRequest controller has created an Order resource to request a new certificate from the ACME server.

Orders are a useful source of information when debugging failures issuing ACME certificates. By running kubectl describe order on a particular order, information can be gleaned about failures in the process:

$ kubectl describe order example-com-2745722290-439160286
...
Reason:
State:         pending
URL:           https://acme-v02.api.letsencrypt.org/acme/order/41123272/265506123
Events:
  Type    Reason   Age   From          Message
  ----    ------   ----  ----          -------
  Normal  Created  1m    cert-manager  Created Challenge resource "example-com-2745722290-439160286-0" for domain "test1.example.com"
  Normal  Created  1m    cert-manager  Created Challenge resource "example-com-2745722290-439160286-1" for domain "test2.example.com"

Here we can see that cert-manager has created two Challenge resources in order to complete the requirements of the ACME order to obtain a signed certificate.

You can then go on to run kubectl describe challenge example-com-2745722290-439160286-0 to further debug the progress of the Order.

Once an Order is successful, you should see an event like the following:

$ kubectl describe order example-com-2745722290-439160286
...
Reason:
State:         valid
URL:           https://acme-v02.api.letsencrypt.org/acme/order/41123272/265506123
Events:
  Type    Reason      Age   From          Message
  ----    ------      ----  ----          -------
  Normal  Created     72s   cert-manager  Created Challenge resource "example-com-2745722290-439160286-0" for domain "test1.example.com"
  Normal  Created     72s   cert-manager  Created Challenge resource "example-com-2745722290-439160286-1" for domain "test2.example.com"
  Normal  OrderValid  4s    cert-manager  Order completed successfully

If the Order is not completing successfully, you can debug the challenges for the Order by running kubectl describe on the Challenge resource.

Challenges

In order to determine why an ACME Certificate is not being issued, we can debug using the ‘Challenge’ resources that cert-manager has created.

In order to determine which Challenge is failing, you can run kubectl get challenges:

$ kubectl get challenges
...
NAME                                 STATE     DOMAIN            REASON                                     AGE
example-com-2745722290-4391602865-0  pending   example.com       Waiting for dns-01 challenge propagation   22s

This shows that the challenge has been presented using the DNS01 solver successfully and now cert-manager is waiting for the ‘self check’ to pass.

You can get more information about the challenge by using kubectl describe:

$ kubectl describe challenge example-com-2745722290-4391602865-0
...
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for dns-01 challenge propagation
  State:       pending
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Started    19s   cert-manager  Challenge scheduled for processing
  Normal  Presented  16s   cert-manager  Presented challenge using dns-01 challenge mechanism

Progress about the state of each challenge will be recorded either as Events or on the Challenge’s status block (as shown above).

March 2020 Let’s Encrypt CAA Rechecking Bug

Following the announcement on March 4 Let’s Encrypt will be revoking a number of certificates due to a bug in the way they validate CAA records, we have created a tool to analyse your existing cert-manager managed certificates and compare their serial numbers to the publicised list of revoked certificates. It’s advised that all users of Let’s Encrypt & cert-manager run a check using this tool to ensure they do not experience any invalid certificate errors in clusters. You can find a copy of the checker tool here: https://github.com/jetstack/letsencrypt-caa-bug-checker.

Last modified January 1, 0001