Mert Özen Further With Every Line
DevOps SSL/TLS Certificates and HTTPS — #05

Certificates on Managed Platforms: Caddy, Azure App Service, Key Vault, Front Door and cert-manager on Kubernetes

Mert Özen Nov 14, 2026 17 min 5 views
Certificates on Managed Platforms: Caddy, Azure App Service, Key Vault, Front Door and cert-manager on Kubernetes

On managed platforms the platform obtains and renews the certificate, but each has a limit that will catch you: Caddy hits rate limits without a /data volume, the App Service managed certificate has no wildcard, Front Door wants a direct CNAME, and cert-manager changes the key on every renewal. With real runs of Caddy, cert-manager and Envoy Gateway against Pebble, we watch pinning break on renewal.

Version note: The examples in this post were run with Caddy 2.11.9, cert-manager 1.21.3, cmctl 2.6.2, Envoy Gateway 1.10, kind 0.33 with Kubernetes 1.37, Helm 4.3 and Let's Encrypt's test server Pebble (November 2026 image) on Docker 29; the Azure sections were not run and are taken from Microsoft Learn pages dated November 2026.

Guess First

Bookstore's mobile app pins the server's public key when it calls the API: if the public key hash of the certificate in the TLS handshake doesn't match the value baked into the app, it drops the connection. This check caused no trouble for two years while the API ran on Nginx and Certbot, because the team ran Certbot with --reuse-key and the key didn't change on renewal. Now the platform team is moving the API to Kubernetes, and cert-manager will obtain the certificate.

Guess: what happens to the mobile app the first time cert-manager renews the certificate? And a second question: the same team runs Caddy as a container in a test environment and deletes and recreates the container on every deploy. If they deploy five times in a week, what does Let's Encrypt say? We'll see both with real runs.

Caddy: Automatic HTTPS with Zero Config

In the previous two posts an ACME client obtained the certificate and the server read it from a file; deploy hooks tied the two together. Caddy merges them into one process. All it needs is a hostname in the config: it obtains the certificate, stores it, renews it and redirects HTTP to HTTPS (HTTP Secure). Since I used Pebble instead of Let's Encrypt in the lab, I wrote the issuer explicitly; in a real setup the whole tls block is unnecessary, because Caddy tries Let's Encrypt by default and falls back to ZeroSSL:

{
	email ops@bookstore.example
}

api.bookstore.example {
	tls {
		issuer acme {
			dir https://pebble:14000/dir
			trusted_roots /etc/caddy/pebble.minica.pem
			profile shortlived
		}
	}
	respond "ok from {host}" 200
}

profile shortlived asks for the 6-day profile among the ACME profiles we saw in the third post; Caddy's documentation marks this setting experimental because the profiles specification is still a draft. I started the container and filtered the log:

$ docker run -d --name tlsseries-caddy --network tlsseries-net --network-alias api.bookstore.example \
    -v ./caddy:/etc/caddy:ro -v tlsseries-caddy-data:/data caddy:2
$ docker logs tlsseries-caddy 2>&1 | jq -r 'select((.logger // "")|test("tls.obtain|acme_client|auto_https"))
    | [.logger, .msg, (.challenge_type // .issuer // "")] | @tsv'
http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS
http.auto_https	enabling automatic HTTP->HTTPS redirects
tls.obtain	acquiring lock
tls.obtain	lock acquired
tls.obtain	obtaining certificate
http.acme_client	trying to solve challenge	http-01
http.acme_client	authorization finalized
http.acme_client	validations succeeded; finalizing order
http.acme_client	got renewal info
http.acme_client	successfully downloaded available certificate chains
tls.obtain	certificate obtained successfully	pebble:14000-dir
tls.obtain	releasing lock

Five seconds after the container came up the certificate was ready. Caddy keeps both HTTP-01 and TLS-ALPN-01 (Application-Layer Protocol Negotiation) enabled; in one of two runs it picked HTTP-01, in the other TLS-ALPN-01 over port 443. So if port 80 is closed for some reason, validation can still pass. I checked the result from inside the container:

$ curl -sI http://api.bookstore.example/ | head -3
HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://api.bookstore.example/
$ ls -l /data/caddy/certificates/pebble-14000-dir/api.bookstore.example/
-rw-------    1 root     root          1271 Nov 13 22:29 api.bookstore.example.crt
-rw-------    1 root     root           509 Nov 13 22:29 api.bookstore.example.json
-rw-------    1 root     root           227 Nov 13 22:29 api.bookstore.example.key
$ openssl x509 -in /data/caddy/certificates/pebble-14000-dir/api.bookstore.example/api.bookstore.example.crt \
    -noout -issuer -serial -dates
issuer=CN=Pebble Intermediate CA 160d5b
serial=6D7F1A0942A2A3E3
notBefore=Nov 13 22:29:27 2026 GMT
notAfter=Nov 19 22:29:26 2026 GMT
$ curl -sS --cacert /tmp/root.pem https://api.bookstore.example/
ok from api.bookstore.example

The redirect comes with 308; unlike 301, it asks the client to keep the method and the body. The certificate, key and metadata sit under /data as files only root can read. There's no live/ and archive/ maze as with Certbot, no symlinks and no deploy hook; because the server and the ACME client are the same process, the renewed certificate is swapped directly in memory.

Renewal in Caddy, ARI and the /data Volume

The got renewal info line in the log is ARI. Thanks to this extension, defined in RFC 9773, right after obtaining the certificate Caddy asked the CA when it should renew:

$ docker logs tlsseries-caddy 2>&1 | jq -c 'select(.msg=="got renewal info")
    | {window_start: (.window_start|todate), window_end: (.window_end|todate),
       selected_time: (.selected_time|todate), recheck_after: (.recheck_after|floor|todate)}'
{"window_start":"2026-11-16T22:29:26Z","window_end":"2026-11-18T22:29:26Z","selected_time":"2026-11-18T15:29:50Z","recheck_after":"2026-11-14T04:29:32Z"}

For a 6-day certificate the CA suggested a window between days 3 and 5. Caddy picked a moment inside the window and will ask again in six hours. If the CA decides on a mass revocation, it pulls the window forward and Caddy renews early on its next check; the same behavior we saw with Certbot in the third post. With a CA without ARI, Caddy renews when a third of the lifetime remains (renewal_window_ratio, default 0.3333). It doesn't stay quiet on errors either: on my first attempt, when it couldn't reach Pebble, the log said it would retry in 60 seconds and that the upper bound for retrying was 2592000 seconds, that is 30 days. According to the docs the wait between attempts grows to at most one day.

Now the second question from Guess First. All of Caddy's state is in the /data folder: the ACME account key, the certificates, the ARI information. I deleted and recreated the container four times; in the first two /data was on a named volume, in the last two there was no volume:

#!/bin/sh
# recreate the Caddy container ($1 = extra docker args) and print the served serial
docker rm -f tlsseries-caddy >/dev/null
docker run -d --name tlsseries-caddy --network tlsseries-net --network-alias api.bookstore.example \
    -v ./caddy:/etc/caddy:ro $1 caddy:2 >/dev/null
sleep 8
printf "obtained=%s " "$(docker logs tlsseries-caddy 2>&1 | grep -c 'certificate obtained successfully')"
docker exec tlsseries-caddy sh -c 'apk add -q openssl >/dev/null 2>&1; echo | openssl s_client -connect localhost:443 -servername api.bookstore.example 2>/dev/null | openssl x509 -noout -serial'
$ ./recreate.sh "-v tlsseries-caddy-data:/data"
obtained=0 serial=6D7F1A0942A2A3E3
$ ./recreate.sh "-v tlsseries-caddy-data:/data"
obtained=0 serial=6D7F1A0942A2A3E3
$ ./recreate.sh ""
obtained=1 serial=4A2D36F8F0CBDE6A
$ ./recreate.sh ""
obtained=1 serial=0743CC64EACD3BCA

With the volume, Caddy loads the certificate from disk and never goes to the CA. Without it, every start means a new account and a new certificate. Pebble doesn't mind; Let's Encrypt does: it issues at most five new certificates for the same set of names in seven days and takes no override requests for this limit. The team deploying five times a week opens its sixth deploy without a certificate, and capacity refills at one certificate every 34 hours. Renewals coordinated through ARI are exempt from all rate limits, but a certificate obtained from scratch doesn't count as a renewal. If you run Caddy in a container, /data must be on a persistent volume, and multiple instances must share the same storage. Caddy takes locks on shared storage so that only one instance requests a given certificate; that's the acquiring lock line in the log.

Caddy or Nginx + Certbot?

In the Web Servers and Hosting series post "Choosing Hosting and a Production Checklist" we mentioned Caddy as an option. From the certificate angle the decision comes down to a few questions. Are you setting up a new server, or do you have an Nginx config that has been maintained for years? On a new setup, especially with several sites on one machine, Caddy cuts the number of moving parts considerably: no timer, no hook, no reload, no file permissions. On an existing Nginx, the Certbot setup from the third post already works; switching web servers just for certificates means rewriting rate limiting, caching and header settings.

The second question is whether you have many domains that aren't known in advance. Imagine Bookstore later letting publishers attach their own domains. Caddy's on-demand TLS obtains the certificate during the first handshake that arrives for that name. When you turn it on, you must restrict which domains are accepted with an ask endpoint; otherwise someone can point random names at your server and burn rate limits in your name. With Nginx + Certbot this scenario is a serious automation project.

The third question is observability. With Certbot, renewal is a separate process and is monitored through exit codes, log files and systemd state. With Caddy, renewal lives inside the JSON (JavaScript Object Notation) log. Whichever you choose, you need to measure renewal success from the outside; that's the subject of the last post in the series.

Azure: App Service, Key Vault and Front Door

This section needs an Azure account, so it wasn't run; the rules come from Microsoft Learn. With managed certificates the platform does most of the work, but each one has a limit that will catch you.

App Service Managed Certificate

The free managed certificate is issued by DigiCert and renewed by App Service; it requires a Basic or higher plan. Know the limits up front: no wildcards, the certificate can't be exported (so you can't use the same certificate anywhere else), private DNS and App Service Environment aren't supported, and the domain can be at most 64 characters. An apex domain (bookstore.example) needs an A record pointing at the app's IP address, and a subdomain needs a CNAME going directly to <app>.azurewebsites.net. If another CNAME sits in between, the certificate can be neither issued nor renewed. If your domain has a CAA record you need to add 0 issue digicert.com; as we saw in the second post, CAA makes CAs not on the list refuse.

This service changed twice in 2025. On 28 July 2025, to comply with the MPIC rule, DigiCert switched validation to an HTTP request to /.well-known/pki-validation/fileauth.txt. The result: apps with IP restrictions, private endpoints or disabled public access could neither obtain nor renew the certificate. In November 2025 Microsoft started answering that request in App Service's front-end layer; the request no longer reaches the app's workers, and the public access requirement is gone. What still applies is that the domain must resolve in public DNS, and Traffic Manager's "nested" and "external" endpoints aren't supported. These two changes show the real nature of a managed certificate: the validation method isn't under your control, and when a CA policy changes, renewal can break silently.

Certificates from Key Vault

If you need a wildcard, an exportable key or a specific CA, you keep the certificate in Key Vault and import it into App Service. App Service reads the certificate in PKCS #12 format, also known as PFX (Personal Information Exchange), from the vault, syncs a new version from the vault within 24 hours on its own and updates the binding; if you don't want to wait you trigger Sync manually in the portal. A common mistake when granting access is giving the role to the app's managed identity. The reader isn't the app but the App Service resource provider: the Microsoft.Azure.WebSites service principal needs the Key Vault Certificate User role (or Get for secrets and certificates in an access policy). Key Vault can renew certificates by itself from its integrated CAs (DigiCert, GlobalSign). It doesn't speak ACME to Let's Encrypt directly; in that case you run an ACME client as a separate job and write the result to the vault.

Managed Certificates on Front Door

Front Door Standard/Premium also uses a DigiCert certificate and validates domain ownership with a TXT (text) record named _dnsauth.<subdomain>. Automatic renewal has a condition: the custom domain's CNAME must point directly at the Front Door endpoint. If the CNAME goes through a chain, if an A record is used, or if the apex domain is attached with CNAME flattening, the domain drops into "Pending Revalidation" 45 days before expiry and you need to enter a new TXT record. A managed certificate can be issued for a wildcard, but it isn't renewed automatically. If you supply your own certificate from Key Vault, choose "Latest" as the secret version; Front Door picks up the new version itself, but according to the docs that can take up to 72 hours. One more surprise: Front Door doesn't accept EC keys in a certificate you bring; it needs RSA (Rivest–Shamir–Adleman).

cert-manager on Kubernetes: ClusterIssuer and Gateway

The de facto standard for certificate automation on Kubernetes is cert-manager. In the Kubernetes series post "External Traffic in Kubernetes: Ingress, the ingress-nginx Retirement and Gateway API with HTTPRoute" we installed Envoy Gateway and mentioned cert-manager in one paragraph. Here we run it end to end. Gateway API support is off by default in the Helm chart; the config you enable is written to the controller's configuration file:

$ helm install cert-manager oci://quay.io/jetstack/charts/cert-manager --version v1.21.3 \
    -n cert-manager --create-namespace --set crds.enabled=true \
    --set config.apiVersion=controller.config.cert-manager.io/v1alpha1 \
    --set config.kind=ControllerConfiguration --set config.gatewayAPI.enabled=true --wait
$ kubectl -n cert-manager logs deploy/cert-manager | grep -i "gateway api"
I1113 22:18:45.826508       1 options.go:296] "enabling the sig-network Gateway API certificate-shim and HTTP-01 solver" logger="cert-manager"

The Gateway API CRDs must be in the cluster before cert-manager starts; if you install them later you have to restart the cert-manager deployment. Here Envoy Gateway's chart had brought the CRDs beforehand. Next, the ClusterIssuer that defines the CA:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: acme-pebble
spec:
  acme:
    # in production: https://acme-v02.api.letsencrypt.org/directory
    server: https://pebble.pebble.svc:14000/dir
    # lab only: the CA that signed Pebble's HTTPS certificate (base64 PEM)
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t...
    email: ops@bookstore.example
    profile: shortlived
    privateKeySecretRef:
      name: acme-pebble-account
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: bookstore
                namespace: bookstore
                kind: Gateway
                sectionName: http

The profile field has existed since cert-manager 1.18. The gatewayHTTPRoute solver attaches a temporary HTTPRoute to the Gateway's http listener during the challenge. The Gateway gets a single annotation:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: bookstore
  namespace: bookstore
  annotations:
    cert-manager.io/cluster-issuer: acme-pebble
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
    - name: https
      protocol: HTTPS
      port: 443
      hostname: api.bookstore.example
      tls:
        mode: Terminate
        certificateRefs:
          - name: api-bookstore-example-tls

cert-manager's gateway-shim creates a Certificate resource for every listener that has a non-empty hostname, mode: Terminate and a reference to a Secret; the Secret name becomes the Certificate's name. Since api.bookstore.example isn't a real domain, in the lab I added a hosts entry to CoreDNS that resolves the name to Envoy's Service IP, and I ran Pebble inside the cluster. Half a minute after applying:

$ kubectl -n bookstore get certificate,certificaterequest,order
NAME                                                    READY   SECRET                      AGE
certificate.cert-manager.io/api-bookstore-example-tls   True    api-bookstore-example-tls   34s

NAME                                                             APPROVED   DENIED   READY   ISSUER        REQUESTER                                         AGE
certificaterequest.cert-manager.io/api-bookstore-example-tls-1   True                True    acme-pebble   system:serviceaccount:cert-manager:cert-manager   34s

NAME                                                                STATE   AGE
order.acme.cert-manager.io/api-bookstore-example-tls-1-2352046867   valid   34s
$ kubectl -n bookstore get events --field-selector involvedObject.kind=Challenge \
    -o custom-columns=TIME:.lastTimestamp,REASON:.reason,MSG:.message
TIME                   REASON           MSG
2026-11-13T22:23:56Z   Started          Challenge scheduled for processing
2026-11-13T22:23:56Z   Presented        Presented challenge using HTTP-01 challenge mechanism
2026-11-13T22:24:27Z   DomainVerified   Domain "api.bookstore.example" verified with "HTTP-01" validation

The chain is the ACME flow poured into Kubernetes resources: the Certificate creates a CertificateRequest, that creates an Order, and the Order creates a Challenge per name. When troubleshooting you walk down in this order. To catch the route the solver creates while a challenge is in progress, I requested another Certificate for a second name:

$ kubectl -n bookstore get httproute cm-acme-http-solver-bd8dd -o yaml | sed -n '/^spec:/,/^status:/p'
spec:
  hostnames:
  - shop.bookstore.example
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: bookstore
    namespace: bookstore
    sectionName: http
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: cm-acme-http-solver-75f57
      namespace: bookstore
      port: 8089
      weight: 1
    matches:
    - path:
        type: Exact
        value: /.well-known/acme-challenge/FzJpcQV2qLepJyM8BTD8OGgQvqqLPTfj6lqf3wHfW5I
status:

The route sends the exact path of the token to a temporary solver pod, and when the challenge finishes the route, Service and pod are deleted. That's why the http listener's allowedRoutes must accept the solver's namespace. One note: in the lab the Gateway's status is Programmed=False with reason AddressNotAssigned; in kind nothing hands out external IPs to LoadBalancer Services. I ran the tests from inside the cluster via the ClusterIP of Envoy's Service:

$ kubectl -n bookstore get certificate api-bookstore-example-tls -o jsonpath='{.status}' | python3 -m json.tool
{
    "conditions": [
        {
            "lastTransitionTime": "2026-11-13T22:24:27Z",
            "message": "Certificate is up to date and has not expired",
            "observedGeneration": 1,
            "reason": "Ready",
            "status": "True",
            "type": "Ready"
        }
    ],
    "notAfter": "2026-11-19T22:24:26Z",
    "notBefore": "2026-11-13T22:24:27Z",
    "renewalTime": "2026-11-17T22:24:26Z",
    "revision": 1
}
$ curl -sS --cacert /tmp/acme-root.pem https://api.bookstore.example/api/health
ok from api.bookstore.example

renewalTime is when two thirds of the 6-day lifetime have passed, that is day 4. With ACME, cert-manager computes this time from the lifetime the CA actually issued.

Renewal in cert-manager: The Key Changes Every Time

Now the first question from Guess First. I wrote a small script that prints the serial number of the served certificate and its SPKI hash, the value used in public key pinning, and triggered a renewal manually with cmctl renew:

#!/bin/sh
# served certificate serial + SPKI pin (sha256 of public key)
echo | openssl s_client -connect api.bookstore.example:443 -servername api.bookstore.example 2>/dev/null > /tmp/s.txt
printf "%s serial=%s spki=%s\n" "$(date -u +%H:%M:%S)" \
  "$(openssl x509 -in /tmp/s.txt -noout -serial | cut -d= -f2)" \
  "$(openssl x509 -in /tmp/s.txt -noout -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64)"
$ cmctl renew -n bookstore api-bookstore-example-tls
Manually triggered issuance of Certificate bookstore/api-bookstore-example-tls
22:25:10 serial=0EDA07096CB27DF2 spki=A/ssnb8nlizg+uwGXPx7trWqlWSTJANl6L04YlsNtcE=
22:25:11 serial=440C9D30EB1CFCC4 spki=yB+4ybCMS7lY5kgRMYWIPaAd3XnGLLtAnBWWsWRjLjI=
22:25:13 serial=440C9D30EB1CFCC4 spki=yB+4ybCMS7lY5kgRMYWIPaAd3XnGLLtAnBWWsWRjLjI=
...
$ curl -sS --cacert /tmp/acme-root.pem --pinnedpubkey "sha256//A/ssnb8nlizg+uwGXPx7trWqlWSTJANl6L04YlsNtcE=" \
    https://api.bookstore.example/api/health
curl: (90) SSL: public key does not match pinned public key

Within two seconds Envoy started serving the new certificate; no restart or reload, because Envoy Gateway pushes the Secret change to Envoy itself. But the SPKI changed. In cert-manager 1.18 the default rotationPolicy flipped from Never to Always; in 1.19 the feature gate that reverted this default was removed, and the old behavior can now only be had by asking for it explicitly per Certificate. So every renewal generates a new private key. From a security standpoint that's the right choice: a leaked key is useful for at most one certificate lifetime. The answer to Guess First: on the first renewal the mobile app refuses the connection, just as curl failed with code 90. The outage lasts until an app store update ships.

You can keep the key fixed: add the cert-manager.io/private-key-rotation-policy: Never annotation to the Gateway. But that doesn't solve the problem, it postpones it. When Let's Encrypt changes its intermediate, when you change CAs, or when the key leaks, the pin breaks again. On managed platforms it's even clearer: the App Service and Front Door docs say every part of the certificate, including the root, can change without notice, and they explicitly ask you not to pin. If you truly need pinning, you at least have to embed the pin of a backup key in the app as well; but in most mobile and service scenarios in 2026 the right answer is to drop pinning and rely on chain validation and Certificate Transparency monitoring.

cert-manager 1.21 brought ARI support as experimental behind the ACMEUseARI feature gate, and 1.21.3 fixed a loop bug in ARI renewal. The production default is renewal at two thirds of the lifetime; follow the release notes before turning ARI on.

Edge Certificates and Origin Certificates: Who Renews What?

If a CDN such as Front Door or Cloudflare sits in front of you, there are two separate TLS connections, each with its own certificate. The CDN manages the edge certificate the browser sees. The origin certificate the CDN sees when connecting to your origin is your responsibility. Front Door expects the origin to present a fully chained certificate from a CA Microsoft trusts that matches the origin hostname. It doesn't accept internal CA or self-signed certificates. The name check can be turned off, but Microsoft doesn't recommend that from a security standpoint. Cloudflare's own modes and origin certificates trusted only by Cloudflare are a topic for the Domain, DNS and CDN series. In practice one of the tools in this post runs at the origin too: an App Service managed certificate, cert-manager or Caddy. That way both certificates renew automatically. A summary of who is responsible for what:

SetupWho renews the certificateYour jobThe limit that will catch you
Nginx + CertbotCertbot (systemd timer)nginx -s reload via a deploy hook, port 80Forgetting the hook: new certificate on disk, old one in memory
Kestrel + CertbotCertbotFile permissions, Kestrel's reloadKestrel watches only Path
CaddyCaddy itselfKeeping /data persistentA container without a volume hits rate limits
App Service managedApp Service (DigiCert)DNS records, CAANo wildcard, not exportable, CNAME must be direct
App Service + Key VaultKey Vault (integrated CA) or your ACME jobVault access for the resource providerSync takes up to 24 hours
Front Door managedFront Door (DigiCert)Pointing the CNAME directly at the endpointManual revalidation for apex and wildcard
cert-managercert-managerIssuer, solver, and DNS resolving from inside the clusterA new key on every renewal

In Production

Managed doesn't remove monitoring. During the 2025 MPIC change, App Service certificates failed to renew without any alarm; the people who noticed were the ones measuring the served certificate's expiry from the outside. cert-manager exposes its own Prometheus metrics: certmanager_certificate_expiration_timestamp_seconds and certmanager_certificate_ready_status. On top of those, set up a check that measures the served certificate from the outside; we'll use both together in the last post of the series.

In cert-manager, define two separate ClusterIssuers for Let's Encrypt staging and production. Try a new Gateway with the staging issuer first. Failed attempts in production consume rate limits: you get five failed authorizations per name and account per hour.

Keep cert-manager up to date. Besides the ARI loop fix, release 1.21.3 also removed trust in the eMudhra CA, which was reported as compromised, and recommended that all users upgrade. Certificate automation is a dependency too; don't sit on patches.

If you run several Caddy instances, share the storage and only enable on-demand TLS with an ask endpoint. If you use managed certificates on Azure, review DNS changes (such as adding an intermediate record to a CNAME chain) as changes that can break certificate renewal.

Common Pitfalls

Pinning: We saw it above. cert-manager changes the key on every renewal, and managed platforms say they can change any part of the certificate. A client that pins the certificate, the key or the intermediate turns renewal itself into an outage.

cert-manager's self-check: before telling the CA it's ready, cert-manager calls the challenge URL itself from inside the cluster. When I requested the second certificate, CoreDNS hadn't loaded the new entry yet and the challenge waited with this message: Waiting for HTTP-01 challenge propagation: failed to perform self check GET request ... no such host. In production the equivalent is split-horizon DNS: the domain resolves correctly from outside but not from inside the cluster, or resolves to an internal IP. The challenge's REASON column usually spells out the problem.

The apex and wildcard limits of managed certificates: the App Service managed certificate doesn't do wildcards and requires an A record on the apex. If you attach an apex to Front Door with CNAME flattening you have to renew manually, and a wildcard managed certificate is never renewed automatically. A team that trusts the word "managed" and attaches an apex or a wildcard just as casually meets Pending Revalidation on the first renewal.

Forgetting the CAA record: if the domain has a CAA record allowing only letsencrypt.org, the App Service and Front Door managed certificate can't be obtained from DigiCert. If the CAA was added later, the error shows up not when you first add the certificate but on the first renewal.

Running Caddy without a volume: the four runs above showed it. It looks harmless in development; run the same image against Let's Encrypt's production directory and you hit the rate limit.

A Small Experiment

★☆☆ Run Caddy on your own machine for localhost (something like caddy reverse-proxy --from localhost --to :5000). See which CA issued the certificate, find the pki/authorities/local folder, and watch whether Caddy tries to add its root to the trust store.

★★☆ Add the cert-manager.io/private-key-rotation-policy: Never annotation to this post's Gateway, renew twice with cmctl renew, and measure with the script whether the SPKI hash changes. Then look at the spec.privateKey field of the Certificate resource: how did the annotation show up there?

★★★ Set up Caddy and Pebble again. Revoke the certificate as in the third post, with certbot revoke and the key Caddy stored in /data; then watch in the log as Caddy's ARI window changes and the certificate renews by itself. How many hours did you have to wait, and why? (Hint: recheck_after.)

In this post we handed obtaining and renewing certificates over to platforms. In the next post we turn having a certificate into a guarantee: where to do the HTTP-to-HTTPS redirect, rolling out HSTS gradually, the preload decision, and mixed content.