Monitoring Certificates and Preventing Outages: Inventory, Expiry Alerts, TLS Testing and Real Incident Examples

Version note: The examples in this post were run with Prometheus blackbox_exporter 0.29.1, Prometheus 3.16 (promtool), testssl.sh 3.2.5, Nginx 1.30.6 and OpenSSL 3.5 on Docker 29; crt.sh and DNS data were retrieved on 14 November 2026.

Guess First

Two messages arrive on Monday morning. The first is from customer support: "The site shows a certificate error." Yet Certbot's overnight log says the renewal succeeded, and the fullchain.pem on disk expires 47 days from now. The second is from monitoring: an "earliest certificate expiry: 2 days" warning for api.bookstore.example. But anyone who looks at the certificate in a browser sees it's valid for another 46 days.

Guess: in the first incident, was the certificate really renewed, and where's the problem? Is the second warning a false alarm, or a real problem that will hurt someone? We'll reproduce both in the lab.

Inventory: Which Certificate Is Where, and Who Owns It?

You can't renew a certificate you don't monitor, and you can't monitor one you don't know exists. In the series' earlier posts Bookstore's certificates lived in seven different places: Nginx + Certbot, Kestrel, Caddy, App Service, Front Door, cert-manager and the internal CA's partner certificates. Every inventory row should answer these questions: which names (SAN, Subject Alternative Name), which CA and profile, who renews it (the tool and the team that owns it), where it's installed (which server, Secret, binding), who would notice that it was renewed, and when it expires. The last two are the ones most often left blank.

Instead of keeping the inventory by hand, collect it from the sources. On cert-manager kubectl get certificate -A, on Azure a Resource Graph query, on Certbot certbot certificates. You also need a source that looks from the outside: Certificate Transparency (CT) logs. In the second post we listed a domain's certificates with crt.sh. To turn that into a check, I wrote a small script that compares the issuers in CT with an allow list. As an example I checked IANA's test domain example.com as if we allowed only Sectigo:

#!/bin/sh
# compare issuers of unexpired certificates in CT logs with an allow list
domain=$1; allowed=$2
curl -sf --retry 3 "https://crt.sh/?q=${domain}&output=json&exclude=expired&deduplicate=Y" \
  | jq -r --arg allowed "$allowed" '
      group_by(.issuer_name)[]
      | {issuer: .[0].issuer_name, count: length, newest: (map(.not_before) | max)}
      | (if (.issuer | test($allowed)) then "ok        " else "UNEXPECTED" end)
        + "  \(.count)  newest=\(.newest)  \(.issuer)"'

$ ./ct-check.sh example.com "Sectigo"
ok          1  newest=2026-09-24T00:00:00  C=GB, O=Sectigo Limited, CN=Sectigo Public Server Authentication CA DV E36
ok          3  newest=2025-12-02T00:00:00  C=GB, O=Sectigo Limited, CN=Sectigo Public Server Authentication CA OV R36
UNEXPECTED  1  newest=2021-08-31T15:55:23  C=GB, ST=Cymru, O=AS207960 Cyfyngedig, OU=AS207960 Department of Random Numbers, CN=AS207960 Test CA
UNEXPECTED  2  newest=2026-09-26T22:49:11  C=US, O=SSL Corporation, CN=Cloudflare TLS Issuing ECC CA 3
UNEXPECTED  1  newest=2023-01-27T01:21:18  Issuer Not Found

A line like that on your own domain points either to a team that isn't in the inventory (such as a marketing site that moved to a CDN) or to someone obtaining certificates in your name. You need to know about both. You can run this script as a daily job and wire the UNEXPECTED lines to an alert, or use a CT monitoring service that does the same. CT only reports what happened; to prevent it there's the CAA record. In DNS you write which CAs may issue certificates for your domain:

$ dig +short CAA google.com
0 issue "pki.goog"

The App Service and Front Door managed certificates in the fifth post came from DigiCert. If you forget to add it to the CAA record, renewal breaks. Setting up CAA records is a topic for the Domain, DNS and CDN series.

Measuring the Served Certificate from the Outside: Blackbox Exporter

The inventory says "what should be". Monitoring must measure "what is", meaning the certificate the client actually sees. Looking at the file on disk, cert-manager's Secret or the ACME client's log isn't enough; all of them can be right while the server still serves the old certificate. Prometheus's blackbox exporter sends a real HTTPS (HTTP Secure) request to the target and exposes the TLS information as metrics:

modules:
  https_2xx:
    prober: http
    timeout: 5s
    http:
      fail_if_not_ssl: true
      tls_config:
        ca_file: /certs/root.crt

ca_file is there only for the lab's test CA. For public certificates you don't write this line and the exporter uses the system trust store. Prometheus normally scrapes the /probe endpoint; I called the same endpoint by hand:

#!/bin/sh
# ask the blackbox exporter to probe a target and print the TLS metrics
docker exec tlsseries-blackbox wget -qO- "http://localhost:9115/probe?module=https_2xx&target=$1" \
  | grep -E '^probe_(success|ssl_earliest_cert_expiry|ssl_last_chain_expiry_timestamp_seconds|ssl_last_chain_info|tls_version_info)'

$ ./probe.sh https://www.bookstore.example
probe_ssl_earliest_cert_expiry 1.795474915e+09
probe_ssl_last_chain_expiry_timestamp_seconds 1.795474915e+09
probe_ssl_last_chain_info{fingerprint_sha256="f572df4f37dffeeec1b92c908a8c6e06df6e096225c43d2ab780be4c7a0cde1c",issuer="CN=Bookstore Lab Issuing CA",serialnumber="073426772d270b25ba70e17801befddb98387be9",subject="CN=www.bookstore.example",subjectalternative="www.bookstore.example,bookstore.example"} 1
probe_success 1
probe_tls_version_info{version="TLS 1.3"} 1

Both timestamps are Unix epoch: 1795474915 is 23 November 2026 23:01:55 UTC (Coordinated Universal Time). probe_ssl_last_chain_info carries the served leaf's serial number and SAN list as labels; we'll use it to catch the "renewed but not served" problem. probe_success matters at least as much as expiry: if the chain can't be validated or the name doesn't match, the probe fails and no expiry metrics are produced at all.

earliest_cert_expiry or last_chain_expiry?

The second question from Guess First. The difference between the two metrics is a single line in the exporter's source code. probe_ssl_earliest_cert_expiry is the earliest expiry among all certificates the server sent (PeerCertificates). probe_ssl_last_chain_expiry_timestamp_seconds finds the earliest expiry within each chain the client could verify (VerifiedChains) and takes the best of those. To see the difference, in the lab I had an old root cross-sign Bookstore's root and gave that cross certificate a two-day lifetime. The server sends the leaf, the intermediate and the cross certificate together:

$ cat new.crt int.crt root-cross.crt > live/fullchain.pem && nginx -s reload
$ ./probe.sh https://www.bookstore.example | grep -E "success|expiry"
probe_ssl_earliest_cert_expiry 1.794783715e+09
probe_ssl_last_chain_expiry_timestamp_seconds 1.798671715e+09
probe_success 1

The first is 15 November 2026, the second 30 December 2026. A client that trusts the root directly builds the chain without ever using the cross certificate and has no trouble. That's exactly the warning from Guess First. But don't rush to call it a false alarm. Clients that don't trust the root directly and only know the old root build the chain through the cross certificate and break two days later. That's exactly what happened in 2021. Let's Encrypt's old root, IdenTrust's DST Root CA X3, expired on 30 September 2021. Most servers also sent the cross-sign of ISRG Root X1 made by that old root in their chain. Modern clients picked the short path, but clients like OpenSSL 1.0.2 preferred the long chain that was sent and got stuck on the expired root. Let's Encrypt removed the cross-sign from the default chain on 8 February 2024, and the cross-sign expired on 30 September 2024; old Android devices were hit hardest. Use last_chain for alerting. Watch earliest as low-priority information along the lines of "the server sends an unnecessary or soon-to-expire certificate".

Alert Rules and Testing Them with promtool

The rules that turn the metrics into alerts:

groups:
  - name: tls-certificates
    rules:
      - alert: TLSProbeFailing
        expr: probe_success{job="blackbox-tls"} == 0
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "TLS probe failing for {{ $labels.instance }} (chain, name or connection)"
      - alert: TLSCertificateExpiringSoon
        expr: (probe_ssl_last_chain_expiry_timestamp_seconds{job="blackbox-tls"} - time()) / 86400 < 10
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} certificate expires in {{ $value | humanize }} days; renewal did not happen"
      - alert: TLSCertificateExpiryCritical
        expr: (probe_ssl_last_chain_expiry_timestamp_seconds{job="blackbox-tls"} - time()) / 86400 < 3
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} certificate expires in {{ $value | humanize }} days"
      - alert: TLSServedChainHasExpiringExtraCertificate
        expr: |
          (probe_ssl_earliest_cert_expiry{job="blackbox-tls"} - time()) / 86400 < 10
          and (probe_ssl_last_chain_expiry_timestamp_seconds{job="blackbox-tls"} - time()) / 86400 >= 10
        for: 1h
        labels:
          severity: info
        annotations:
          summary: "{{ $labels.instance }} sends a certificate outside the verified chain that expires soon"

The thresholds rest on the automation. Without ARI, a 47-day certificate is renewed when a third of its lifetime remains, about 16 days before expiry. If it has dropped below 10 days, renewal has been failing for at least six days. That's something a human needs to look at, but it's not a fire yet. 3 days is a fire. Before shipping the rules I wrote unit tests with promtool. The scenarios match the lab: www with stuck renewal, api carrying a soon-to-expire cross certificate in its chain, shop with the wrong SAN:

rule_files:
  - tls-alerts.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      # www: renewal is stuck, 9 days left
      - series: 'probe_ssl_last_chain_expiry_timestamp_seconds{job="blackbox-tls",instance="https://www.bookstore.example"}'
        values: '777600x180'
      - series: 'probe_ssl_earliest_cert_expiry{job="blackbox-tls",instance="https://www.bookstore.example"}'
        values: '777600x180'
      # api: chain fine for 46 days, but an extra cross-signed certificate expires in 2 days
      - series: 'probe_ssl_last_chain_expiry_timestamp_seconds{job="blackbox-tls",instance="https://api.bookstore.example"}'
        values: '3974400x180'
      - series: 'probe_ssl_earliest_cert_expiry{job="blackbox-tls",instance="https://api.bookstore.example"}'
        values: '172800x180'
      # shop: wrong SAN, probe fails
      - series: 'probe_success{job="blackbox-tls",instance="https://shop.bookstore.example"}'
        values: '0x180'
    alert_rule_test:
      - eval_time: 2h
        alertname: TLSCertificateExpiringSoon
        exp_alerts:
          - exp_labels:
              severity: warning
              job: blackbox-tls
              instance: https://www.bookstore.example
            exp_annotations:
              summary: "https://www.bookstore.example certificate expires in 8.917 days; renewal did not happen"
      - eval_time: 2h
        alertname: TLSCertificateExpiryCritical
        exp_alerts: []
      - eval_time: 2h
        alertname: TLSServedChainHasExpiringExtraCertificate
        exp_alerts:
          - exp_labels:
              severity: info
              job: blackbox-tls
              instance: https://api.bookstore.example
            exp_annotations:
              summary: "https://api.bookstore.example sends a certificate outside the verified chain that expires soon"
      - eval_time: 15m
        alertname: TLSProbeFailing
        exp_alerts:
          - exp_labels:
              severity: critical
              job: blackbox-tls
              instance: https://shop.bookstore.example
            exp_annotations:
              summary: "TLS probe failing for https://shop.bookstore.example (chain, name or connection)"

$ promtool check rules tls-alerts.yml
Checking tls-alerts.yml
  SUCCESS: 4 rules found

$ promtool test rules tls-alerts.test.yml
Processing test file: tls-alerts.test.yml
  SUCCESS

In the test, time starts at zero, so 777600 means "expires in nine days". Fixed thresholds have a limit: with 6-day certificates a 10-day threshold alerts all the time. Separate short-lived certificates into their own job or by a label, and pick the threshold according to the lifetime; for example 2 days for the 6-day ones. Routing, silencing and runbook links for alerts are the subject of the alerting post in the Observability series.

External measurement tells you the outcome; the ACME client itself knows the cause. Wire that to alerts too: on cert-manager the certmanager_certificate_ready_status{condition="False"} metric we saw in the fifth post, on Caddy the could not get certificate from issuer error line of the tls.obtain logger, on Certbot the systemd unit's failed state. That way you hear about it the day after the first failed attempt, not 10 days before the certificate expires.

Testing the TLS Config: testssl.sh and Mozilla's Profiles

A certificate can be valid while the TLS settings are outdated. Online scanners like SSL Labs only scan servers reachable from the internet. testssl.sh runs the same checks against a server on the internal network too, and it can go into CI. I scanned this config, copied from an old server:

ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5:@SECLEVEL=0;

$ docker run --rm --network tlsseries-net8 ghcr.io/testssl/testssl.sh:3.2 --quiet --color 0 -p -s www.bookstore.example:443
 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      offered (deprecated)
 TLS 1.1    offered (deprecated)
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NULL ciphers (no encryption)                      not offered (OK)
 Anonymous NULL Ciphers (no authentication)        not offered (OK)
 Export ciphers (w/o ADH+NULL)                     not offered (OK)
 LOW: 64 Bit + DES, RC[2,4], MD5 (w/o export)      not offered (OK)
 Triple DES Ciphers / IDEA                         not offered
 Obsoleted CBC ciphers (AES, ARIA etc.)            offered
 Strong encryption (AEAD ciphers) with no FS       not offered
 Forward Secrecy strong encryption (AEAD ciphers)  offered (OK)

TLS 1.0 and 1.1 are on. Both were formally deprecated by RFC 8996 in 2021, and modern browsers no longer connect with them. @SECLEVEL=0 is a tell too: OpenSSL 3 already refuses TLS 1.0 at its default security level; the config forces it open. Mozilla's SSL Configuration Generator offers ready-made profiles. The current guidelines (version 6.0) have only two profiles: modern (TLS 1.3 only) and intermediate (TLS 1.2 and 1.3). The old profile that used to exist is gone. I wrote the intermediate profile with the values from the guidelines' JSON (JavaScript Object Notation) output:

# Mozilla "intermediate" (guidelines 6.0)
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ecdh_curve X25519MLKEM768:X25519:prime256v1:secp384r1;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305;
ssl_prefer_server_ciphers off;

$ docker run --rm --network tlsseries-net8 ghcr.io/testssl/testssl.sh:3.2 --quiet --color 0 -p -s -f www.bookstore.example:443
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 Obsoleted CBC ciphers (AES, ARIA etc.)            not offered
 Forward Secrecy strong encryption (AEAD ciphers)  offered (OK)
 KEMs offered                 X25519MLKEM768
 Elliptic curves offered:     prime256v1 secp384r1 X25519 

(I kept the lines that changed or are new.) TLS 1.0/1.1 and the CBC ciphers are off, and post-quantum hybrid key exchange (X25519MLKEM768) is on. The oldest clients the intermediate profile supports include Android 4.4.2, Internet Explorer 11 on Windows 10 and Java 8u161; if you have no client older than these, this is the right choice. The rest of HTTPS in Nginx is in the Web Servers and Hosting series post "HTTPS, Security Headers, Rate Limiting and Load Balancing in Nginx".

testssl.sh looks at the certificate too. After adding the lab root and running with -S, these lines stand out:

 Chain of trust               Ok
 Certificate Validity (UTC)   expires < 60 days (46) (2026-11-13 23:01 --> 2026-12-30 23:01)
 Certificate Revocation List  --
 OCSP URI                     --
                              NOT ok -- neither CRL nor OCSP URI provided

The "expires < 60 days" warning will show up for every certificate in the 47-day era. The tools' thresholds were set for yearly certificates; choose your own threshold based on the lifetime. The missing CRL or OCSP address comes from the lab's test CA. If you see this line on a public certificate, check the CA's details; as we saw in the second post, Let's Encrypt now provides only CRLs.

Real Incident Types and a Short Runbook

Renewed but Not Reloaded

The first incident from Guess First. While Nginx was running with the old certificate, the renewal script put the new files in place but didn't send Nginx a reload:

$ openssl x509 -in live/fullchain.pem -noout -serial -enddate
serial=073426772D270B25BA70E17801BEFDDB98387BEA
notAfter=Dec 30 23:01:55 2026 GMT
$ ./probe.sh https://www.bookstore.example | grep -E "earliest|serialnumber" | sed -E 's/fingerprint_sha256="[^"]*",//'
probe_ssl_earliest_cert_expiry 1.795474915e+09
probe_ssl_last_chain_info{issuer="CN=Bookstore Lab Issuing CA",serialnumber="073426772d270b25ba70e17801befddb98387be9",subject="CN=www.bookstore.example",subjectalternative="www.bookstore.example,bookstore.example"} 1
$ nginx -s reload
$ ./probe.sh https://www.bookstore.example | grep -E "earliest|serialnumber" | sed -E 's/fingerprint_sha256="[^"]*",//'
probe_ssl_earliest_cert_expiry 1.798671715e+09
probe_ssl_last_chain_info{issuer="CN=Bookstore Lab Issuing CA",serialnumber="073426772d270b25ba70e17801befddb98387bea",subject="CN=www.bookstore.example",subjectalternative="www.bookstore.example,bookstore.example"} 1

On disk ...BEA, served ...BE9. The renewal succeeded; Nginx reads the certificate only at startup and on reload. The answer: the certificate was renewed, but nobody told the server. The --deploy-hook from the third post exists exactly for this. Even though Kestrel has its own file watcher, as we saw in the fourth post it watches only Path. You catch this incident by comparing the served serial number with the one on disk, or simply with the expiry alert.

Missing Intermediate and Wrong SAN

When the server sends only the leaf, and when the same certificate is served under another name:

$ ./probe.sh https://www.bookstore.example
probe_success 0
... err="Get \"https://172.20.0.2\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
$ ./probe.sh https://shop.bookstore.example
probe_success 0
... err="Get \"https://172.20.0.2\": tls: failed to verify certificate: x509: certificate is valid for www.bookstore.example, bookstore.example, not shop.books

(The error lines are from the exporter's debug=true output, shortened.) In the first post we saw a missing intermediate work in a desktop browser and break in curl; Go's TLS client doesn't download a missing intermediate through AIA either, so the blackbox exporter catches this error before browsers do. In both cases there's no expiry metric; only the TLSProbeFailing alert fires. When a new name is added (a shop subdomain), forgetting to add it to the certificate is one of the most common deploy mistakes.

Root Expiry and Clock Skew

We saw root expiry above with DST Root CA X3. Clock skew matters more with short-lived certificates. I validated a freshly issued 6-day certificate from the point of view of a client whose clock is two days behind and one whose clock is nine days ahead:

$ openssl x509 -in shortlived.crt -noout -startdate -enddate
notBefore=Nov 13 23:03:01 2026 GMT
notAfter=Nov 19 23:03:01 2026 GMT
$ openssl verify -CAfile r.crt -attime $((NOW - 2*86400)) shortlived.crt
CN=www.bookstore.example
error 9 at 0 depth lookup: certificate is not yet valid
error shortlived.crt: verification failed
$ openssl verify -CAfile r.crt -attime $((NOW + 7*86400)) shortlived.crt
CN=www.bookstore.example
error 10 at 0 depth lookup: certificate has expired
error shortlived.crt: verification failed

With yearly certificates a two-day skew would go unnoticed; with a 6-day one, that client can't connect right after a renewal. Devices whose clocks aren't synchronized (kiosks, warehouse scanners, machines left off for a long time) and test environments that tamper with the clock inside a container produce this error. If the error message is "certificate is not yet valid", look at the client's clock first.

Runbook

• See the served certificate: serial number, notAfter, SAN and the chain sent, with openssl s_client -connect host:443 -servername host -showcerts.
• Compare with the one on disk: if the serial numbers differ, it's a reload or deploy hook problem.
• Validate the chain: is the intermediate missing, is an unnecessary or expired cross certificate being sent?
• Name: is the requested host in the SAN list, and which certificate comes back without SNI?
• Look at the ACME client's log: challenge errors, rate limits, CAA refusal, DNS API permissions.
• In an emergency, obtain and install the certificate by hand, but don't close the incident without fixing the automation; the next renewal will break in the same place.

In Production

As lifetimes shrink to 47 days by 2029, a task done once a year turns into a weekly one. If you can fill in this list for every certificate, you're ready:

• Automation: a tool renews every certificate, and installation (reload, binding, Secret) is automatic too.
• ARI: clients support ARI; if the CA asks for early renewal, nobody has to log in to a server.
• External monitoring: a blackbox probe for every name, a lifetime-based threshold on last_chain, plus a probe_success alert.
• ACME errors: a client's failed attempt becomes an alert on day one.
• No pinning: no client left that pins the certificate, the key or the intermediate.
• Inventory and CT: an issuer allow list, a check for unexpected issuance and an up-to-date CAA record.
• Clocks: NTP synchronization on servers and devices in the field.

Monitor the monitoring system too. The blackbox exporter's own certificate, Prometheus's target list and Alertmanager's routing can break as well. Regularly test that the alert chain works end to end with a deliberately misnamed target.

Common Pitfalls

Monitoring the file on disk: a cron that runs openssl x509 -enddate against the file sees the unreloaded server as "healthy". In the first incident above, the certificate on disk was valid for 47 days. Monitoring must measure what the client sees.

Alerting on probe_ssl_earliest_cert_expiry: on a server with an unnecessary cross certificate it produces constant false alarms. Once the team gets used to silencing it, they miss the real one. Critical alerts on last_chain, earliest as low-priority information.

Trusting expiry while the probe fails: on a chain or name error no expiry metric is produced, and an absent series doesn't trigger an expiry alert. Without a probe_success == 0 alert, in the worst case no alert fires at all.

Monitoring only the main domain: every name in the inventory, including internal services and partner endpoints, should be a separate target. Client certificates like the partner certificates in the seventh post can't be measured with blackbox; monitor them on the issuer side, where they're issued.

Keeping old tool thresholds: a 30-day "expiring soon" threshold becomes an alert that always fires with 47-day certificates and a meaningless one with 6-day certificates.

A Small Experiment

★☆☆ Run ./ct-check.sh on your own site and look at the CAA record. Are all the issuers you see in CT CAs that the CAA allows?

★★☆ Wire this post's blackbox config and alert rules to a Prometheus, use the Caddy and Pebble setup from the fifth post as a target, and write one more rule with a separate threshold for 6-day certificates. Add a 6-day scenario to the promtool test.

★★★ In the lab, reproduce the four incidents (an unreloaded renewal, a missing intermediate, a wrong SAN, a soon-to-expire cross certificate) one after another and measure which alert fires for each and how long it takes. Which one did no alert catch?

The series ends here. We started inside a certificate and went through obtaining it, installing it, renewing it automatically, enforcing HTTPS, client identity and monitoring. What comes next lives in two neighboring series: the Domain, DNS and CDN series for the DNS records and CDNs certificates depend on, and the Observability series for the infrastructure behind this post's alerts.