Case Study: Let's Encrypt DNS-01

Last week I shared how IPng Networks deployed a loadbalanced frontend cluster of NGINX webservers that have public IPv4 / IPv6 addresses, but talk to a bunch of internal webservers that are in a private network which isn’t directly connected to the internet, so called IPng Site Local [ref] with addresses 198.19.0.0/16 and 2001:678:d78:500::/56.

I wrote in [that article] that IPng will be using ACME HTTP-01 validation, which asks the certificate authority, in this case Let’s Encrypt, to contact the webserver on a well-known URI for each domain that I’m requesting a certificate for. Unsurprisingly, several folks reached out to me asking “well what about DNS-01”, and one sentence caught their eye:

Some SSL certificate providers allow for wildcards (ie. *.ipng.ch), but I’m going to keep it relatively simple and use [Let’s Encrypt] which offers free certificates with a validity of three months.

I could’ve seen this one coming! The sentence can be read to imply it doesn’t, but of course Let’s Encrypt offers wildcard certificates. It just doesn’t satisfy my relatively simple qualifier of the second part of the sentence … So here I go, down the rabbit hole that is understanding (for myself, and possibly for readers of this article), how the DNS-01 challenge works, in greater detail. Hopefully after writing this (me) and reading this (you), we can all agree that I was wrong, and that using DNS-01 is relatively simple after all.

Overview

I’ve installed three frontend NGINX servers (running at Coloclue AS8283, IPng AS8298 and IP-Max AS25091), and one LEGO certificate machine (running in the internal IPng Site Local network). In the [previous article], I described the setup and the use of Let’s Encrypt with HTTP-01 challenges. I’ll skip that here.

HTTP-01 vs DNS-01

LEGO

Today, most SSL authorities and their customers use the Automatic Certificate Management Environment or ACME protocol which is described in [RFC8555]. It defines a way for certificate authorities to check the websites that they are asked to issue a certificate for using so-called challenges. One popular challenge is the so-called HTTP-01, in which the certificate authority will visit a well-known URI on the website domain for which the certificate is being requested, namely /.well-known/acme-challenge/, which described in [RFC5785]. The CA will expect the webserver to respond with an agreed upon string of numbers at that location, in which case proof of ownership is established and a certificate is issued.

In some situations, this HTTP-01 challenge can be difficult to perform:

  • If the webserver is not reachable from the internet, or not reachable from the Let’s Encrypt servers, for example if it is on an intranet, such as IPng Site Local itself.
  • If the operator would prefer a wildcard certificate, proving ownership of all possible sub-domains is no longer feasible with HTTP-01 but proving ownership of the parent domain is.

One possible solution for these cases is to use the ACME challenge DNS-01, which doesn’t use the webserver running on go.ipng.ch to prove ownership, but the nameserver that serves ipng.ch instead. The Let’s Encrypt GO client [ref] supports both challenges types.

The flow of requests in a DNS-01 challenge is as follows:

ACME Flow DNS01
  1. First, the LEGO client registers itself with the ACME-DNS server running on auth.ipng.ch. After successful registration, LEGO is given a username, password, and access to one DNS recordname $(RRNAME). It is expected that the operator sets up a CNAME for a well-known record _acme-challenge.ipng.ch which points to that $(RRNAME).auth.ipng.ch. This happens only once.

  2. When a certificate is needed, the LEGO client contacts the Certificate Authority and requests validation for the hostname go.ipng.ch. The CA will will inform the client of a random number $(RANDOM) that it expects to see in a a well-known TXT record for _acme-challenge.ipng.ch (which is the CNAME set up previously).

  3. The LEGO client now uses the username and password it received in step 1, to update the TXT record of its $(RRNAME).auth.ipng.ch record to contain the $(RANDOM) number it learned in step 2.

  4. The CA will issue a TXT query for _acme-challenge.ipng.ch, which is a CNAME to $(RRNAME).auth.ipng.ch, which ultimately responds to the TXT query with the $(RANDOM) number.

  5. After validating that the response on the TXT records contains the agreed upon random number, the CA knows that the operator of the nameserver is the same as the certificate requestor for the domain. It issues a certificate to the LEGO client, which stores it on its local filesystem.

  6. Similar to any other challenge, the LEGO machine can now distribute the private key and certificate to all NGINX machines, which are now capable of serving SSL traffic under the given names.

One thing worth noting, is that the TXT query is for domain names, not hostnames, in other words, anything in the ipng.ch domain will solicit a query to _acme-challenge.ipng.ch by the DNS-01 challenge. It is for this reason, that the challenge allows for wildcard certificates, which can greatly reduce operational complexity and the total number of certificates needed.

ACME DNS

Originally, DNS providers were expected to give the ability for their clients to directly update the well-known _acme-challenge TXT record, and while many commercial providers allow for this, IPng Networks runs just plain-old [NSD] as authoritative nameservers (shown above as nsd0, nsd1 and nsd2). So what todo? Luckily, it was quickly understood by the community that if there is a lookup for TXT record of _acme-challenge.ipng.ch, that it would be absolutely OK to make some form of DNS-symlink by means of a CNAME.

One really great solution that leverages this ability is written by Joona Hoikkala, called [ACME-DNS]. It’s sole purpose is to allow for an API, served over https, to register new clients, let those clients update their TXT record(s), and then serve them out in DNS. It’s meant to be a multi-tenant system, by which I mean one ACME-DNS instance can host millions of domains from thousands of distinct users.

Installing

I noticed that ACME-DNS relies on features in relatively modern Go, and the standard version that comes with Debian Bullseye is a tad old, so first I need to install Go v1.19 from backports, before I can continue with the build of the binary:

lego@lego:~$ sudo apt -t bullseye-backports install golang
lego@lego:~/src$ git clone https://github.com/joohoi/acme-dns
lego@lego:~/src/acme-dns$ export GOPATH=/tmp/acme-dns
lego@lego:~/src/acme-dns$ go build
lego@lego:~/src/acme-dns$ sudo cp acme-dns /usr/local/bin/acme-dns
lego@lego:~/src/acme-dns$ cat << EOF | sudo tee /lib/systemd/system/acme-dns.service
[Unit]
Description=Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and
securely
After=network.target

[Service]
User=lego
Group=lego
AmbientCapabilities=CAP_NET_BIND_SERVICE
WorkingDirectory=~
ExecStart=/usr/local/bin/acme-dns -c /home/lego/acme-dns/config.cfg
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

This authoritative nameserver will want to listen on UDP and TCP ports 53, for which it either needs to run as root, or perhaps better, run as non-privileged user with the CAP_NET_BIND_SERVICE capability. The only other difference with the provided unit file, is that I’ll be running this as the lego user, with a configuration file and working path in its home-directory.

Configuring

Step 1. Delegate auth.ipng.ch

The first thing I should do is configure the subdomain for ACME-DNS, which I decide will be hosted on auth.ipng.ch. I assign it an NS, an A and a AAAA record, and then update the ipng.ch domain:

$ORIGIN ipng.ch.
$TTL 86400
@ IN SOA ns.paphosting.net. hostmaster.ipng.ch. ( 2023032401 28800 7200 604800 86400)
                NS      ns.paphosting.nl.
                NS      ns.paphosting.net.
                NS      ns.paphosting.eu.

; ACME DNS
auth            NS      auth.ipng.ch.
                A       194.1.163.93
                AAAA    2001:678:d78:3::93

This snippet will make a DNS delegation for sub-domain auth.ipng.ch to the server also called auth.ipng.ch and because the downstream delegation is in the same domain, I need to provide glue records, that tell clients who are querying for auth.ipng.ch where to find that nameserver. At this point, any request for *.auth.ipng.ch will end up being forwarded to the authoritative nameserver, which can be found at either 194.1.163.93 or 2001:678:d78:3::93.

Step 2. Start ACME DNS

After having built the acme-dns server and given it a suitable systemd unit file, and knowing that it’s going to be responsible for the sub-domain auth.ipng.ch, I give it the following straight forward configuration file:

lego@lego:~$ mkdir ~/acme-dns/
lego@lego:~$ cat << EOF > acme-dns/config.cfg
[general]
listen = "[::]:53"
protocol = "both"
domain = "auth.ipng.ch"
nsname = "auth.ipng.ch"
nsadmin = "hostmaster.ipng.ch"
records = [
    "auth.ipng.ch. NS auth.ipng.ch.",
    "auth.ipng.ch. A 194.1.163.93",
    "auth.ipng.ch. AAAA 2001:678:d78:3::93",
]
debug = false

[database]
engine = "sqlite3"
connection = "/home/lego/acme-dns/acme-dns.db"

[api]
ip = "[::]"
disable_registration = false
port = "443"
tls = "letsencrypt"
acme_cache_dir = "/home/lego/acme-dns/api-certs"
notification_email = "hostmaster+dns-auth@ipng.ch"
corsorigins = [ "*" ]
use_header = false
header_name = "X-Forwarded-For"

[logconfig]
loglevel = "debug"
logtype = "stdout"
logformat = "text"
EOF
lego@lego:~$ sudo systemctl enable acme-dns
lego@lego:~$ sudo systemctl start acme-dns

The first part of this tells the server how to construct the SOA record (domain, nsname and nsadmin), and which records to put in the apex, nominally the NS/A/AAAA records that describe the nameserver which is authoritative for the auth.ipng.ch domain. Then, the database part is where user credentials will be stored, and the API portion shows how users will be able to interact with the controlplane part of the service, notably registering new clients, and updating nameserver TXT records for existing clients.

Turtles

Interestingly, the API is served on HTTPS port 443, and for that it needs, you guessed it, a certificate! ACME-DNS eats its own dogfood, which I can appreciate: it will use DNS-01 validation to get a certificate for auth.ipng.ch itself, by serving the challenge for well known record _acme-challenge.auth.ipng.ch, so it’s turtles all the way down!

Step 3. Register a new client

Seeing as many public DNS providers allow programmatic setting of the contents of the zonefiles, for them it’s a matter of directly being driven by LEGO. But for me, running NSD, I am going to be using the ACME DNS server to fulfill that purpose, so I have to configure it to do that for me.

In the explanation of DNS-01 challenges above, you’ll remember I made a mention of registering. Here’s a closer look at what that means:

lego@lego:~$ curl -s -X POST https://auth.ipng.ch/register | json_pp
{
   "allowfrom" : [],
   "fulldomain" : "76f88564-740b-4483-9bc0-86d1fb531e20.auth.ipng.ch",
   "password" : "<redacted>",
   "subdomain" : "76f88564-740b-4483-9bc0-86d1fb531e20",
   "username" : "e4608fdf-9a69-4930-8cf1-57218738792d"
}

What happened here is that, using the HTTPS endpoint, I asked the ACME-DNS server to create for me an empty DNS record, which it did on 76f88564-740b-4483-9bc0-86d1fb531e20.auth.ipng.ch. Further, if I offer the given username and password, I am able to update that record’s value. Let’s take a look:

lego@lego:~$ dig +short TXT 02e3acfc-bbca-46bb-9cee-8eab52c73c30.auth.ipng.ch

lego@lego:~$ curl -s -X POST -H "X-Api-User: 5f3591d1-0d13-4816-a329-7965a8639ab5" \
  -H "X-Api-Key: <redacted>" \
  -d '{"subdomain": "02e3acfc-bbca-46bb-9cee-8eab52c73c30", \
       "txt": "___Hello_World_token_______________________"}' \
  https://auth.ipng.ch/update

Numbers everywhere, but I learned a lot here! Notice how the first time I sent the dig request for the 02e3acfc-bbca-46bb-9cee-8eab52c73c30.auth.ipng.ch it did not respond anything (an empty record). But then, using the username/password I could update the record with a 41 character string, and I was informed of the fulldomain key there, which is the one that I should be configuring in the domain(s) for which I want to get a certificate.

I configure it in the ipng.ch and ipng.nl domain as follows (taking ipng.nl as an example):

$ORIGIN ipng.nl.
$TTL 86400
@ IN SOA ns.paphosting.net. hostmaster.ipng.nl. ( 2023032401 28800 7200 604800 86400)
                IN  NS ns.paphosting.nl.
                IN  NS ns.paphosting.net.
                IN  NS ns.paphosting.eu.
                CAA     0 issue "letsencrypt.org"
                CAA     0 issuewild "letsencrypt.org"
                CAA     0 iodef "mailto:hostmaster@ipng.ch"
_acme-challenge CNAME   8ee2969b-571c-4b3a-b6a0-6d6221130c96.auth.ipng.ch.

The records here are a CAA which is a type of DNS record used to provide additional confirmation for the Certificate Authority when validating an SSL certificate. This record allows me to specify which certificate authorities are authorized to deliver SSL certificates for the domain. Then, the well known _acme-challenge.ipng.nl record is merely telling the client by means of a CNAME to go ask for 8ee2969b-571c-4b3a-b6a0-6d6221130c96.auth.ipng.ch instead.

Putting this part all together now, I can issue a query for that ipng.nl domain …

lego@lego:~$ dig +short TXT _acme-challenge.ipng.nl.
"___Hello_World_token_______________________"

… and would you look at that! The query for the ipng.nl domain, is a CNAME to the specific uuid record in the auth.ipng.ch domain, where ACME-DNS is serving it with the response that I can programmatically set to different values, yee-haw!

Step 4. Run LEGO

The LEGO client has all sorts of challenge providers linked in. Once again, Debian is a bit behind on things, shipping version 3.2.0-3.1+b5 in Bullseye, although upstream is much further along. So I purge the Debian package and download the v4.10.2 amd64 package directly from its [Github] releases page. The ACME-DNS handler was only added in v4 of the client. But now all that’s left for me to do is run it:

lego@lego:~$ export ACME_DNS_API_BASE=https://auth.ipng.ch/
lego@lego:~$ export ACME_DNS_STORAGE_PATH=/home/lego/acme-dns/credentials.json
lego@lego:~$ /home/lego/bin/lego --path /etc/lego/ --email noc@ipng.ch --accept-tos --dns acme-dns \
  --domains ipng.ch --domains *.ipng.ch \
  --domains ipng.nl --domains *.ipng.nl \
  run

The LEGO client goes through the ACME flow that I described at the top of this article, and ends up spitting out a certificate \o/

lego@lego:~$ openssl x509 -noout -text -in /etc/lego/certificates/ipng.ch.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:58:8f:c1:25:00:e2:f3:d3:3f:d6:ed:ba:bc:1d:0d:54:ea
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = R3
        Validity
            Not Before: Mar 21 20:24:08 2023 GMT
            Not After : Jun 19 20:24:07 2023 GMT
        Subject: CN = ipng.ch
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:*.ipng.ch, DNS:*.ipng.nl, DNS:ipng.ch, DNS:ipng.nl

Et voila! Wildcard certificates for multiple domains using ACME-DNS.