Last week I shared how IPng Networks deployed a loadbalanced frontend cluster of NGINX webservers that have public IPv4 / IPv6 addresses, but talk to a bunch of internal webservers that are in a private network which isn’t directly connected to the internet, so called IPng Site Local [ref] with addresses 198.19.0.0/16 and 2001:678:d78:500::/56.
I wrote in [that article] that IPng will be using ACME HTTP-01 validation, which asks the certificate authority, in this case Let’s Encrypt, to contact the webserver on a well-known URI for each domain that I’m requesting a certificate for. Unsurprisingly, several folks reached out to me asking “well what about DNS-01”, and one sentence caught their eye:
Some SSL certificate providers allow for wildcards (ie.
*.ipng.ch), but I’m going to keep it relatively simple and use [Let’s Encrypt] which offers free certificates with a validity of three months.
I could’ve seen this one coming! The sentence can be read to imply it doesn’t, but of course Let’s Encrypt offers wildcard certificates. It just doesn’t satisfy my relatively simple qualifier of the second part of the sentence … So here I go, down the rabbit hole that is understanding (for myself, and possibly for readers of this article), how the DNS-01 challenge works, in greater detail. Hopefully after writing this (me) and reading this (you), we can all agree that I was wrong, and that using DNS-01 is relatively simple after all.
Overview
I’ve installed three frontend NGINX servers (running at Coloclue AS8283, IPng AS8298 and IP-Max AS25091), and one LEGO certificate machine (running in the internal IPng Site Local network). In the [previous article], I described the setup and the use of Let’s Encrypt with HTTP-01 challenges. I’ll skip that here.
HTTP-01 vs DNS-01
Today, most SSL authorities and their customers use the Automatic Certificate Management Environment
or ACME protocol which is described in [RFC8555]. It
defines a way for certificate authorities to check the websites that they are asked to issue a
certificate for using so-called challenges. One popular challenge is the so-called HTTP-01, in
which the certificate authority will visit a well-known URI on the website domain for which the
certificate is being requested, namely /.well-known/acme-challenge/, which described in
[RFC5785]. The CA will expect the webserver to respond
with an agreed upon string of numbers at that location, in which case proof of ownership is
established and a certificate is issued.
In some situations, this HTTP-01 challenge can be difficult to perform:
- If the webserver is not reachable from the internet, or not reachable from the Let’s Encrypt servers, for example if it is on an intranet, such as IPng Site Local itself.
- If the operator would prefer a wildcard certificate, proving ownership of all possible
sub-domains is no longer feasible with
HTTP-01but proving ownership of the parent domain is.
One possible solution for these cases is to use the ACME challenge DNS-01, which doesn’t use the
webserver running on go.ipng.ch to prove ownership, but the nameserver that serves ipng.ch
instead. The Let’s Encrypt GO client [ref] supports both
challenges types.
The flow of requests in a DNS-01 challenge is as follows:
-
First, the LEGO client registers itself with the ACME-DNS server running on
auth.ipng.ch. After successful registration, LEGO is given a username, password, and access to one DNS recordname $(RRNAME). It is expected that the operator sets up a CNAME for a well-known record_acme-challenge.ipng.chwhich points to that$(RRNAME).auth.ipng.ch. This happens only once. -
When a certificate is needed, the LEGO client contacts the Certificate Authority and requests validation for the hostname
go.ipng.ch. The CA will will inform the client of a random number $(RANDOM) that it expects to see in a a well-known TXT record for_acme-challenge.ipng.ch(which is the CNAME set up previously). -
The LEGO client now uses the username and password it received in step 1, to update the TXT record of its
$(RRNAME).auth.ipng.chrecord to contain the $(RANDOM) number it learned in step 2. -
The CA will issue a TXT query for
_acme-challenge.ipng.ch, which is a CNAME to$(RRNAME).auth.ipng.ch, which ultimately responds to the TXT query with the $(RANDOM) number. -
After validating that the response on the TXT records contains the agreed upon random number, the CA knows that the operator of the nameserver is the same as the certificate requestor for the domain. It issues a certificate to the LEGO client, which stores it on its local filesystem.
-
Similar to any other challenge, the LEGO machine can now distribute the private key and certificate to all NGINX machines, which are now capable of serving SSL traffic under the given names.
One thing worth noting, is that the TXT query is for domain names, not hostnames, in other
words, anything in the ipng.ch domain will solicit a query to _acme-challenge.ipng.ch by the
DNS-01 challenge. It is for this reason, that the challenge allows for wildcard certificates,
which can greatly reduce operational complexity and the total number of certificates needed.
ACME DNS
Originally, DNS providers were expected to give the ability for their clients to directly update the
well-known _acme-challenge TXT record, and while many commercial providers allow for this, IPng
Networks runs just plain-old [NSD] as authoritative
nameservers (shown above as nsd0, nsd1 and nsd2). So what todo? Luckily, it was quickly
understood by the community that if there is a lookup for TXT record of _acme-challenge.ipng.ch,
that it would be absolutely OK to make some form of DNS-symlink by means of a CNAME.
One really great solution that leverages this ability is written by Joona Hoikkala, called [ACME-DNS]. It’s sole purpose is to allow for an API, served over https, to register new clients, let those clients update their TXT record(s), and then serve them out in DNS. It’s meant to be a multi-tenant system, by which I mean one ACME-DNS instance can host millions of domains from thousands of distinct users.
Installing
I noticed that ACME-DNS relies on features in relatively modern Go, and the standard version that comes with Debian Bullseye is a tad old, so first I need to install Go v1.19 from backports, before I can continue with the build of the binary:
lego@lego:~$ sudo apt -t bullseye-backports install golang
lego@lego:~/src$ git clone https://github.com/joohoi/acme-dns
lego@lego:~/src/acme-dns$ export GOPATH=/tmp/acme-dns
lego@lego:~/src/acme-dns$ go build
lego@lego:~/src/acme-dns$ sudo cp acme-dns /usr/local/bin/acme-dns
lego@lego:~/src/acme-dns$ cat << EOF | sudo tee /lib/systemd/system/acme-dns.service
[Unit]
Description=Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and
securely
After=network.target
[Service]
User=lego
Group=lego
AmbientCapabilities=CAP_NET_BIND_SERVICE
WorkingDirectory=~
ExecStart=/usr/local/bin/acme-dns -c /home/lego/acme-dns/config.cfg
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
This authoritative nameserver will want to listen on UDP and TCP ports 53, for which it either needs to
run as root, or perhaps better, run as non-privileged user with the CAP_NET_BIND_SERVICE
capability. The only other difference with the provided unit file, is that I’ll be running this as
the lego user, with a configuration file and working path in its home-directory.
Configuring
Step 1. Delegate auth.ipng.ch
The first thing I should do is configure the subdomain for ACME-DNS, which I decide will be hosted on
auth.ipng.ch. I assign it an NS, an A and a AAAA record, and then update the ipng.ch domain:
$ORIGIN ipng.ch.
$TTL 86400
@ IN SOA ns.paphosting.net. hostmaster.ipng.ch. ( 2023032401 28800 7200 604800 86400)
NS ns.paphosting.nl.
NS ns.paphosting.net.
NS ns.paphosting.eu.
; ACME DNS
auth NS auth.ipng.ch.
A 194.1.163.93
AAAA 2001:678:d78:3::93
This snippet will make a DNS delegation for sub-domain auth.ipng.ch to the server also called
auth.ipng.ch and because the downstream delegation is in the same domain, I need to provide glue
records, that tell clients who are querying for auth.ipng.ch where to find that nameserver. At
this point, any request for *.auth.ipng.ch will end up being forwarded to the authoritative
nameserver, which can be found at either 194.1.163.93 or 2001:678:d78:3::93.
Step 2. Start ACME DNS
After having built the acme-dns server and given it a suitable systemd unit file, and knowing that
it’s going to be responsible for the sub-domain auth.ipng.ch, I give it the following straight
forward configuration file:
lego@lego:~$ mkdir ~/acme-dns/
lego@lego:~$ cat << EOF > acme-dns/config.cfg
[general]
listen = "[::]:53"
protocol = "both"
domain = "auth.ipng.ch"
nsname = "auth.ipng.ch"
nsadmin = "hostmaster.ipng.ch"
records = [
"auth.ipng.ch. NS auth.ipng.ch.",
"auth.ipng.ch. A 194.1.163.93",
"auth.ipng.ch. AAAA 2001:678:d78:3::93",
]
debug = false
[database]
engine = "sqlite3"
connection = "/home/lego/acme-dns/acme-dns.db"
[api]
ip = "[::]"
disable_registration = false
port = "443"
tls = "letsencrypt"
acme_cache_dir = "/home/lego/acme-dns/api-certs"
notification_email = "hostmaster+dns-auth@ipng.ch"
corsorigins = [ "*" ]
use_header = false
header_name = "X-Forwarded-For"
[logconfig]
loglevel = "debug"
logtype = "stdout"
logformat = "text"
EOF
lego@lego:~$ sudo systemctl enable acme-dns
lego@lego:~$ sudo systemctl start acme-dns
The first part of this tells the server how to construct the SOA record (domain, nsname and
nsadmin), and which records to put in the apex, nominally the NS/A/AAAA records that describe the
nameserver which is authoritative for the auth.ipng.ch domain. Then, the database part is where
user credentials will be stored, and the API portion shows how users will be able to interact with
the controlplane part of the service, notably registering new clients, and updating nameserver TXT
records for existing clients.
Interestingly, the API is served on HTTPS port 443, and for that it needs, you guessed it, a
certificate! ACME-DNS eats its own dogfood, which I can appreciate: it will use DNS-01 validation
to get a certificate for auth.ipng.ch itself, by serving the challenge for well known record
_acme-challenge.auth.ipng.ch, so it’s turtles all the way down!
Step 3. Register a new client
Seeing as many public DNS providers allow programmatic setting of the contents of the zonefiles, for them it’s a matter of directly being driven by LEGO. But for me, running NSD, I am going to be using the ACME DNS server to fulfill that purpose, so I have to configure it to do that for me.
In the explanation of DNS-01 challenges above, you’ll remember I made a mention of registering. Here’s
a closer look at what that means:
lego@lego:~$ curl -s -X POST https://auth.ipng.ch/register | json_pp
{
"allowfrom" : [],
"fulldomain" : "76f88564-740b-4483-9bc0-86d1fb531e20.auth.ipng.ch",
"password" : "<redacted>",
"subdomain" : "76f88564-740b-4483-9bc0-86d1fb531e20",
"username" : "e4608fdf-9a69-4930-8cf1-57218738792d"
}
What happened here is that, using the HTTPS endpoint, I asked the ACME-DNS server to create for me an empty
DNS record, which it did on 76f88564-740b-4483-9bc0-86d1fb531e20.auth.ipng.ch. Further, if I offer
the given username and password, I am able to update that record’s value. Let’s take a look:
lego@lego:~$ dig +short TXT 02e3acfc-bbca-46bb-9cee-8eab52c73c30.auth.ipng.ch
lego@lego:~$ curl -s -X POST -H "X-Api-User: 5f3591d1-0d13-4816-a329-7965a8639ab5" \
-H "X-Api-Key: <redacted>" \
-d '{"subdomain": "02e3acfc-bbca-46bb-9cee-8eab52c73c30", \
"txt": "___Hello_World_token_______________________"}' \
https://auth.ipng.ch/update
Numbers everywhere, but I learned a lot here! Notice how the first time I sent the dig request for
the 02e3acfc-bbca-46bb-9cee-8eab52c73c30.auth.ipng.ch it did not respond anything (an empty
record). But then, using the username/password I could update the record with a 41 character
string, and I was informed of the fulldomain key there, which is the one that I should be
configuring in the domain(s) for which I want to get a certificate.
I configure it in the ipng.ch and ipng.nl domain as follows (taking ipng.nl as an example):
$ORIGIN ipng.nl.
$TTL 86400
@ IN SOA ns.paphosting.net. hostmaster.ipng.nl. ( 2023032401 28800 7200 604800 86400)
IN NS ns.paphosting.nl.
IN NS ns.paphosting.net.
IN NS ns.paphosting.eu.
CAA 0 issue "letsencrypt.org"
CAA 0 issuewild "letsencrypt.org"
CAA 0 iodef "mailto:hostmaster@ipng.ch"
_acme-challenge CNAME 8ee2969b-571c-4b3a-b6a0-6d6221130c96.auth.ipng.ch.
The records here are a CAA which is a type of DNS record used to provide additional confirmation
for the Certificate Authority when validating an SSL certificate. This record allows me to specify
which certificate authorities are authorized to deliver SSL certificates for the domain. Then, the
well known _acme-challenge.ipng.nl record is merely telling the client by means of a CNAME to go
ask for 8ee2969b-571c-4b3a-b6a0-6d6221130c96.auth.ipng.ch instead.
Putting this part all together now, I can issue a query for that ipng.nl domain …
lego@lego:~$ dig +short TXT _acme-challenge.ipng.nl.
"___Hello_World_token_______________________"
… and would you look at that! The query for the ipng.nl domain, is a CNAME to the specific uuid record in the auth.ipng.ch domain, where ACME-DNS is serving it with the response that I can programmatically set to different values, yee-haw!
Step 4. Run LEGO
The LEGO client has all sorts of challenge providers linked in. Once again, Debian is a bit behind on things, shipping version 3.2.0-3.1+b5 in Bullseye, although upstream is much further along. So I purge the Debian package and download the v4.10.2 amd64 package directly from its [Github] releases page. The ACME-DNS handler was only added in v4 of the client. But now all that’s left for me to do is run it:
lego@lego:~$ export ACME_DNS_API_BASE=https://auth.ipng.ch/
lego@lego:~$ export ACME_DNS_STORAGE_PATH=/home/lego/acme-dns/credentials.json
lego@lego:~$ /home/lego/bin/lego --path /etc/lego/ --email noc@ipng.ch --accept-tos --dns acme-dns \
--domains ipng.ch --domains *.ipng.ch \
--domains ipng.nl --domains *.ipng.nl \
run
The LEGO client goes through the ACME flow that I described at the top of this article, and ends up spitting out a certificate \o/
lego@lego:~$ openssl x509 -noout -text -in /etc/lego/certificates/ipng.ch.crt
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
03:58:8f:c1:25:00:e2:f3:d3:3f:d6:ed:ba:bc:1d:0d:54:ea
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Let's Encrypt, CN = R3
Validity
Not Before: Mar 21 20:24:08 2023 GMT
Not After : Jun 19 20:24:07 2023 GMT
Subject: CN = ipng.ch
X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:*.ipng.ch, DNS:*.ipng.nl, DNS:ipng.ch, DNS:ipng.nl
Et voila! Wildcard certificates for multiple domains using ACME-DNS.