+---
+title: Let's Encrypt Tiny
+categories: Sysadmin
+---
+
+I think all HTTP communication on the internet should be encrypted -- and thanks
+to [Let's Encrypt](https://letsencrypt.org/), we are now much closer to this
+goal than we were two years ago. However, when I set up Let's Encrypt on my
+server (which is more than a year ago by now), I was not very happy with the
+official client: The client manages multiple certificates with different sets of
+domains per certificate, but I found it entirely unclear which commands would
+replace existing certificates or create a new one. Moreover, I have some
+special needs: I've set up DNSSEC with TLSA records containing hashes of my
+certificates, so replacing a certificate has to also update DNS and deal with
+the fact that DNS entries get cached. Lucky enough, Let's Encrypt is based on
+open standards, so I was not forced to use their client!
+
+To make a long story short, I decided to write my own Let's Encrypt client,
+which I describe in this post.
+
+<!-- MORE -->
+
+## Let's Encrypt Tiny
+
+The client is based on [acme-tiny](https://github.com/diafygi/acme-tiny), a
+beautifully small Python library (<200 lines) speaking the ACME protocol.
+That's the protocol developed by Let's Encrypt to communicate with an automated
+CA. I duly called my client "Let's Encrypt Tiny", and with less than 250 lines
+I think that name is still fair. For now,
+[Let's Encrypt Tiny](https://github.com/RalfJung/server-scripts/blob/master/letsencrypt-tiny)
+resides in my [server-scripts](https://github.com/RalfJung/server-scripts)
+repository, and it will stay there until anyone else has an interesting in using
+it. ;)
+
+The central concept of Let's Encrypt Tiny is a "certificate line" -- a sequence
+of certificates, possibly for different private keys, that "belong together" in
+the sense that each is considered (by their owner) the successor of the
+previous. Services like apache are configured to use a particular certificate
+line for a particular domain (well, in my case, it's the same line for all
+domains, but one could imagine different setups). Each certificate line has a
+separate config file, whose most important job is to configure the set of
+domains in the latest certificate of the line. The key operations on a
+certificate line are to create a *fresh* key and obtain a certificate for it,
+and to obtain a new certificate for the *existing* key. Let's Encrypt
+certificates expire after 90 days, so we have to perform renewal at least that
+often, but there is no reason to always generate a fresh private key. One
+reason to keep the previous key is that the TLSA record in DNS can be configured
+to be a hash of just the key, so a renewal for an existing key can be done
+without changing DNS.
+
+The other important concept in Let's Encrypt Tiny is the idea of a "staging
+key", as opposed to the "live key". This is needed to properly support
+DNSSEC+TLSA. Just briefly, the idea of TLSA is to not use certificate
+authorities to determine the correct certificate for a domain (CAs have proven
+untrustworthy again and again, and a single failing CA undermines the security
+of the entire system), but instead use DNS. If DNS is secured with DNSSEC --
+which is much more resilient against a single entity failing than the CA system
+-- then we can just put a hash of the certificate, or a hash of the public key,
+into the DNS, side-stepping the entire CA system. Unfortunately, TLSA has not
+seen widespread adoption, though a
+[Firefox extension](https://addons.mozilla.org/en-US/firefox/addon/dnssec-validator/)
+is available (and extensions for some other browsers as well). Still, I like
+this technology, so I have deployed it on my server.
+
+However, one consequence of TLSA records is that a freshly generated key cannot
+immediately be used (i.e., by the web server): The DNS still contains the old
+key's data, and that data gets cached! In Let's Encrypt Tiny, this key first
+gets "staged". Next, we have to update the DNS zone to contain TLSA records for
+*both* the old and the new key. Then we have to wait until the TTL
+(time-to-live) of that record passes, to make sure that no caches still contain
+*only* the old key. Finally, we "unstage" the key so all the servers (web
+server, jabber server, and so on) to use the new certificate and key, which are
+now "live".
+
+## Configuration
+
+Let's look at an example: Here is the configuration file for this server,
+ralfj.de, with comments explaining the purpose of the various options:
+
+```
+# The domains currently making up this certificate line:
+domains =
+ ralfj.de
+ www.ralfj.de
+ lists.ralfj.de
+ git.ralfj.de
+ ns.ralfj.de
+ ipv4.ns.ralfj.de
+ ipv6.ns.ralfj.de
+ jabber.ralfj.de
+ conference.jabber.ralfj.de
+# ... this list goes on.
+
+# The size of the RSA secret key (in bits).
+key-length = 4096
+
+[timing]
+# Max. age of the private key before we generate a new one.
+max-key-age-days = 256
+# How long a new private key is "staged" before it is used as the live key.
+# 0 disables staging.
+staging-hours = 0
+# How many days before the certificate expires should be request a new one?
+renew-cert-before-expiry-days = 15
+
+[hooks]
+# Script to execute after the certificate changed.
+post-certchange = /root/letsencrypt/cert-hook
+# Script to execute after the key changed. Only needed when using DNSSEC+TLSA.
+#post-keychange = /root/letsencrypt/key-hook
+
+[acme]
+# File storing the ACME account private key (created if missing).
+account-key = /etc/ssl/private/letsencrypt/account.key
+# Directory to put the ACME challenges into. Must be mapped to
+# /.well-known/acme-challenge on all domains listed above. For example, in
+# apache, put the following directive somewhere global:
+# Alias /.well-known/acme-challenge/ /srv/acme-challenge/
+challenge-dir = /srv/acme-challenge/
+
+[dirs]
+# Directory for certificates.
+certs = /etc/ssl/mycerts/letsencrypt
+# Directory for private keys.
+keys = /etc/ssl/private/letsencrypt
+# A place to put old certificates and keys.
+backups = /etc/ssl/old/letsencrypt
+
+[files]
+# Filename prefix (without extension) for the live key and certificate.
+live = live
+# Filename prefix (without extension) for the staging key and certificate.
+staging = staging
+```
+
+With this configuration, Let's Encrypt Tiny creates files
+`/etc/ssl/mycerts/letsencrypt/live.crt` and
+`/etc/ssl/private/letsencrypt/live.key`. These files are always the "tip" of
+the certificate line and should be configured in the various servers -- however,
+most servers will need these files to be massaged a bit. First of all, we also
+need a key chain, and the intermediate CA used by Let's Encrypt actually changes
+over time. Moreover, some servers want certificate and key in one file, while
+others want the certificates to be bundled with the keychain and expect the
+private key in a separate file. Sometimes, the Diffie-Hellman parameters are
+also expected in the same file as the certificate -- every SSL-supporting server
+seems to handle this slightly differently.
+
+This is all handled by the certificate hook, which creates the various derived files:
+```
+#!/bin/sh
+cd /etc/ssl/mycerts/letsencrypt
+export PATH="/usr/sbin/:/sbin/:$PATH"
+
+# Determine the intermediate CA used by this certificate. We expect the
+# intermediate certificates to be stored in files in /etc/ssl/chains/,
+# e.g. /etc/ssl/chains/letsencrypt-X3.crt.
+ISSUER=$(openssl x509 -issuer -in live.crt -noout | sed 's/.*Authority \(X[0-9]\+\).*/\1/')
+ISSUER_FILE=/etc/ssl/chains/letsencrypt-"$ISSUER".crt
+if ! [ -f "$ISSUER_FILE" ]; then
+ echo "Cannot find certificate for issuer $(openssl x509 -issuer -in live.crt -noout)"
+ exit 1
+fi
+
+# Create derived files. We expect /etc/ssl/dh2048.pem to contain the DH
+# parameters, generated with
+# openssl dhparam -out /etc/ssl/dh2048.pem 2048
+cat "$ISSUER_FILE" > live.chain # just the chain
+cat live.crt /etc/ssl/dh2048.pem > live.crt+dh # Certificate plus DH parameters
+cat live.crt live.chain > live.crt+chain # Certificate plus chain
+
+# Fill in here: the code to restart/reload all relevant services.
+```
+
+With this, the apache SSL configuration looks as follows:
+
+```
+# Certificate, Key, and DH parameters
+SSLCertificateFile /etc/ssl/mycerts/live.crt+dh
+SSLCertificateKeyFile /etc/ssl/private/live.key
+SSLCertificateChainFile /etc/ssl/mycerts/live.chain
+
+# configure SSL ciphers and protocols
+SSLProtocol All -SSLv2 -SSLv3
+# TODO: Once OpenSSL supports GMC with more than just AES, revisit this
+# NOTE: The reason we support non-FS ciphers is stupid middleboxes that don't support FS
+SSLCipherSuite 'kEECDH+AESGCM:kEDH+AESGCM:kEECDH:kEDH:AESGCM:ALL:!3DES:!EXPORT:!LOW:!MEDIUM:!aNULL:!eNULL'
+SSLHonorCipherOrder on
+```
+
+(My cipher suite is deliberately not the one from
+[bettercrypto.org](https://bettercrypto.org) because I prefer to not update it
+with every change in OpenSSL's supported ciphers.)
+
+## Obtaining the first certificate
+
+You can now run `letsencrypt-tiny -c letsencrypt.conf init` to perform the
+initial setup.
+
+In the future, to change the set of domains, first edit the config file and then
+run `letsencrypt-tiny -c letsencrypt.conf -k renew`. The `-k` tells Let's
+Encrypt Tiny to also run the certificate hook.
+
+## Automation via cron
+
+Let's Encrypt certificates expire after 90 days, so we want renewal to be
+automated. To this end, just make sure that `letsencrypt-tiny -c
+letsencrypt.conf -k cron` gets run regularly, like once a day. I have the
+following in root's crontab (`sudo crontab -e`):
+
+```
+32 6 * * * /root/server-scripts/letsencrypt-tiny -c /root/letsencrypt/conf -k cron
+```
+
+This will check the time intervals you configured above, and act accordingly.
+If any action is taken, the script will print that information on standard
+output; if you have email set up on your server, this means you will get an
+email notification.
+
+## DNSSEC and TLSA
+
+Everything described so far should give you a working SSL setup if you do not
+use DNSSEC+TLSA. If you *do* use DNSSEC+TLSA, like I do on this server, you
+need to enable the `post-keychange` hook and have it regenerate your DNS zone,
+and you need to increase `staging-hours`. The zone should always contain the
+hash of the live key, and, if a staging key exists, also the hash of the staging
+key.
+
+I am managing my DNS zones with
+[zonemaker](https://www.ralfj.de/projects/zonemaker/), and wrote some Python
+code to automatically generate TLSA records using the `tlsa` tool:
+
+```
+def TLSA_from_crt(protocol, port, crtfile):
+ crtfile = "/etc/ssl/mycerts/"+crtfile
+ open(crtfile).close() # check if the file exists
+ # make sure we match on *the key only*, so that we can renew the certificate without harm
+ zone_line = subprocess.check_output(["tlsa", "--selector", str(TLSA.Selector.SubjectPublicKeyInfo), "--certificate", crtfile, "example.org"]).decode("utf-8")
+ m = re.match("^[0-9a-zA-Z_.-]+ IN TLSA ([0-9]+) ([0-9]+) ([0-9]+) ([0-9a-zA-Z]+)$", zone_line)
+ assert m is not None
+ assert int(m.group(1)) == TLSA.Usage.EndEntity
+ assert int(m.group(2)) == TLSA.Selector.SubjectPublicKeyInfo
+ return TLSA(protocol, port, TLSA.Usage.EndEntity, TLSA.Selector.SubjectPublicKeyInfo, int(m.group(3)), m.group(4))
+
+def TLSA_for_LE(protocol = Protocol.TCP, port = 443):
+ # add both the live and (potentially) staging certificate to the letsencrypt TLSA record set
+ r = [TLSA_from_crt(protocol, port, "letsencrypt/live.crt")]
+ try:
+ r.append(TLSA_from_crt(protocol, port, "letsencrypt/staging.crt"))
+ except IOError:
+ pass
+ return r
+```
+
+Now I add `TLSA_for_LE(port = 443)` to the records of my domains. Finally, the
+key hook just runs zonemaker and has bind reload the zone (it will automatically
+also be resigned). Now, whenever a staging key is created, it is automatically
+added to my zone. At least 25h later (I have the TTL set to 24h), the key gets
+unstaged, and the old TLSA record is removed from the zone.
+
+That's it! If you have any questions, feel free to report
+[issues at GitHub](https://github.com/RalfJung/server-scripts/issues).
projects / web.git / commitdiff