[web.git] / ralf / _posts / 2017-12-26-lets-encrypt.md

---
title: Let's Encrypt Tiny
categories: Sysadmin
---

I think all HTTP communication on the internet should be encrypted -- and thanks
to [Let's Encrypt](https://letsencrypt.org/), we are now much closer to this
goal than we were two years ago.  However, when I set up Let's Encrypt on my
server (which is more than a year ago by now), I was not very happy with the
official client: The client manages multiple certificates with different sets of
domains per certificate, but I found it entirely unclear which commands would
replace existing certificates or create a new one.  Moreover, I have some
special needs: I've set up DNSSEC with TLSA records containing hashes of my
certificates, so replacing a certificate has to also update DNS and deal with
the fact that DNS entries get cached.  Lucky enough, Let's Encrypt is based on
open standards, so I was not forced to use their client!

To make a long story short, I decided to write my own Let's Encrypt client,
which I describe in this post.

<!-- MORE -->

## Let's Encrypt Tiny

The client is based on [acme-tiny](https://github.com/diafygi/acme-tiny), a
beautifully small Python library (<200 lines) speaking the ACME protocol.
That's the protocol developed by Let's Encrypt to communicate with an automated
CA.  I duly called my client "Let's Encrypt Tiny", and with less than 250 lines
I think that name is still fair.  For now,
[Let's Encrypt Tiny](https://github.com/RalfJung/server-scripts/blob/master/letsencrypt-tiny)
resides in my [server-scripts](https://github.com/RalfJung/server-scripts)
repository, and it will stay there until anyone else has an interesting in using
it. ;)

The central concept of Let's Encrypt Tiny is a "certificate line" -- a sequence
of certificates, possibly for different private keys, that "belong together" in
the sense that each is considered (by their owner) the successor of the
previous.  Services like apache are configured to use a particular certificate
line for a particular domain (well, in my case, it's the same line for all
domains, but one could imagine different setups).  Each certificate line has a
separate config file, whose most important job is to configure the set of
domains in the latest certificate of the line.  The key operations on a
certificate line are to create a *fresh* key and obtain a certificate for it,
and to obtain a new certificate for the *existing* key.  Let's Encrypt
certificates expire after 90 days, so we have to perform renewal at least that
often, but there is no reason to always generate a fresh private key.  One
reason to keep the previous key is that the TLSA record in DNS can be configured
to be a hash of just the key, so a renewal for an existing key can be done
without changing DNS.

The other important concept in Let's Encrypt Tiny is the idea of a "staging
key", as opposed to the "live key".  This is needed to properly support
DNSSEC+TLSA.  Just briefly, the idea of TLSA is to not use certificate
authorities to determine the correct certificate for a domain (CAs have proven
untrustworthy again and again, and a single failing CA undermines the security
of the entire system), but instead use DNS.  If DNS is secured with DNSSEC --
which is much more resilient against a single entity failing than the CA system
-- then we can just put a hash of the certificate, or a hash of the public key,
into the DNS, side-stepping the entire CA system.  Unfortunately, TLSA has not
seen widespread adoption, though a
[Firefox extension](https://addons.mozilla.org/en-US/firefox/addon/dnssec-validator/)
is available (and extensions for some other browsers as well).  Still, I like
this technology, so I have deployed it on my server.

However, one consequence of TLSA records is that a freshly generated key cannot
immediately be used (i.e., by the web server): The DNS still contains the old
key's data, and that data gets cached!  In Let's Encrypt Tiny, this key first
gets "staged".  Next, we have to update the DNS zone to contain TLSA records for
*both* the old and the new key.  Then we have to wait until the TTL
(time-to-live) of that record passes, to make sure that no caches still contain
*only* the old key.  Finally, we "unstage" the key so all the servers (web
server, jabber server, and so on) to use the new certificate and key, which are
now "live".

## Configuration

Let's look at an example: Here is the configuration file for this server,
ralfj.de, with comments explaining the purpose of the various options:

```
# The domains currently making up this certificate line:
domains =
  ralfj.de
  www.ralfj.de
  lists.ralfj.de
  git.ralfj.de
  ns.ralfj.de
  ipv4.ns.ralfj.de
  ipv6.ns.ralfj.de
  jabber.ralfj.de
  conference.jabber.ralfj.de
# ... this list goes on.

# The size of the RSA secret key (in bits).
key-length = 4096

[timing]
# Max. age of the private key before we generate a new one.
max-key-age-days = 256
# How long a new private key is "staged" before it is used as the live key.
# 0 disables staging.
staging-hours = 0
# How many days before the certificate expires should be request a new one?
renew-cert-before-expiry-days = 15

[hooks]
# Script to execute after the certificate changed.
post-certchange = /root/letsencrypt/cert-hook
# Script to execute after the key changed.  Only needed when using DNSSEC+TLSA.
#post-keychange = /root/letsencrypt/key-hook

[acme]
# File storing the ACME account private key (created if missing).
account-key = /etc/ssl/private/letsencrypt/account.key
# Directory to put the ACME challenges into.  Must be mapped to
# /.well-known/acme-challenge on all domains listed above.  For example, in
# apache, put the following directive somewhere global:
#   Alias /.well-known/acme-challenge/ /srv/acme-challenge/
challenge-dir = /srv/acme-challenge/

[dirs]
# Directory for certificates.
certs = /etc/ssl/mycerts/letsencrypt
# Directory for private keys.
keys = /etc/ssl/private/letsencrypt
# A place to put old certificates and keys.
backups = /etc/ssl/old/letsencrypt

[files]
# Filename prefix (without extension) for the live key and certificate.
live = live
# Filename prefix (without extension) for the staging key and certificate.
staging = staging
```

With this configuration, Let's Encrypt Tiny creates files
`/etc/ssl/mycerts/letsencrypt/live.crt` and
`/etc/ssl/private/letsencrypt/live.key`.  These files are always the "tip" of
the certificate line and should be configured in the various servers -- however,
most servers will need these files to be massaged a bit.  First of all, we also
need a key chain, and the intermediate CA used by Let's Encrypt actually changes
over time.  Moreover, some servers want certificate and key in one file, while
others want the certificates to be bundled with the keychain and expect the
private key in a separate file.  Sometimes, the Diffie-Hellman parameters are
also expected in the same file as the certificate -- every SSL-supporting server
seems to handle this slightly differently.

This is all handled by the certificate hook, which creates the various derived files:
```
#!/bin/sh
cd /etc/ssl/mycerts/letsencrypt
export PATH="/usr/sbin/:/sbin/:$PATH"

# Determine the intermediate CA used by this certificate.  We expect the
# intermediate certificates to be stored in files in /etc/ssl/chains/,
# e.g. /etc/ssl/chains/letsencrypt-X3.crt.
ISSUER=$(openssl x509 -issuer -in live.crt -noout | sed 's/.*Authority \(X[0-9]\+\).*/\1/')
ISSUER_FILE=/etc/ssl/chains/letsencrypt-"$ISSUER".crt
if ! [ -f "$ISSUER_FILE" ]; then
    echo "Cannot find certificate for issuer $(openssl x509 -issuer -in live.crt -noout)"
    exit 1
fi

# Create derived files.  We expect /etc/ssl/dh2048.pem to contain the DH
# parameters, generated with
#   openssl dhparam -out /etc/ssl/dh2048.pem 2048
cat "$ISSUER_FILE" > live.chain # just the chain
cat live.crt /etc/ssl/dh2048.pem > live.crt+dh # Certificate plus DH parameters
cat live.crt live.chain > live.crt+chain # Certificate plus chain

# Fill in here:  the code to restart/reload all relevant services.
```

With this, the apache SSL configuration looks as follows:

```
# Certificate, Key, and DH parameters
SSLCertificateFile      /etc/ssl/mycerts/live.crt+dh
SSLCertificateKeyFile   /etc/ssl/private/live.key
SSLCertificateChainFile /etc/ssl/mycerts/live.chain

# configure SSL ciphers and protocols
SSLProtocol All -SSLv2 -SSLv3
# TODO: Once OpenSSL supports GMC with more than just AES, revisit this
# NOTE: The reason we support non-FS ciphers is stupid middleboxes that don't support FS
SSLCipherSuite 'kEECDH+AESGCM:kEDH+AESGCM:kEECDH:kEDH:AESGCM:ALL:!3DES:!EXPORT:!LOW:!MEDIUM:!aNULL:!eNULL'
SSLHonorCipherOrder     on      
```

(My cipher suite is deliberately not the one from
[bettercrypto.org](https://bettercrypto.org) because I prefer to not update it
with every change in OpenSSL's supported ciphers.)

## Obtaining the first certificate

You can now run `letsencrypt-tiny -c letsencrypt.conf init` to perform the
initial setup.

In the future, to change the set of domains, first edit the config file and then
run `letsencrypt-tiny -c letsencrypt.conf -k renew`.  The `-k` tells Let's
Encrypt Tiny to also run the certificate hook.

## Automation via cron

Let's Encrypt certificates expire after 90 days, so we want renewal to be
automated.  To this end, just make sure that `letsencrypt-tiny -c
letsencrypt.conf -k cron` gets run regularly, like once a day.  I have the
following in root's crontab (`sudo crontab -e`):

```
32  6  *   *   *     /root/server-scripts/letsencrypt-tiny -c /root/letsencrypt/conf -k cron
```

This will check the time intervals you configured above, and act accordingly.
If any action is taken, the script will print that information on standard
output; if you have email set up on your server, this means you will get an
email notification.

## DNSSEC and TLSA

Everything described so far should give you a working SSL setup if you do not
use DNSSEC+TLSA.  If you *do* use DNSSEC+TLSA, like I do on this server, you
need to enable the `post-keychange` hook and have it regenerate your DNS zone,
and you need to increase `staging-hours`.  The zone should always contain the
hash of the live key, and, if a staging key exists, also the hash of the staging
key.

I am managing my DNS zones with
[zonemaker](https://www.ralfj.de/projects/zonemaker/), and wrote some Python
code to automatically generate TLSA records using the `tlsa` tool:

```
def TLSA_from_crt(protocol, port, crtfile):
    crtfile = "/etc/ssl/mycerts/"+crtfile
    open(crtfile).close() # check if the file exists
    # make sure we match on *the key only*, so that we can renew the certificate without harm
    zone_line = subprocess.check_output(["tlsa", "--selector", str(TLSA.Selector.SubjectPublicKeyInfo), "--certificate", crtfile, "example.org"]).decode("utf-8")
    m = re.match("^[0-9a-zA-Z_.-]+ IN TLSA ([0-9]+) ([0-9]+) ([0-9]+) ([0-9a-zA-Z]+)$", zone_line)
    assert m is not None
    assert int(m.group(1)) == TLSA.Usage.EndEntity
    assert int(m.group(2)) == TLSA.Selector.SubjectPublicKeyInfo
    return TLSA(protocol, port, TLSA.Usage.EndEntity, TLSA.Selector.SubjectPublicKeyInfo, int(m.group(3)), m.group(4))

def TLSA_for_LE(protocol = Protocol.TCP, port = 443):
    # add both the live and (potentially) staging certificate to the letsencrypt TLSA record set
    r = [TLSA_from_crt(protocol, port, "letsencrypt/live.crt")]
    try:
        r.append(TLSA_from_crt(protocol, port, "letsencrypt/staging.crt"))
    except IOError:
        pass
    return r
```

Now I add `TLSA_for_LE(port = 443)` to the records of my domains.  Finally, the
key hook just runs zonemaker and has bind reload the zone (it will automatically
also be resigned).  Now, whenever a staging key is created, it is automatically
added to my zone.  At least 25h later (I have the TTL set to 24h), the key gets
unstaged, and the old TLSA record is removed from the zone.

That's it!  If you have any questions, feel free to report
[issues at GitHub](https://github.com/RalfJung/server-scripts/issues).