A common nuisance when first connecting with SSH to a server is to verify the fingerprint. Especially if you have many servers with multiple users, when everyone needs to know all the fingerprints of all the servers. This can easily be improved with SSH CA host certificates.

For example if we were to sign a server’s public key when we provision it, everyone that already trusts the CA can then also connect to that server without having to manually verify the fingerprint.

User certificates on the other hand can help us authorize users to a server without manually managing each individual key. By deploying a SSH user CA to a remote host anyone with a valid certificate can connect to it.

Manpages

The manpages of ssh-keygen(1) and sshd(8) contains everything you need to known about SSH Certificates, of most interest is the section “CERTIFICATES” of ssh-keygen(1) and sections “AUTHORIZED_KEYS FILE FORMAT” and “SSH_KNOWN_HOSTS FILE FORMAT” of sshd(8). Use them as a reference and think of this post as a way to get started.

Host certificates

Creating

The CA itself is generated like any other SSH key.

ssh-keygen -f host_ca -t ed25519

Trusting

Users can then choose to trust this CA by adding the a line like toe following to their known_hosts file (defined in sshd(8) section “AUTHORIZED_KEYS FILE FORMAT”):

@cert-authority *.example.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIzHKhkOE4C58Zgg/7AO2xXVGKBSAt2iKs9vJgkCu8hh user@host

The second field is to limit the scope of the key, in this case we trust it to verify everything from *.example.com but not for example secret.example.test.

Signing

This is how we sign a server’s public key, that is, generate a certificate for that server:

ssh-keygen -s "$HOST_CA" -I "$IDENTIFIER" -h -n "$SERVER_DOMAIN" "$PUBLIC_KEY"

A quick explanation to the arguments:

Its also possible to limit when a certificate is valid with -V.

In order to sign the public key ssh_host_ed25519_key.pub of server server.example.com with the host_ca CA we can run:

ssh-keygen -s "host_ca" -I "user_1@host_ca" -h -n "server.example.com" "ssh_host_ed25519_key.pub"

This will create a file called ssh_host_ed25519_key-cert.pub that we need to configure the server to use. This is done with the HostCertificate directive from sshd_config(5).

HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

Demoing

Lets put everything together into a working example.

ssh-keygen -f host_ca -t ed25519
echo "@cert-authority *.example.com $(cat host_ca.pub)" >> ~/.ssh/known_hosts
scp server.example.com:/etc/ssh/ssh_host_ed25519_key.pub .
ssh-keygen -s host_ca -I "user_1@host_ca" -h -n server.example.com \
    ssh_host_ed25519_key.pub
scp ssh_host_ed25519_key-cert.pub server.example.com:
ssh -t server.example.com 'echo "HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub" | sudo tee --append /etc/ssh/sshd_config'
ssh -t server.example.com sudo systemctl reload ssh

Lets test it out

echo "@cert-authority *.example.com $(cat host_ca.pub)" \
    | sudo -u newuser -i bash -c "cat > ~/.ssh/known_hosts"
sudo -u newuser ssh server.example.com

Notice how it does not ask you to verify the fingerprint, as we already trusted the CA.

User certificates

By installing a CA’s public key into a remote users authorized_keys file (or globally on the server) everyone who has their public key signed by the CA (i.e. a certificate) will have access to the server.

Creating

We create user CA’s in the exact same manner as host CA’s, or ordinary ssh keys for that matter.

ssh-keygen -f "user_ca" -t "ed25519"

Trusting

By adding the user CA’s public key to a remote user’s authorized_keys file we will grant access to anyone who has a valid certificate from the CA. For this to work we also need to give the key the cert-authority option. The format is specified by section “AUTHORIZED_KEYS FILE FORMAT” of sshd(8).

For example:

cert-authority ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK624pJT4N/5RrU9AE4I5U1fZCVGwlyqM4nylreB15oB user@host

It is also possible to globally install a CA key by using the TrustedUserCAKeys of sshd_config(5). In that case it is however very important to limit the scope of the certificates as a certificate will otherwise give access to any account on the server.

Signing

A user certificate is generated in much the same way as a host certificate, the big difference is that the -h argument is missing.

ssh-keygen -s "$USER_CA" -I "$IDENTIFIER" "$PUBLIC_KEY"

The other difference is the principals defined with -n. With user certificates they define which users the certificate is valid for, instead of which hosts. If you have installed the CA globally (with TrustedUserCAKeys in sshd_config) it is strongly recommended you use this, as the certificate otherwise will grant access to any account.

ssh-keygen -s "$USER_CA" -I "$IDENTIFIER" -n "$USER" "$PUBLIC_KEY"

Demoing

Lets try out a working example of user certificates:

ssh-keygen -f user_ca -t ed25519
echo "cert-authority $(cat user_ca.pub)" \
    | ssh server.example.com "cat >> .ssh/authorized_keys"

And test if it works:

# We use sudo to be able to write to newuser's homedir in order to simplify
# this demo
sudo ssh-keygen -s user_ca -I "newuser@user_ca" ~newuser/.ssh/id_ed25519.pub
sudo -u newuser -i ssh user@server.example.com

Notice that no password is required, in fact, if you followed this post in order you should have been logged in without neither verifying a fingerprint nor providing a password.

Revocation

Revoking keys is also possible, however because of the length of this post and the fact that I have yet to configure any key revocation lists I have chosen not to discuss that in this post. You can however read about it in the section “KEY REVOCATION LISTS” of ssh-keygen(1).