Unattended SSH with Smartcard

I have several backup servers that run the excellent rsnapshot software, which uses Secure Shell (SSH) for remote access. The SSH private key of the backup server can be a weak link in the overall security. To see how it can be a problem, consider if someone breaks into your backup server and manages to copy your SSH private key, they will now have the ability to login to all machines that you take backups off (and that should be all of your machines, right?).

The traditional way to mitigate SSH private key theft is by password protecting the private key. This works poorly in an unattended server environment because either the decryption password needs to be stored in disk (where the attacker can read it) or the decrypted private key has to be available in decrypted form in memory (where attacker can read it).

A better way to deal with the problem is to move the SSH private key to a smartcard. The idea is that the private key cannot be copied by an attacker who roots your backup server. (Careful readers may have spotted a flaw here, and I need to explain one weakness with my solution: an attacker will still be able to login to all your systems by going through your backup server, however it will require an open inbound network connection to your backup server and the attacker will never know what your private key is. What this does is to allow you to more easily do damage control by removing the smartcard from the backup server.)

In this writeup, I’ll explain how to accomplish all this on a Debian/Ubuntu-system using a OpenPGP smartcard, a Gemalto USB Shell Token v2 with gpg-agent/scdaemon from GnuPG together with OpenSSH.


First we need to install some packages. The goal is to configure OpenSSH to talk to the gpg-agent which will start and talk to scdaemon which in turn talks to pcscd which talks to the smart card reader. For some strange reason, the scdaemon binary is shipped with GnuPG’s S/MIME interface in the gpgsm package.

# apt-get install pcscd gnupg-agent gpgsm

The above command should install and start pcscd and if all works well, you should be able to check the status of the smartcard using GnuPG.

# gpg --card-status

You need to initialize the smartcard and generate a private key on it, again using GnuPG. If you trust GnuPG more than the smartcard to generate a good private key, you may generate the private key using GnuPG and then move it onto the smartcard (hint: use the keytocard command). Make sure you don’t leave a copy of the private key on the same machine!

# gpg --card-edit
gpg: detected reader `Gemalto GemPC Key 00 00'
...
gpg/card> admin
Admin commands are allowed

gpg/card> name
Cardholder's surname: 
Cardholder's given name: host.example.org
gpg: 3 Admin PIN attempts remaining before card is permanently locked

Please enter the Admin PIN
gpg: gpg-agent is not available in this session
                 
gpg/card> lang
Language preferences: en

gpg/card> generate
Make off-card backup of encryption key? (Y/n) n

Please enter the PIN
What keysize do you want for the Signature key? (2048) 
What keysize do you want for the Encryption key? (2048) 
What keysize do you want for the Authentication key? (2048) 
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: host.example.org
Email address: 
Comment: 
You selected this USER-ID:
    "host.example.org"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
gpg: existing key will be replaced
gpg: please wait while key is being generated ...
gpg: key generation completed (33 seconds)
gpg: signatures created so far: 0
gpg: existing key will be replaced
gpg: please wait while key is being generated ...
gpg: key generation completed (18 seconds)
gpg: signatures created so far: 1
gpg: signatures created so far: 2
gpg: existing key will be replaced
gpg: please wait while key is being generated ...
gpg: key generation completed (23 seconds)
gpg: signatures created so far: 3
gpg: signatures created so far: 4
gpg: key 12345678 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   2048R/12345678 2011-09-19
      Key fingerprint = 1234 5678 1234 5678 1234  5678 1234 5678 1234 5678
uid                  host.example.org
sub   2048R/23456789 2011-09-19
sub   2048R/34567890 2011-09-19


gpg/card> quit
# 

Now for the interesting part. OpenSSH talks to an agent for private key handling, and GnuPG’s gpg-agent supports this protocol when the --enable-ssh-support parameter is given. During startup, gpg-agent will print some environment variables that needs to be set when ssh is run. Normally gpg-agent is invoked by the Xsession.d login scripts, so that the environment variables are inherited by all your processes. However, for an unattended machine without any normal login process, we need to write a script to start gpg-agent. First do these manual steps, to confirm that everything works.

# gpg-agent --daemon --enable-ssh-support > /var/run/gpg-agent-info.env
# . /var/run/gpg-agent-info.env
# ssh-add -L
ssh-rsa AAAAB3N... cardno:000500000BD8
#

If the final step printed a SSH public id, the (sometimes) tricky part in getting the hardware to work is (hopefully) complete. What remains is to script things so that gpg-agent is started on boot and to make sure that your backup scripts has the proper environment variables before launching whatever processes will launch ssh. Further, since we will be running unattended, we need a mechanism to unlock the smartcard using a PIN interactively once on each boot of the machine. I prefer manually entering the PIN on every boot over having the PIN stored in a file on the disk.

I will use the /etc/rc.local mechanism to start gpg-agent, like this:

# cat> /etc/rc.local 
#!/bin/sh -e
exec gpg-agent --daemon --enable-ssh-support 
    --pinentry-program /usr/local/sbin/pinentry-unattended 
    --write-env-file /var/run/gpg-agent-info.env
^D

The astute reader will now ask what /usr/local/sbin/pinentry-unattended is and why it is needed. Now here is the situation. scdaemon will normally query the user for a PIN using a tool called pinentry which reads and write to the user’s TTY directly. This won’t work in unattended mode, so we want the scdaemon to signal failure here — unless we are actually unlocking the smartcard manually. Here is the entire script:

#!/bin/sh
# /usr/local/sbin/pinentry-unattended -- by Simon Josefsson
if test x"$PINENTRY_USER_DATA" = xinteractive; then
    exec pinentry "$@"
fi
exit 1

What remains is a script to unlock the smartcard by providing the PIN. This is typically invoked manually if the server has restarted for some reason. Don’t worry, any ssh sessions invoked by cron until you have managed to unlock the smartcard will fail with an authentication error — it won’t hang waiting for a PIN to be entered.

#!/bin/sh
# /usr/local/sbin/unlock-smartcard -- by Simon Josefsson.
. /var/run/gpg-agent-info.env; export GPG_AGENT_INFO SSH_AUTH_SOCK SSH_AGENT_PID
gpg-connect-agent 'scd killscd' /bye > /dev/null
while ! gpg-connect-agent 'scd serialno' /bye | grep -q SERIALNO; do
    sleep 1
done
PINENTRY_USER_DATA=interactive
export PINENTRY_USER_DATA
checkpin

And the script checkpin is as follows:

#!/bin/sh
# /usr/local/sbin/checkpin -- by Simon Josefsson.
id=`gpg-connect-agent 'scd serialno' /bye | head -1 | cut -d  -f3`
gpg-connect-agent "scd checkpin $id" /bye | grep -q OK

At this point, you should have everything configured and installed. Don’t forget to chmod +x the scripts. The typical use-pattern is as follows. After the machine has been started, gpg-agent is running but the smartcard is not unlocked with the PIN. You need to manually login to the machine and run ‘unlock-smartcard’ and enter the PIN. In the script that runs the backup jobs, invoked via cron, make sure that the first line of the scripts reads (assuming Bourne shell script syntax):

. /var/run/gpg-agent-info.env; export GPG_AGENT_INFO SSH_AUTH_SOCK SSH_AGENT_PID

To avoid needlessly attempting ssh connections if the smartcard is not unlocked, your backup script can also call the checkpin code and abort if it doesn’t return true.

checkpin || exit 1

Some final words about debugging. A basic command to run to check that the GnuPG side is working is gpg --card-status, it should print some information about the smartcard if successful. To check that the SSH agent part is working, use ssh-add -L. If you get error messages, try killing the scdaemon process by running killall -9 scdaemon and let gpg-agent respawn a new scdaemon process.

That’s it! If you like my writeup, please flattr it. 🙂