I have several backup servers that run the excellent rsnapshot software, which uses Secure Shell (SSH) for remote access. The SSH private key of the backup server can be a weak link in the overall security. To see how it can be a problem, consider if someone breaks into your backup server and manages to copy your SSH private key, they will now have the ability to login to all machines that you take backups off (and that should be all of your machines, right?).
The traditional way to mitigate SSH private key theft is by password protecting the private key. This works poorly in an unattended server environment because either the decryption password needs to be stored in disk (where the attacker can read it) or the decrypted private key has to be available in decrypted form in memory (where attacker can read it).
A better way to deal with the problem is to move the SSH private key to a smartcard. The idea is that the private key cannot be copied by an attacker who roots your backup server. (Careful readers may have spotted a flaw here, and I need to explain one weakness with my solution: an attacker will still be able to login to all your systems by going through your backup server, however it will require an open inbound network connection to your backup server and the attacker will never know what your private key is. What this does is to allow you to more easily do damage control by removing the smartcard from the backup server.)
In this writeup, I’ll explain how to accomplish all this on a Debian/Ubuntu-system using a OpenPGP smartcard, a Gemalto USB Shell Token v2 with gpg-agent/scdaemon from GnuPG together with OpenSSH.
First we need to install some packages. The goal is to configure OpenSSH to talk to the gpg-agent which will start and talk to scdaemon which in turn talks to pcscd which talks to the smart card reader. For some strange reason, the scdaemon binary is shipped with GnuPG’s S/MIME interface in the gpgsm package.
# apt-get install pcscd gnupg-agent gpgsm
The above command should install and start
pcscd and if all works well, you should be able to check the status of the smartcard using GnuPG.
# gpg --card-status
You need to initialize the smartcard and generate a private key on it, again using GnuPG. If you trust GnuPG more than the smartcard to generate a good private key, you may generate the private key using GnuPG and then move it onto the smartcard (hint: use the
keytocard command). Make sure you don’t leave a copy of the private key on the same machine!
# gpg --card-edit gpg: detected reader `Gemalto GemPC Key 00 00' ... gpg/card> admin Admin commands are allowed gpg/card> name Cardholder's surname: Cardholder's given name: host.example.org gpg: 3 Admin PIN attempts remaining before card is permanently locked Please enter the Admin PIN gpg: gpg-agent is not available in this session gpg/card> lang Language preferences: en gpg/card> generate Make off-card backup of encryption key? (Y/n) n Please enter the PIN What keysize do you want for the Signature key? (2048) What keysize do you want for the Encryption key? (2048) What keysize do you want for the Authentication key? (2048) Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) Key does not expire at all Is this correct? (y/N) y You need a user ID to identify your key; the software constructs the user ID from the Real Name, Comment and Email Address in this form: "Heinrich Heine (Der Dichter) <firstname.lastname@example.org>" Real name: host.example.org Email address: Comment: You selected this USER-ID: "host.example.org" Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o gpg: existing key will be replaced gpg: please wait while key is being generated ... gpg: key generation completed (33 seconds) gpg: signatures created so far: 0 gpg: existing key will be replaced gpg: please wait while key is being generated ... gpg: key generation completed (18 seconds) gpg: signatures created so far: 1 gpg: signatures created so far: 2 gpg: existing key will be replaced gpg: please wait while key is being generated ... gpg: key generation completed (23 seconds) gpg: signatures created so far: 3 gpg: signatures created so far: 4 gpg: key 12345678 marked as ultimately trusted public and secret key created and signed. gpg: checking the trustdb gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u pub 2048R/12345678 2011-09-19 Key fingerprint = 1234 5678 1234 5678 1234 5678 1234 5678 1234 5678 uid host.example.org sub 2048R/23456789 2011-09-19 sub 2048R/34567890 2011-09-19 gpg/card> quit #
Now for the interesting part. OpenSSH talks to an agent for private key handling, and GnuPG’s gpg-agent supports this protocol when the
--enable-ssh-support parameter is given. During startup, gpg-agent will print some environment variables that needs to be set when
ssh is run. Normally gpg-agent is invoked by the Xsession.d login scripts, so that the environment variables are inherited by all your processes. However, for an unattended machine without any normal login process, we need to write a script to start gpg-agent. First do these manual steps, to confirm that everything works.
# gpg-agent --daemon --enable-ssh-support > /var/run/gpg-agent-info.env # . /var/run/gpg-agent-info.env # ssh-add -L ssh-rsa AAAAB3N... cardno:000500000BD8 #
If the final step printed a SSH public id, the (sometimes) tricky part in getting the hardware to work is (hopefully) complete. What remains is to script things so that
gpg-agent is started on boot and to make sure that your backup scripts has the proper environment variables before launching whatever processes will launch
ssh. Further, since we will be running unattended, we need a mechanism to unlock the smartcard using a PIN interactively once on each boot of the machine. I prefer manually entering the PIN on every boot over having the PIN stored in a file on the disk.
I will use the
/etc/rc.local mechanism to start gpg-agent, like this:
# cat> /etc/rc.local #!/bin/sh -e exec gpg-agent --daemon --enable-ssh-support --pinentry-program /usr/local/sbin/pinentry-unattended --write-env-file /var/run/gpg-agent-info.env ^D
The astute reader will now ask what
/usr/local/sbin/pinentry-unattended is and why it is needed. Now here is the situation. scdaemon will normally query the user for a PIN using a tool called
pinentry which reads and write to the user’s TTY directly. This won’t work in unattended mode, so we want the scdaemon to signal failure here — unless we are actually unlocking the smartcard manually. Here is the entire script:
#!/bin/sh # /usr/local/sbin/pinentry-unattended -- by Simon Josefsson if test x"$PINENTRY_USER_DATA" = xinteractive; then exec pinentry "$@" fi exit 1
What remains is a script to unlock the smartcard by providing the PIN. This is typically invoked manually if the server has restarted for some reason. Don’t worry, any ssh sessions invoked by cron until you have managed to unlock the smartcard will fail with an authentication error — it won’t hang waiting for a PIN to be entered.
#!/bin/sh # /usr/local/sbin/unlock-smartcard -- by Simon Josefsson. . /var/run/gpg-agent-info.env; export GPG_AGENT_INFO SSH_AUTH_SOCK SSH_AGENT_PID gpg-connect-agent 'scd killscd' /bye > /dev/null while ! gpg-connect-agent 'scd serialno' /bye | grep -q SERIALNO; do sleep 1 done PINENTRY_USER_DATA=interactive export PINENTRY_USER_DATA checkpin
And the script
checkpin is as follows:
#!/bin/sh # /usr/local/sbin/checkpin -- by Simon Josefsson. id=`gpg-connect-agent 'scd serialno' /bye | head -1 | cut -d -f3` gpg-connect-agent "scd checkpin $id" /bye | grep -q OK
At this point, you should have everything configured and installed. Don’t forget to
chmod +x the scripts. The typical use-pattern is as follows. After the machine has been started, gpg-agent is running but the smartcard is not unlocked with the PIN. You need to manually login to the machine and run ‘unlock-smartcard’ and enter the PIN. In the script that runs the backup jobs, invoked via cron, make sure that the first line of the scripts reads (assuming Bourne shell script syntax):
. /var/run/gpg-agent-info.env; export GPG_AGENT_INFO SSH_AUTH_SOCK SSH_AGENT_PID
To avoid needlessly attempting ssh connections if the smartcard is not unlocked, your backup script can also call the
checkpin code and abort if it doesn’t return true.
checkpin || exit 1
Some final words about debugging. A basic command to run to check that the GnuPG side is working is
gpg --card-status, it should print some information about the smartcard if successful. To check that the SSH agent part is working, use
ssh-add -L. If you get error messages, try killing the
scdaemon process by running
killall -9 scdaemon and let
gpg-agent respawn a new
That’s it! If you like my writeup, please flattr it. 🙂
You should also use command-limited ssh keys, that will help with the case where someone steals a key.
Nice writeup, but it’s rather more simple to use a keypair specifically for rsnapshot and use the from= and command= options in authorized_keys to limit that key’s usage to only calling rsnapshot from your backup server. Even if your key is compromised all an attacker can do is trigger a backup from your backup server.
Isn’t it way easier to restrict a password-less SSH key to run only the wanted automated command using the command=”…” setting in ~/.ssh/authorized_keys as documented in the sshd(8) manpage?
Hmm, an inbound tcp connection is needed. The attacker can surely just use some reverse shellcode to make the traffic appear normal?
Also, once the attacker can login to your servers he can of course modify authorized_keys…
or, one could just configure their authorized_keys file properly and add an allowed host in front of the key. In this case the backup server IP address. This would make the key only valid when used from that IP.
from=”188.8.131.52″ ssh-rsa xxxxxxxxxxxx…
Or push data to backup server instead of pulling, or use something like bacula + encrypted backups
As mentioned by others, from= and command= will get you a long way — you might want to have a look at:
Also, your blog reminds me that I should probably write up the fact that you can avoid root access when triggering backups over ssh, as described here:
For things other than backup servers, it can be worth further restricting what can be done by the initiating host — a good example of that is Debian’s push mirror script:
great tutorial and very detail 🙂
thanks for sharing really appreciate this tutorial help me alot to solved my customer ssh problem 🙂