My PSKC Library allows you to parse, validate and generate PSKC data. The PSKC Library is written in C, uses LibXML, and is licensed under LGPLv2+. In practice, PSKC is most commonly used to transport secret keys for OATH HOTP/TOTP devices (and other OTP devices) between the personalization machine and the OTP validation server. Yesterday I released version 2.0.0 of OATH Toolkit with the new PSKC Library. See my earlier introduction to OATH Toolkit for background. OATH Toolkit is packaged for Debian/Ubuntu and I hope to refresh the package to include libpskc/pskctool soon.

To get a feeling for the PSKC data format, consider the most minimal valid PSKC data:

<?xml version="1.0"?>
<KeyContainer xmlns="urn:ietf:params:xml:ns:keyprov:pskc" Version="1.0">
  <KeyPackage/>
</KeyContainer>

The library can easily be used to export PSKC data into a comma-separated value (CSV) format, in fact the PSKC library tutorial concludes with that as an example. There is complete API documentation for the library. The command line tool is more useful for end-users and allows you to parse and inspect PSKC data. Below is an illustration of how you would use it to parse some PSKC data, first we show the content of a file “pskc-figure2.xml”:

<?xml version="1.0" encoding="UTF-8"?>
<KeyContainer Version="1.0"
	      Id="exampleID1"
	      xmlns="urn:ietf:params:xml:ns:keyprov:pskc">
  <KeyPackage>
    <Key Id="12345678"
         Algorithm="urn:ietf:params:xml:ns:keyprov:pskc:hotp">
      <Issuer>Issuer-A</Issuer>
      <Data>
        <Secret>
          <PlainValue>MTIzNA==
          </PlainValue>
        </Secret>
      </Data>
    </Key>
  </KeyPackage>
</KeyContainer>

Here is how you would parse and pretty print that PSKC data:

jas@latte:~$ pskctool -c pskc-figure2.xml 
Portable Symmetric Key Container (PSKC):
	Version: 1.0
	Id: exampleID1
	KeyPackage 0:
		DeviceInfo:
		Key:
			Id: 12345678
			Issuer: Issuer-A
			Algorithm: urn:ietf:params:xml:ns:keyprov:pskc:hotp
			Key Secret (base64): MTIzNA==

jas@latte:~$

For more information, see the OATH Toolkit website and the PSKC Library Manual.

The traditional way to mitigate SSH private key theft is by password protecting the private key. This works poorly in an unattended server environment because either the decryption password needs to be stored in disk (where the attacker can read it) or the decrypted private key has to be available in decrypted form in memory (where attacker can read it).

A better way to deal with the problem is to move the SSH private key to a smartcard. The idea is that the private key cannot be copied by an attacker who roots your backup server. (Careful readers may have spotted a flaw here, and I need to explain one weakness with my solution: an attacker will still be able to login to all your systems by going through your backup server, however it will require an open inbound network connection to your backup server and the attacker will never know what your private key is. What this does is to allow you to more easily do damage control by removing the smartcard from the backup server.)

In this writeup, I’ll explain how to accomplish all this on a Debian/Ubuntu-system using a OpenPGP smartcard, a Gemalto USB Shell Token v2 with gpg-agent/scdaemon from GnuPG together with OpenSSH.


First we need to install some packages. The goal is to configure OpenSSH to talk to the gpg-agent which will start and talk to scdaemon which in turn talks to pcscd which talks to the smart card reader. For some strange reason, the scdaemon binary is shipped with GnuPG’s S/MIME interface in the gpgsm package.

# apt-get install pcscd gnupg-agent gpgsm

The above command should install and start pcscd and if all works well, you should be able to check the status of the smartcard using GnuPG.

# gpg --card-status

You need to initialize the smartcard and generate a private key on it, again using GnuPG. If you trust GnuPG more than the smartcard to generate a good private key, you may generate the private key using GnuPG and then move it onto the smartcard (hint: use the keytocard command). Make sure you don’t leave a copy of the private key on the same machine!

# gpg --card-edit
gpg: detected reader `Gemalto GemPC Key 00 00'
...
gpg/card> admin
Admin commands are allowed

gpg/card> name
Cardholder's surname: 
Cardholder's given name: host.example.org
gpg: 3 Admin PIN attempts remaining before card is permanently locked

Please enter the Admin PIN
gpg: gpg-agent is not available in this session
                 
gpg/card> lang
Language preferences: en

gpg/card> generate
Make off-card backup of encryption key? (Y/n) n

Please enter the PIN
What keysize do you want for the Signature key? (2048) 
What keysize do you want for the Encryption key? (2048) 
What keysize do you want for the Authentication key? (2048) 
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: host.example.org
Email address: 
Comment: 
You selected this USER-ID:
    "host.example.org"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
gpg: existing key will be replaced
gpg: please wait while key is being generated ...
gpg: key generation completed (33 seconds)
gpg: signatures created so far: 0
gpg: existing key will be replaced
gpg: please wait while key is being generated ...
gpg: key generation completed (18 seconds)
gpg: signatures created so far: 1
gpg: signatures created so far: 2
gpg: existing key will be replaced
gpg: please wait while key is being generated ...
gpg: key generation completed (23 seconds)
gpg: signatures created so far: 3
gpg: signatures created so far: 4
gpg: key 12345678 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   2048R/12345678 2011-09-19
      Key fingerprint = 1234 5678 1234 5678 1234  5678 1234 5678 1234 5678
uid                  host.example.org
sub   2048R/23456789 2011-09-19
sub   2048R/34567890 2011-09-19


gpg/card> quit
# 

Now for the interesting part. OpenSSH talks to an agent for private key handling, and GnuPG’s gpg-agent supports this protocol when the --enable-ssh-support parameter is given. During startup, gpg-agent will print some environment variables that needs to be set when ssh is run. Normally gpg-agent is invoked by the Xsession.d login scripts, so that the environment variables are inherited by all your processes. However, for an unattended machine without any normal login process, we need to write a script to start gpg-agent. First do these manual steps, to confirm that everything works.

# gpg-agent --daemon --enable-ssh-support > /var/run/gpg-agent-info.env
# . /var/run/gpg-agent-info.env
# ssh-add -L
ssh-rsa AAAAB3N... cardno:000500000BD8
#

If the final step printed a SSH public id, the (sometimes) tricky part in getting the hardware to work is (hopefully) complete. What remains is to script things so that gpg-agent is started on boot and to make sure that your backup scripts has the proper environment variables before launching whatever processes will launch ssh. Further, since we will be running unattended, we need a mechanism to unlock the smartcard using a PIN interactively once on each boot of the machine. I prefer manually entering the PIN on every boot over having the PIN stored in a file on the disk.

I will use the /etc/rc.local mechanism to start gpg-agent, like this:

# cat> /etc/rc.local 
#!/bin/sh -e
exec gpg-agent --daemon --enable-ssh-support 
    --pinentry-program /usr/local/sbin/pinentry-unattended 
    --write-env-file /var/run/gpg-agent-info.env
^D

The astute reader will now ask what /usr/local/sbin/pinentry-unattended is and why it is needed. Now here is the situation. scdaemon will normally query the user for a PIN using a tool called pinentry which reads and write to the user’s TTY directly. This won’t work in unattended mode, so we want the scdaemon to signal failure here — unless we are actually unlocking the smartcard manually. Here is the entire script:

#!/bin/sh
# /usr/local/sbin/pinentry-unattended -- by Simon Josefsson
if test x"$PINENTRY_USER_DATA" = xinteractive; then
    exec pinentry "$@"
fi
exit 1

What remains is a script to unlock the smartcard by providing the PIN. This is typically invoked manually if the server has restarted for some reason. Don’t worry, any ssh sessions invoked by cron until you have managed to unlock the smartcard will fail with an authentication error — it won’t hang waiting for a PIN to be entered.

#!/bin/sh
# /usr/local/sbin/unlock-smartcard -- by Simon Josefsson.
. /var/run/gpg-agent-info.env; export GPG_AGENT_INFO SSH_AUTH_SOCK SSH_AGENT_PID
gpg-connect-agent 'scd killscd' /bye > /dev/null
while ! gpg-connect-agent 'scd serialno' /bye | grep -q SERIALNO; do
    sleep 1
done
PINENTRY_USER_DATA=interactive
export PINENTRY_USER_DATA
checkpin

And the script checkpin is as follows:

#!/bin/sh
# /usr/local/sbin/checkpin -- by Simon Josefsson.
id=`gpg-connect-agent 'scd serialno' /bye | head -1 | cut -d  -f3`
gpg-connect-agent "scd checkpin $id" /bye | grep -q OK

At this point, you should have everything configured and installed. Don’t forget to chmod +x the scripts. The typical use-pattern is as follows. After the machine has been started, gpg-agent is running but the smartcard is not unlocked with the PIN. You need to manually login to the machine and run ‘unlock-smartcard’ and enter the PIN. In the script that runs the backup jobs, invoked via cron, make sure that the first line of the scripts reads (assuming Bourne shell script syntax):

. /var/run/gpg-agent-info.env; export GPG_AGENT_INFO SSH_AUTH_SOCK SSH_AGENT_PID

To avoid needlessly attempting ssh connections if the smartcard is not unlocked, your backup script can also call the checkpin code and abort if it doesn’t return true.

checkpin || exit 1

Some final words about debugging. A basic command to run to check that the GnuPG side is working is gpg --card-status, it should print some information about the smartcard if successful. To check that the SSH agent part is working, use ssh-add -L. If you get error messages, try killing the scdaemon process by running killall -9 scdaemon and let gpg-agent respawn a new scdaemon process.

That’s it! If you like my writeup, please flattr it.

OATH’s most visible specification has been the HOTP algorithm which is a way to generate event-based one-time passwords from a shared secret using HMAC-SHA1. HOTP has been published through the IETF as RFC 4226. Built on top of HOTP is the time-based variant called TOTP, which requires a clock in the token. OATH do some other work too, like specifying a data format for transferring the token configuration data (e.g., serial number and shared secret) called PSKC.

The aim of my project OATH Toolkit is to provide an implementation of various OATH related technologies. I’m intentionally leaving it open ended because you never know what they may specify that I find interesting. However, the primary goal has been to focus on HOTP and TOTP. Throughout 2010, the project was called HOTP Toolkit but that name made it difficult to support TOTP in a non-confusing way. During the last month, after discussion with Daniel Pocock on Dynalogin which is a potential consumer of my package, I took the time to create a fork of the HOTP Toolkit and the OATH Toolkit was born.

Now what does the OATH Toolkit actually do? Primarily it provides a library called liboath that implements HOTP and TOTP. Liboath is a relatively small library, and my goal is to keep it well documented and of high quality. There is GTK-DOC generated API documentation. Of course there is a command line tool to go with it, called oathtool which makes working with HOTP/TOTP from the command line easier. It can generate and validate one-time passwords. Let’s say you want to generate the first four OTP based on the dummy key 1234.

jas@latte:~$ oathtool -w4 1234
376439
299783
041392
819202
158134
jas@latte:~$

By default the tool is using HOTP, but you can switch it into TOTP mode with the –totp parameter. The output OTP will now depend on the current time on your machine, unless you specify the time manually using the –now parameter.

jas@latte:~$ oathtool –now=”2011-01-20 15:46 UTC” –totp 1234
527971
jas@latte:~$

The tool can do more, check the oathtool man page for all the details.

The final component of the OATH Toolkit is a PAM module pam_oath. With it, you can login to your machine using an OTP and optionally a password. Right now the user and password management is simplistic, but that should improve over time. To setup single-factor authentication for su you would create a file containing the user information and HOTP key as /etc/users.oath like this:

HOTP root – 1234

Then configure PAM to use the pam_oath module like this in /etc/pam.d/su:

auth requisite pam_oath.so debug usersfile=/etc/users.oath window=20

The user file will be rewritten every time you su to hold the current state. There is a README for the PAM-module with more documentation.

That’s it for an intro! From the OATH Toolkit webpage we link to binary packages for Debian and Ubuntu so please try the OATH Toolkit yourself and provide feedback to the oath-toolkit-help mailing list.

What is important is to demonstrate how GNU SASL can now talk IMAP authenticated with GS2-KRB5 using KfW on native Windows. Below are two screenshots. The first shows how the MIT Network Identity Manager is invoked automatically to get a ticket. The next screenshot is showing the completed GS2-KRB5 login.

I have prepared GNU SASL with KfW binaries. The manual contains a new section about Kerberos on Windows with details how to build it. The build makefile is available too, if you want to build your own.

wget -q ftp://alpha.gnu.org/gnu/gsasl/gsasl-1.5.0.tar.gz
tar xfz gsasl-1.5.0.tar.gz 
cd gsasl-1.5.0
./configure --prefix=$HOME CFLAGS=-g
make all check install

Next you need to configure Kerberos on your system. If you have already have a configuration, make sure to take a backup of your configuration files. First let’s tell your machine about my interop Kerberos V5 realm by creating a file ~/Library/Preferences/edu.mit.Kerberos with the following content:

[libdefaults]
        default_realm = interop.josefsson.org

[realms]
        interop.josefsson.org = {
                kdc = interop.josefsson.org:88
        }

[domain_realm]
        interop.josefsson.org = interop.josefsson.org

Next get a ticket for a dummy user. Use the password ‘pass’ at the prompt:

espresso:~ jas$ kinit user@interop.josefsson.org
Please enter the password for user@interop.josefsson.org: 
espresso:~ jas$ klist
Kerberos 5 ticket cache: 'API:Initial default ccache'
Default principal: user@interop.josefsson.org

Valid Starting     Expires            Service Principal
03/31/10 00:59:04  03/31/10 10:59:02  krbtgt/interop.josefsson.org@interop.josefsson.org
	renew until 01/01/70 01:00:00

klist: No Kerberos 4 tickets in credentials cache
espresso:~ jas$ 

Next connect to my test IMAP server using the command-line ‘gsasl’ tool (long lines cut for legibility):

espresso:~ jas$ ~/bin/gsasl interop.josefsson.org -auser
Trying `interop.josefsson.org'...
* OK IMAP4rev1
. CAPABILITY
* CAPABILITY ... AUTH=GS2-KRB5
. OK CAPABILITY Completed
. AUTHENTICATE GS2-KRB5
+ 
biwsAQBuggINMIICCa...
+ YHEGCSqGSIb3EgEC...

. OK AUTHENTICATE GS2-KRB5 authentication successful
Client authentication finished (server trusted)...
Enter application data (EOF to finish):
. list "" "*"
* LIST (NoInferiors) NIL INBOX
* LIST (NoInferiors) "/" .bashrc
* LIST (NoInferiors) "/" .bash_logout
* LIST (NoInferiors) "/" .profile
* LIST (NoInferiors) "/" .bash_history
. OK LIST Completed
. logout
* BYE Session terminating.
. OK LOGOUT Completed
Session finished...
. LOGOUT
espresso:~ jas$ 

There! You have just completed a GS2-KRB5 authenticated session.

Of course, this works just as well on your favorite GNU/Linux system, but I thought I’d explain it for a non-GNU platform and GSS-API library to show that the code is quite portable despite its experimental status.

Btw, the server is running GNU Shishi as the Kerberos V5 KDC and GNU MailUtils as the IMAP server. The IMAP server is using GNU SASL in server mode.

Now back to coding.

As you can see, my script looks for functions mentioned in the latest POSIX specification as being thread unsafe. In the last POSIX release, they actually removed some older interfaces (e.g., gethostbyname) so the script also checks for thread-unsafe functions mentioned in one older POSIX specification.

Russel’s approach is to look for man pages of functions ending with _r and labeling the non-_r-function as a thread unsafe function. Russel’s and my approach are quite different, so I wanted to compare the results. There is potential for me to add more functions to search for. I still want to preserve my approach of explicitly listing known thread unsafe functions, though.

Running Russel’s command, I get a list of functions that my script catches that Russel’s doesn’t, and vice versa. For reference, the functions that my script catches that Russel’s doesn’t are:

basename catgets dbm_clearerr dbm_close dbm_delete dbm_error dbm_fetch dbm_firstkey dbm_nextkey dbm_open dbm_store dirname dlerror endgrent endpwent endutxent ftw gcvt getc_unlocked getchar_unlocked getenv getopt getutxent getutxid getutxline inet_ntoa l64a lgamma lgammaf lgammal localeconv nftw nl_langinfo putc_unlocked putchar_unlocked putenv pututxline setenv setgrent setpwent setutxent strsignal system unsetenv wcstombs wctomb

The list contains lgamma, lgammaf, and lgammal which are all excluded by Russel’s command. I don’t understand why — according to the man page, the functions uses a global variable for sign, which doesn’t seem thread safe. So it seems right to include them?

What’s more interesting (for me) is the list of functions that Russel’s script catches that my script currently doesn’t. Here is the list:

erand48 ether_aton ether_ntoa fgetgrent fgetpwent fgetspent getaliasbyname getaliasent gethostbyname2 getmntent getnetgrent getrpcbyname getrpcbynumber getrpcent getspent getspnam getutent getutid getutline initstate jrand48 lcong48 nrand48 qecvt qfcvt random seed48 setstate sgetspent srand48 srandom tmpnam

I started looking into each function. For erand48 there is a erand48_r function in glibc, and the former does indeed seem to use a global variable. However, as far as I can tell from the POSIX specification, erand48 should be thread safe. So I filed a glibc bug about it. The same concern may hold for jrand48, lcong48, nrand48, seed48, and srand48.

I noticed that initstate, random, setstate, and srandom are defined by latest POSIX, but not mentioned as a thread-unsafe functions. Possibly a bug in the POSIX specification?

I also noticed that I had missed to include tmpnam even though it is mentioned separately in the POSIX link.

The rest of the functions are not documented by POSIX, and presumably thread unsafe (although I didn’t read the man page or source code for each of them).

In the end, I ended up adding several new functions to check for. The latest script is always available from:

http://git.savannah.gnu.org/cgit/gsasl.git/tree/tests/threadsafety

So, finally, did the updated script catch any use of thread-unsafe functions in GNU SASL? Nope.

I’m happy and proud to notice that I have been nominated for their award, for my work on security packages for the GNU project. Too bad I cannot make it to the conference this year.