Let’s Encrypt Clients

As many others, I have been following the launch of Let’s Encrypt. Let’s Encrypt is a new zero-cost X.509 Certificate Authority that supports the Automated Certificate Management Environment (ACME) protocol. ACME allow you to automate creation and retrieval of HTTPS server certificates. As anyone who has maintained a number of HTTPS servers can attest, this process has unfortunately been manual, error-prone and differ between CAs.

On some of my personal domains, such as this blog.josefsson.org, I have been using the CACert authority to sign the HTTPS server certificate. The problem with CACert is that the CACert trust anchors aren’t shipped with sufficient many operating systems and web browsers. The user experience is similar to reaching a self-signed server certificate. For organization-internal servers that you don’t want to trust external parties for, I continue to believe that running your own CA and distributing it to your users is better than using a public CA (compare my XMPP server certificate setup). But for public servers, availability without prior configuration is more important. Therefor I decided that my public HTTPS servers should use a CA/Browser Forum-approved CA with support for ACME, and as long as Let’s Encrypt is trustworthy and zero-cost, they are a good choice.

I was in need of a free software ACME client, and set out to research what’s out there. Unfortunately, I did not find any web pages that listed the available options and compared them. The Let’s Encrypt CA points to the “official” Let’s Encrypt client, written by Jakub Warmuz, James Kasten, Peter Eckersley and several others. The manual contain pointers to two other clients in a seamingly unrelated section. Those clients are letsencrypt-nosudo by Daniel Roesler et al, and simp_le by (again!) Jakub Warmuz. From the letsencrypt.org’s client-dev mailing list I also found letsencrypt.sh by Gerhard Heift and LetsEncryptShell by Jan Mojžíš. Is anyone aware of other ACME clients?

By comparing these clients, I learned what I did not like in them. I wanted something small so that I can audit it. I want something that doesn’t require root access. Preferably, it should be able to run on my laptop, since I wasn’t ready to run something on the servers. Generally, it has to be Secure, which implies something about how it approaches private key handling. The letsencrypt official client can do everything, and has plugin for various server software to automate the ACME negotiation. All the cryptographic operations appear to be hidden inside the client, which usually means it is not flexible. I really did not like how it was designed, it looks like your typical monolithic proof-of-concept design. The simp_le client looked much cleaner, and gave me a good feeling. The letsencrypt.sh client is simple and written in /bin/sh shell script, but it appeared a bit too simplistic. The LetsEncryptShell looked decent, but I wanted something more automated.

What all of these clients did not have, and that letsencrypt-nosudo client had, was the ability to let me do the private-key operations. All the operations are done interactively on the command-line using OpenSSL. This would allow me to put the ACME user private key, and the HTTPS private key, on a YubiKey, using its PIV applet and techniques similar to what I used to create my SSH host CA. While the HTTPS private key has to be available on the HTTPS server (used to setup TLS connections), I wouldn’t want the ACME user private key to be available there. Similarily, I wouldn’t want to have the ACME or the HTTPS private key on my laptop. The letsencrypt-nosudo tool is otherwise more rough around the edges than the more cleaner simp_le client. However the private key handling aspect was the deciding matter for me.

After fixing some hard-coded limitations on RSA key sizes, getting the cert was as simple as following the letsencrypt-nosudo instructions. I’ll follow up with a later post describing how to put the ACME user private key and the HTTPS server certificate private key on a YubiKey and how to use that with letsencrypt-nosudo.

So you can now enjoy browsing my blog over HTTPS! Thank you Let’s Encrypt!

Offline GnuPG Master Key and Subkeys on YubiKey NEO Smartcard

I have moved to a new OpenPGP key. There are many tutorials and blog posts on GnuPG key generation around, but none of them matched exactly the setup I wanted to have. So I wrote down the steps I took, to remember them if I need to in the future. Briefly my requirements were as follows:

  • The new master GnuPG key is on an USB stick.
  • The USB stick is only ever used on an offline computer.
  • There are subkeys stored on a YubiKey NEO smartcard for daily use.
  • I want to generate the subkeys using GnuPG so I have a backup.
  • Some non-default hash/cipher preferences encoded into the public key.

Continue reading Offline GnuPG Master Key and Subkeys on YubiKey NEO Smartcard

OpenPGP Key Transition Statement

I have created a new OpenPGP key 54265e8c and will be transitioning away from my old key. If you have signed my old key, I would appreciate signatures on my new key as well. I have created a transition statement that can be downloaded from https://josefsson.org/key-transition-2014-06-22.txt.

Below is the signed statement.

Hash: SHA512

OpenPGP Key Transition Statement for Simon Josefsson

I have created a new OpenPGP key and will be transitioning away from
my old key.  The old key has not been compromised and will continue to
be valid for some time, but I prefer all future correspondence to be
encrypted to the new key, and will be making signatures with the new
key going forward.

I would like this new key to be re-integrated into the web of trust.
This message is signed by both keys to certify the transition.  My new
and old keys are signed by each other.  If you have signed my old key,
I would appreciate signatures on my new key as well, provided that
your signing policy permits that without re-authenticating me.

The old key, which I am transitioning away from, is:

pub   1280R/B565716F 2002-05-05
      Key fingerprint = 0424 D4EE 81A0 E3D1 19C6  F835 EDA2 1E94 B565 716F

The new key, to which I am transitioning, is:

pub   3744R/54265E8C 2014-06-22
      Key fingerprint = 9AA9 BDB1 1BB1 B99A 2128  5A33 0664 A769 5426 5E8C

The entire key may be downloaded from: https://josefsson.org/54265e8c.txt

To fetch the full new key from a public key server using GnuPG, run:

  gpg --keyserver keys.gnupg.net --recv-key 54265e8c

If you already know my old key, you can now verify that the new key is
signed by the old one:

  gpg --check-sigs 54265e8c

If you are satisfied that you've got the right key, and the User IDs
match what you expect, I would appreciate it if you would sign my key:

  gpg --sign-key 54265e8c

You can upload your signatures to a public keyserver directly:

  gpg --keyserver keys.gnupg.net --send-key 54265e8c

Or email simon@josefsson.org (possibly encrypted) the output from:

  gpg --armor --export 54265e8c

If you'd like any further verification or have any questions about the
transition please contact me directly.

To verify the integrity of this statement:

  wget -q -O- https://josefsson.org/key-transition-2014-06-22.txt|gpg --verify

Version: GnuPG v1.4.12 (GNU/Linux)


Replicant 4.2 on Samsung S3

Since November 2013 I have been using Replicant on my Samsung S3 as an alternative OS. The experience has been good for everyday use. The limits (due to non-free software components) compared to a “normal” S3 (running vendor ROM or CyanogenMod) is lack of GPS/wifi/bluetooth/NFC/frontcamera functionality — although it is easy to get some of that working again, including GPS, which is nice for my geocaching hobby. The Replicant software is stable for being an Android platform; better than my Nexus 7 (2nd generation) tablet which I got around the same time that runs an unmodified version of Android. The S3 has crashed around ten times in these four months. I’ve lost track of the number of N7 crashes, especially after the upgrade to Android 4.4. I use the N7 significantly less than the S3, reinforcing my impression that Replicant is a stable Android. I have not had any other problem that I couldn’t explain, and have rarely had to reboot the device.

The Replicant project recently released version 4.2 and while I don’t expect the release to resolve any problem for me, I decided it was time to upgrade and learn something new. I initially tried the official ROM images, and later migrated to using my own build of the software (for no particular reason other than that I could).

Before the installation, I wanted to have a full backup of the phone to avoid losing data. I use SMS Backup+ to keep a backup of my call log, SMS and MMS on my own IMAP server. I use oandbackup to take a backup of all software and settings on the phone. I use DAVDroid for my contacts and calendar (using a Radicale server), and reluctantly still use aCal in order to access my Google Calendar (because Google does not implement RFC 5397 properly so it doesn’t work with DAVDroid). Alas all that software is not sufficient for backup purposes, for example photos are still not copied elsewhere. In order to have a complete backup of the phone, I’m using rsync over the android debug bridge (adb). More precisely, I connect the phone using a USB cable, push a rsyncd configuration file, start the rsync daemon on the phone, forward the TCP/IP port, and then launch rsync locally. The following commands are used:

jas@latte:~$ cat rsyncd.conf
uid = root
gid = root
path = /
jas@latte:~$ adb push rsyncd.conf /extSdCard/rsyncd.conf
* daemon not running. starting it now on port 5037 *
* daemon started successfully *
0 KB/s (57 bytes in 0.059s)
jas@latte:~$ adb root
jas@latte:~$ adb shell rsync --daemon --no-detach --config=/extSdCard/rsyncd.conf &
jas@latte:~$ adb forward tcp:6010 tcp:873
jas@latte:~$ sudo rsync -av --delete --exclude /dev --exclude /acct --exclude /sys --exclude /proc rsync://localhost:6010/root/ /root/s3-bup/

Now feeling safe that I would not lose any data, I remove the SIM card from my phone (to avoid having calls, SMS or cell data interrupt during the installation) and follow the Replicant Samsung S3 installation documentation. Installation was straightforward. I booted up the newly installed ROM and familiarized myself with it. My first reaction was that the graphics felt a bit slower compared to Replicant 4.0, but it is hard to tell for certain.

After installation, I took a quick rsync backup of the freshly installed phone, to have a starting point for future backups. Since my IMAP and CardDav/CalDav servers use certificates signed by CACert I first had to install the CACert trust anchors, to get SMS Backup+ and DAVDroid to connect. For some reason it was not sufficient to add only the root CACert certificate, so I had to add the intermediate CA cert as well. To load the certs, I invoke the following commands, selecting ‘Install from SD Card’ when the menu is invoked (twice).

adb push root.crt /sdcard/
adb shell am start -n "com.android.settings/.Settings\"\$\"SecuritySettingsActivity"
adb push class3.crt /sdcard/
adb shell am start -n "com.android.settings/.Settings\"\$\"SecuritySettingsActivity"

I restore apps with oandbackup, and I select a set of important apps that I want restored with settings preserved, including aCal, K9, Xabber, c:geo, OsmAnd~, NewsBlur, Google Authenticator. I install SMS Backup+ from FDroid separately and configure it, SMS Backup+ doesn’t seem to want to restore anything if the app was restored with settings using oandbackup. I install and configure the DAVdroid account with the server URL, and watch it populate my address book and calendar with information.

After organizing the icons on the launcher screen, and changing the wallpaper, I’m up and running with Replicant 4.2. This upgrade effort took me around two evenings to complete, with around half of the time consumed by exploring different ways to do the rsync backup before I settled on the rsync daemon approach. Compared to the last time, when I spent almost two weeks researching various options and preparing for the install, this felt like a swift process.

Continue reading Replicant 4.2 on Samsung S3

Necrotizing Fasciitis

Dear World,

On the morning of December 24th I felt an unusual pain in my left hand between the thumb and forefinger. The pain increased and in the afternoon I got a high fever, at some point above 40 degrees Celsius or 104 degree Fahrenheit. I went to the emergency department and was hospitalized during the night between the 24th and 25th of December. On the afternoon of December 26th I underwent surgery to find out what was happening, and was then diagnosed with Necrotizing Fasciitis (the wikipedia article on NF gives a fair summary), caused by the common streptococcus bacteria (again see wikipedia article on Streptococcus). A popular name for the disease is flesh-eating bacteria. Necrotizing Fasciitis is a rare and aggresive infection, often deadly if left untreated, that can move through the body at speeds of a couple of centimeters per hour.

I have gone through 6 surgeries, leaving wounds all over my left hand and arm. I have felt afraid of what the disease will do to me, anxiety over what will happen in the future, confusion and uncertainty about how a disease like this can exist and that I get the right treatment since so little appears to be known about it. The feeling of loneliness and that nobody is helping, or even can help, has also been present. I have experienced pain. Even though pain is something I’m less afraid of (I have a back problem) compared to other feelings, I needed help from several pain killers. I’ve received normal Paracetamol, stronger NSAID’s (e.g., Ketorolac/Toradol), several Opioid pain-killers including Alfentanil/Rapifen, Tramadol/Tradolan, OxyContin/OxyNorm, and Morphine. After the first and second surgery, nothing helped and I was still screaming with pain and kicking the bed. After the first surgery, I received a local anesthetic (a plexus block). After the second surgery, the doctors did not want to masquerade my pain, because sign of pain indicate further growth of the infection, and I was given the pain-dissociative drug Ketamine/Ketalar and the stress-releasing Clonidine/Catapresan. Once the third surgery removed all of the infection, pain went down, and I experienced many positive feelings. I am very grateful to be alive. I felt a strong sense of inner power when I started to fight back against the decease. I find joy in even the simplest of things, like being able to drink water or seeing trees outside the window. I cried out of happiness when I saw our children’s room full of toys. I have learned many things about the human body, and I am curious by nature so I look forward to learn more. I hope to be able to draw strength from this incident, to help me prioritize better in my life.

My loving wife Åsa has gone through a nightmare as a consequence of my diagnosis. At day she had to cope with daily life taking care of our wonderful 1-year old daughter Ingrid and 3-year old boy Alfred. All three of them had various degrees of strep throat with fever, caused by the same bacteria — and anyone with young kids know how intense that alone can be. She gave me strength over the phone. She kept friends and relatives up to date about what happened, with the phone ringing all the time. She worked to get information out from the hospital about my status, sometimes being rudely treated and just being hanged up on. After a call with the doctor after the third surgery, when the infection had spread from the hand to the upper arm (5cm away from my torso), she started to plan for a life without me.

My last operation were on Thursday January 2nd and I left hospital the same day. I’m writing this on the Saturday of January 4rd, although some details and external links have been added after that. I have regained access to my arm and hand and doing rehab to regain muscle control, while my body is healing. I’m doing relaxation exercises to control pain and relax muscles, and took the last strong drug yesterday. Currently I take antibiotics (more precisely Clindamycin/Dalacin) and the common Paracetamol-based pain-killer Alvedon together with on-demand use of an also common NSAID containing Ibuprofen (Ipren). My wife and I were even out at a restaurant tonight.

Fortunately I was healthy when this started, and with bi-weekly training sessions for the last 2 years I was physically at my strongest peak in my 38 year old life (weighting 78kg or 170lb, height 182cm or 6 feet). I started working out to improve back issues, increase strength, and prepare for getting older. Exercise has never been my thing although I think it is fun to run medium distances (up to 10km).

I want thank everyone who helped me and our family through this, both professionally and personally, but I don’t know where to start. You know who you are. You are the reason I’m alive.

Naturally, I want to focus on getting well and spend time with my family now. I don’t yet know to what extent I will recover, but the prognosis is good. Don’t expect anything from me in the communities and organization that I’m active in (e.g., GNU, Debian, IETF, Yubico). I will come back as energy, time and priorities permits.

Nordic Free Software Award 2009

Last night at FSCONS I was awarded the Nordic Free Software Award, sharing the price with Daniel Stenberg who incidentally (or perhaps not) I have been collaborating with on some projects. Receiving a price like this is a great motivator and I feel humbled when thinking about the many excellent hackers that were attending the FSCONS that cheered me on. Thank you everyone.

Now back to coding.

Thread Safe Functions

I have read Russel Coker’s nice article on identifying use of thread unsafe functions. This reminded me of a script I wrote a long time ago that is part of GNU SASL‘s regression suite: threadsafety.

As you can see, my script looks for functions mentioned in the latest POSIX specification as being thread unsafe. In the last POSIX release, they actually removed some older interfaces (e.g., gethostbyname) so the script also checks for thread-unsafe functions mentioned in one older POSIX specification.

Russel’s approach is to look for man pages of functions ending with _r and labeling the non-_r-function as a thread unsafe function. Russel’s and my approach are quite different, so I wanted to compare the results. There is potential for me to add more functions to search for. I still want to preserve my approach of explicitly listing known thread unsafe functions, though.

Running Russel’s command, I get a list of functions that my script catches that Russel’s doesn’t, and vice versa. For reference, the functions that my script catches that Russel’s doesn’t are:

basename catgets dbm_clearerr dbm_close dbm_delete dbm_error dbm_fetch dbm_firstkey dbm_nextkey dbm_open dbm_store dirname dlerror endgrent endpwent endutxent ftw gcvt getc_unlocked getchar_unlocked getenv getopt getutxent getutxid getutxline inet_ntoa l64a lgamma lgammaf lgammal localeconv nftw nl_langinfo putc_unlocked putchar_unlocked putenv pututxline setenv setgrent setpwent setutxent strsignal system unsetenv wcstombs wctomb

The list contains lgamma, lgammaf, and lgammal which are all excluded by Russel’s command. I don’t understand why — according to the man page, the functions uses a global variable for sign, which doesn’t seem thread safe. So it seems right to include them?

What’s more interesting (for me) is the list of functions that Russel’s script catches that my script currently doesn’t. Here is the list:

erand48 ether_aton ether_ntoa fgetgrent fgetpwent fgetspent getaliasbyname getaliasent gethostbyname2 getmntent getnetgrent getrpcbyname getrpcbynumber getrpcent getspent getspnam getutent getutid getutline initstate jrand48 lcong48 nrand48 qecvt qfcvt random seed48 setstate sgetspent srand48 srandom tmpnam

I started looking into each function. For erand48 there is a erand48_r function in glibc, and the former does indeed seem to use a global variable. However, as far as I can tell from the POSIX specification, erand48 should be thread safe. So I filed a glibc bug about it. The same concern may hold for jrand48, lcong48, nrand48, seed48, and srand48.

I noticed that initstate, random, setstate, and srandom are defined by latest POSIX, but not mentioned as a thread-unsafe functions. Possibly a bug in the POSIX specification?

I also noticed that I had missed to include tmpnam even though it is mentioned separately in the POSIX link.

The rest of the functions are not documented by POSIX, and presumably thread unsafe (although I didn’t read the man page or source code for each of them).

In the end, I ended up adding several new functions to check for. The latest script is always available from:


So, finally, did the updated script catch any use of thread-unsafe functions in GNU SASL? Nope.

Redmine on Debian Lenny Using Lighttpd

The GnuTLS trac installation is in a poor shape. To fix that, I looked into alternatives and found Redmine. Redmine appears to do most things that I liked in Trac (wiki, roadmap and issue tracking) plus it supports more than one project (would come in handy for my other projects) and has built-in git support. I would like to see better spam handling and OpenID support, but it is good enough for our purposes now, and there are similar concerns with trac.

However, getting it up and running with lighttpd on a modern debian lenny installation was not trivial, and I needed some help from #redmine (thanks stbuehler). After finally getting it up and running, I made a copy of the machine using rsync and rsnapshot, so I could re-create a working configuration if I get stuck, and then re-installed the virtual machine.

The notes below are the steps required to set up Redmine using Lighttpd and MySQL on a Debian Lenny. I’m posting this to help others searching for the error messages I got, and to help my own memory in case I need to re-install the server sometime.
Continue reading Redmine on Debian Lenny Using Lighttpd

FSCONS / Nordic Free Software Award Nomination

The Free Software & Culture conference FSCONS is held in Gothenburg October 24-26th. Having been there and given talks last year, I can recommend it for anyway interested in what’s going on the free software and culture world.

I’m happy and proud to notice that I have been nominated for their award, for my work on security packages for the GNU project. Too bad I cannot make it to the conference this year.