Towards pluggable GSS-API modules

GSS-API is a standardized framework that is used by applications to, primarily, support Kerberos V5 authentication. GSS-API is standardized by IETF and supported by protocols like SSH, SMTP, IMAP and HTTP, and implemented by software projects such as OpenSSH, Exim, Dovecot and Apache httpd (via mod_auth_gssapi). The implementations of Kerberos V5 and GSS-API that are packaged for common GNU/Linux distributions, such as Debian, include MIT Kerberos, Heimdal and (less popular) GNU Shishi/GSS.

When an application or library is packaged for a GNU/Linux distribution, a choice is made which GSS-API library to link with. I believe this leads to two problematic consequences: 1) it is difficult for end-users to chose between Kerberos implementation, and 2) dependency bloat for non-Kerberos users. Let’s discuss these separately.

  1. No system admin or end-user choice over the GSS-API/Kerberos implementation used

    There are differences in the bug/feature set of MIT Kerberos and that of Heimdal’s, and definitely that of GNU Shishi. This can lead to a situation where an application (say, Curl) is linked to MIT Kerberos, and someone discovers a Kerberos related problem that would have been working if Heimdal was used, or vice versa. Sometimes it is possible to locally rebuild a package using another set of dependencies. However doing so has a high maintenance cost to track security fixes in future releases. It is an unsatisfying solution for the distribution to flip flop between which library to link to, depending on which users complain the most. To resolve this, a package could be built in two variants: one for MIT Kerberos and one for Heimdal. Both can be shipped. This can help solve the problem, but the question of which variant to install by default leads to similar concerns, and will also eventually leads to dependency conflicts. Consider an application linked to libraries (possible in several steps) where one library only supports MIT Kerberos and one library only supports Heimdal.

    The fact remains that there will continue to be multiple Kerberos implementations. Distributions will continue to support them, and will be faced with the dilemma of which one to link to by default. Distributions and the people who package software will have little guidance on which implementation to chose from their upstream, since most upstream support both implementations. The result is that system administrators and end-users are not given a simple way to have flexibility about which implementation to use.
  2. Dependency bloat for non-Kerberos use-cases.

    Compared to the number of users of GNU/Linux systems out there, the number of Kerberos users on GNU/Linux systems is smaller. Here distributions face another dilemma. Should they enable GSS-API for all applications, to satisfy the Kerberos community, or should they be conservative with adding dependencies to reduce attacker surface for the non-Kerberos users? This is a dilemma with no clear answer, and one approach has been to ship two versions of a package: one with Kerberos support and one without. Another option here is for upstream to support loadable modules, for example Dovecot implement this and Debian ship with a separate ‘dovecot-gssapi’ package that extend the core Dovecot seamlessly. Few except some larger projects appear to be willing to carry that maintenance cost upstream, so most only support build-time linking of the GSS-API library.

    There are a number of real-world situations to consider, but perhaps the easiest one to understand for most GNU/Linux users is OpenSSH. The SSH protocol supports Kerberos via GSS-API, and OpenSSH implement this feature, and most GNU/Linux distributions ship a SSH client and SSH server linked to a GSS-API library. Someone made the choice of linking it to a GSS-API library, for the arguable smaller set of people interested in it, and also the choice which library to link to. Rebuilding OpenSSH locally without Kerberos support comes with a high maintenance cost. Many people will not need or use the Kerberos features of the SSH client or SSH server, and having it enabled by default comes with a security cost. Having a vulnerability in OpenSSH is critical for many systems, and therefor its dependencies are a reasonable concern. Wouldn’t it be nice if OpenSSH was built in a way that didn’t force you to install MIT Kerberos or Heimdal? While still making it easy for Kerberos users to use it, of course.

Hopefully I have made the problem statement clear above, and that I managed to convince you that the state of affairs is in need of improving. I learned of the problems from my personal experience with maintaining GNU SASL in Debian, and for many years I ignored this problem.

Let me introduce Libgssglue!

Matryoshka Dolls
Matryoshka Dolls – photo CC-4.0-BY-NC by PngAll

Libgssglue is a library written by Kevin W. Coffman based on historical GSS-API code, the initial release was in 2004 (using the name libgssapi) and the last release was in 2012. Libgssglue provides a minimal GSS-API library and header file, so that any application can link to it instead of directly to MIT Kerberos or Heimdal (or GNU GSS). The administrator or end-user can select during run-time which GSS-API library to use, through a global /etc/gssapi_mech.conf file or even a local GSSAPI_MECH_CONF environment variable. Libgssglue is written in C, has no external dependencies, and is BSD-style licensed. It was developed for the CITI NFSv4 project but libgssglue ended up not being used.

I have added support to build GNU SASL with libgssglue — the changes required were only ./configure.ac-related since GSS-API is a standardized framework. I have written a fairly involved CI/CD check that builds GNU SASL with MIT Kerberos, Heimdal, libgssglue and GNU GSS, sets ups a local Kerberos KDC and verify successful GSS-API and GS2-KRB5 authentications. The ‘gsasl’ command line tool connects to a local example SMTP server, also based on GNU SASL (linked to all variants of GSS-API libraries), and to a system-installed Dovecot IMAP server that use the MIT Kerberos GSS-API library. This is on Debian but I expect it to be easily adaptable to other GNU/Linux distributions. The check triggered some (expected) Shishi/GSS-related missing features, and triggered one problem related to authorization identities that may be a bug in GNU SASL. However, testing shows that it is possible to link GNU SASL with libgssglue and have it be operational with any choice of GSS-API library that is shipped with Debian. See GitLab CI/CD code and its CI/CD output.

This experiment worked so well that I contacted Kevin to learn that he didn’t have any future plans for the project. I have adopted libgssglue and put up a Libgssglue GitLab project page, and pushed out a libgssglue 0.5 release fixing only some minor build-related issues. There are still some missing newly introduced GSS-API interfaces that could be added, but I haven’t been able to find any critical issues with it. Amazing that an untouched 10 year old project works so well!

My current next steps are:

  • Release GNU SASL with support for Libgssglue and encourage its use in documentation.
  • Make GNU SASL link to Libgssglue in Debian, to avoid a hard dependency on MIT Kerberos, but still allowing a default out-of-the-box Kerberos experience with GNU SASL.
  • Maintain libgssglue upstream and implement self-checks, CI/CD testing, new GSS-API interfaces that have been defined, and generally fix bugs and improve the project. Help appreciated!
  • Maintain the libgssglue package in Debian.
  • Look into if there are applications in Debian that link to a GSS-API library that could instead be linked to libgssglue to allow flexibility for the system administrator and reduce dependency bloat.

What do you think? Happy Hacking!

What’s wrong with SCRAM?

Simple Authentication and Security Layer (SASL, RFC4422) is the framework that was abstracted from the IMAP and POP protocols. Among the most popular mechanisms are PLAIN (clear-text passwords, usually under TLS), CRAM-MD5 (RFC2195), and GSSAPI (for Kerberos V5). The DIGEST-MD5 mechanism was an attempt to improve upon the CRAM-MD5 mechanism, but ended up introducing a lot of complexity and insufficient desirable features and deployment was a mess — read RFC6331 for background on why it has been deprecated.

SCRAM!

The effort to develop SCRAM (RFC5802) came, as far as I can tell, from the experiences with DIGEST-MD5 and the desire to offer something better than CRAM-MD5. In protocol design discussions, SCRAM is often still considered as “new” even though the specification was published in 2011 and even that had been in the making for several years. Developers that implement IMAP and SMTP still usually start out with supporting PLAIN and CRAM-MD5. The focus of this blog post is to delve into why this is and inspire the next step in this area. My opinion around this topic has existed for a couple of years already, formed while implementing SCRAM in GNU SASL, and my main triggers to write something about them now are 1) Martin Lambers‘ two-post blog series that first were negative about SCRAM and then became positive, and 2) my desire to work on or support new efforts in this area.

Let’s take a step back and spend some time analyzing PLAIN and CRAM-MD5. What are the perceived advantages and disadvantages?

Advantages: PLAIN and CRAM-MD5 solves the use-case of password-based user authentication, and are easy to implement.

Main disadvantages with PLAIN and CRAM-MD5:

  • PLAIN transfers passwords in clear text to the server (sometimes this is considered an advantage, but from a security point of view, it isn’t).
  • CRAM-MD5 requires that the server stores the password in plaintext (impossible to use a hashed or encrypted format).
  • Non-ASCII support was not there from the start.

A number of (debatable) inconveniences with PLAIN and CRAM-MD5 exists:

  • CRAM-MD5 does not support the notion of authorization identities.
  • The authentication is not bound to a particular secure channel, opening up for tunneling attacks.
  • CRAM-MD5 is based on HMAC-MD5 that is cryptographically “old” (but has withhold well) – the main problem today is that usually MD5 is not something you want to implement since there is diminishing other uses for it.
  • Servers can impersonate the client against other servers since they know the password.
  • Neither offer to authenticate the server to the client.

If you are familiar with SCRAM, you know that it solves these issues. So why hasn’t everyone jumped on it and CRAM-MD5 is now a thing of the past? In the first few years, my answer was that things take time and we’ll see improvements. Today we are ten years later; there are many SCRAM implementations out there, and the Internet has generally migrated away from protocols that have much larger legacy issues (e.g., SSL), but we are still doing CRAM-MD5. I think it is time to become critical of the effort and try to learn from the past. Here is my attempt at summarizing the concerns I’ve seen come up:

  • The mechanism family concept add complexity, in several ways:
    • The specification is harder to understand.
    • New instances of the mechanism family (SCRAM-SHA-256) introduce even more complexity since they tweak some of the poor choices made in the base specification.
    • Introducing new hashes to the family (like the suggested SHA3 variants) adds deployment costs since databases needs new type:value pairs to hold more than one “SCRAM” hashed password.
    • How to negotiate which variant to use is not well-defined. Consider if the server only has access to a SCRAM-SHA-1 hashed password for user X and a SCRAM-SHA-256 hashed password for user Y. What mechanisms should it offer to an unknown client? Offering both is likely to cause authentication failures, and the fall-back behaviour of SASL is poor.
  • The optional support for channel bindings and the way they are negotiated adds complexity.
  • The original default ‘tls-unique’ channel binding turned out to be insecure, and it cannot be supported in TLS 1.3.
  • Support for channel bindings requires interaction between TLS and SASL layers in an application.
  • The feature that servers cannot impersonate a client is dubious: the server only needs to participate in one authentication exchange with the client to gain this ability.
  • SCRAM does not offer any of the cryptographic properties of a Password-authenticated key agreement.

What other concerns are there? I’m likely forgetting some. Some of these are debatable and were intentional design choices.

Can we save SCRAM? I’m happy to see the effort to introduce a new channel binding and update the SCRAM specifications to use it for TLS 1.3+. I brought up a similar approach back in the days when some people were still insisting on ‘tls-unique’. A new channel binding solves some of the issues above.

It is hard to tell what the main reason for not implementing SCRAM more often is. A sense of urgency appears to be lacking. My gut feeling is that to an implementer SCRAM looks awfully similar to DIGEST-MD5. Most of the problems with DIGEST-MD5 could be fixed, but the fixes add more complexity.

How to proceed from here? I see a couple of options:

  • Let time go by to see increased adoption. Improving the channel binding situation will help.
  • Learn from the mistakes and introduce a new simple SCRAM, which could have the following properties:
    • No mechanism family, just one mechanism instance.
    • Hash is hard-coded, just like CRAM-MD5.
    • TLS and a channel binding is required and always used.
  • Review one of the PAKE alternatives and specify a SASL mechanism for it. Preferably without repeating the mistakes of CRAM-MD5, DIGEST-MD5 and SCRAM.
  • Give up on having “complex” authentication mechanisms inside SASL, and help some PAKE variant become implemented through a TLS library, and SASL applications should just use EXTERNAL to use TLS user authentication.

Thoughts?

I feel the following XKCD is appropriate here.

GNU SASL with SCRAM-SHA-1-PLUS

I have finished the SCRAM implementation in GNU SASL. The remaining feature to be added were support for the “enhanced” SCRAM-SHA-1-PLUS variant instead of just the normal SCRAM-SHA-1 mechanism. The difference is that the latter supports channel bindings to TLS, which makes it possible to detect man-in-the-middle attacks even if TLS is not used with server authentication. In GnuTLS we recently added an API for applications to extract channel bindings, which you will need to use in order to use SCRAM-SHA-1-PLUS. I announced the experimental version 1.5.4 release together with a writeup on how to test it. With this, our support for SCRAM should be complete.

GS2-KRB5 in GNU SASL 1.5.0

I have worked in the IETF on the specification for the next generation GSSAPI-to-SASL bridge called GS2 (see my status page for background) for a couple of years now. The specification is (finally!) in the RFC editor’s queue, and is supposed to be stable and final although we are still tuning some details. The next step is to implement the protocol and do interop testing. A couple of months of implementation and testing work culminated in tonight’s release of GNU SASL 1.5.0 (see announcement here). Or should I say that the work can now begin…
Continue reading GS2-KRB5 in GNU SASL 1.5.0

Thread Safe Functions

I have read Russel Coker’s nice article on identifying use of thread unsafe functions. This reminded me of a script I wrote a long time ago that is part of GNU SASL‘s regression suite: threadsafety.

As you can see, my script looks for functions mentioned in the latest POSIX specification as being thread unsafe. In the last POSIX release, they actually removed some older interfaces (e.g., gethostbyname) so the script also checks for thread-unsafe functions mentioned in one older POSIX specification.

Russel’s approach is to look for man pages of functions ending with _r and labeling the non-_r-function as a thread unsafe function. Russel’s and my approach are quite different, so I wanted to compare the results. There is potential for me to add more functions to search for. I still want to preserve my approach of explicitly listing known thread unsafe functions, though.

Running Russel’s command, I get a list of functions that my script catches that Russel’s doesn’t, and vice versa. For reference, the functions that my script catches that Russel’s doesn’t are:

basename catgets dbm_clearerr dbm_close dbm_delete dbm_error dbm_fetch dbm_firstkey dbm_nextkey dbm_open dbm_store dirname dlerror endgrent endpwent endutxent ftw gcvt getc_unlocked getchar_unlocked getenv getopt getutxent getutxid getutxline inet_ntoa l64a lgamma lgammaf lgammal localeconv nftw nl_langinfo putc_unlocked putchar_unlocked putenv pututxline setenv setgrent setpwent setutxent strsignal system unsetenv wcstombs wctomb

The list contains lgamma, lgammaf, and lgammal which are all excluded by Russel’s command. I don’t understand why — according to the man page, the functions uses a global variable for sign, which doesn’t seem thread safe. So it seems right to include them?

What’s more interesting (for me) is the list of functions that Russel’s script catches that my script currently doesn’t. Here is the list:

erand48 ether_aton ether_ntoa fgetgrent fgetpwent fgetspent getaliasbyname getaliasent gethostbyname2 getmntent getnetgrent getrpcbyname getrpcbynumber getrpcent getspent getspnam getutent getutid getutline initstate jrand48 lcong48 nrand48 qecvt qfcvt random seed48 setstate sgetspent srand48 srandom tmpnam

I started looking into each function. For erand48 there is a erand48_r function in glibc, and the former does indeed seem to use a global variable. However, as far as I can tell from the POSIX specification, erand48 should be thread safe. So I filed a glibc bug about it. The same concern may hold for jrand48, lcong48, nrand48, seed48, and srand48.

I noticed that initstate, random, setstate, and srandom are defined by latest POSIX, but not mentioned as a thread-unsafe functions. Possibly a bug in the POSIX specification?

I also noticed that I had missed to include tmpnam even though it is mentioned separately in the POSIX link.

The rest of the functions are not documented by POSIX, and presumably thread unsafe (although I didn’t read the man page or source code for each of them).

In the end, I ended up adding several new functions to check for. The latest script is always available from:

http://git.savannah.gnu.org/cgit/gsasl.git/tree/tests/threadsafety

So, finally, did the updated script catch any use of thread-unsafe functions in GNU SASL? Nope.

Password-based Authentication Protocol

There was a large increase in activity on password-based SASL authentication mechanism in the Prague IETF, with three new proposals. Unfortunately, I was travelling over the I-D cutoff, so my document didn’t make it. However, I’ve now finished a -00 document for it. The goal was initially to just specify a GSS-API mechanism, but it seemed easier to specify a framework-agnostic protocol (with some influences from GSS-API and SASL) and then specify the mapping to GSS-API and SASL.

http://josefsson.org/password-auth/