How not to respond to POODLEBLEED

November 12, 2014

Or, how not to make all your websites unavailable to users (and crawlers) running browsers that don’t support TLSv1.2 (i.e. most of them1).

Yes, this is a post in which I admit to a mistake and potentially make myself look silly. But if it helps one person avoid or correct a similar mistake it’ll have been worth it.

Sometime around the middle of October I finally got around to responding to POODLEBLEED (I took my time, but proper TLS on my sites is a nice-to-have, not a must-have, and I was busy I’m lazy).

The way I responded to POODLEBLEED was by applying the intermediate compatibility  SSL settings recommended by Mozilla to the Apache configuration that runs dhpiggott.net, piggott.me.uk and dhpcs.com.

I did this by writing the following to /etc/apache2/conf-available/ssl.conf (and then running a2enconf ssl and restarting Apache):

SSLProtocol TLSv1 TLSv1.1 TLSv1.2
SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA

SSLHonorCipherOrder On
SSLCompression Off

SSLUseStapling On
SSLStaplingResponderTimeout 5
SSLStaplingReturnResponderErrors Off
SSLStaplingCache shmcb:/var/run/ocsp(128000)

Though disabling SSLv3 like this was an appropriate response to POODLEBLOOD and it increased the security of my sites, in applying it I also inadvertently and unnecessarily decreased the availability of them.

Of course, as far as I could tell, everything was working just fine (I tested access from Chrome on Ubuntu [14.10] and Chrome on Android [4.4]). If I’d had reason to believe anything was wrong I would have disabled the configuration and figured out what was wrong before re-enabling a corrected version.

A week or two after I’d applied this configuration it was brought to my attention by Ben Phillips of thread.com that none of my sites could be accessed using his install of Chrome (38.0.2125.104) on Debian. The error was ERR_SSL_VERSION_OR_CIPHER_MISMATCH.

I increased the level of logging for mod_ssl to debug and asked Ben to retry. This is what I saw on Apache’s side:

[Thu Oct 30 16:42:13.965487 2014] [ssl:info] [pid 30721] [client 192.0.2.1:54079] AH01964: Connection to child 11 established (server maple.dhpiggott.net:443)
[Thu Oct 30 16:42:13.965833 2014] [ssl:info] [pid 30721] [client 192.0.2.1:54079] AH02008: SSL library error 1 in handshake (server maple.dhpiggott.net:443)
[Thu Oct 30 16:42:13.965869 2014] [ssl:info] [pid 30721] SSL Library Error: error:1408A10B:SSL routines:SSL3_GET_CLIENT_HELLO:wrong version number
[Thu Oct 30 16:42:13.965881 2014] [ssl:info] [pid 30721] [client 192.0.2.1:54079] AH01998: Connection closed to child 11 with abortive shutdown (server maple.dhpiggott.net:443)

(that IP address is one reserved for examples/documentation as per RFC5737).

While looking through the error logs I noticed many failures like that above – from various IP addresses. I identified one as Googlebot. I then did a search for a query that my blog normally ranks highly for2. I wasn’t surprised to find that it was not included in the results.

Puzzled as to why many users and crawlers were failing to connect, I created a  Debian Wheezy VM and proceeded to replicate the problem. The included Iceweasel (Firefox) browser worked just fine. I then tried Chromium. It did not.

To cut a slightly long story a bit shorter I’ll skip details of the one step I tried that didn’t lead anywhere – namely using ssltap from libnss3-tools to capture traces of Chromium trying to talk to Apache.

Ultimately the root cause came to me when I noticed a pattern in which browsers were and weren’t working and then reviewing the relevant documentation, including Mozilla’s guide:

Yes, the clients that were failing to connect were those that did not support TLSv1.2.

I said before that I was using the intermediate compatibility Apache configuration recommended by Mozilla at https://wiki.mozilla.org/Security/Server_Side_TLS#Intermediate_compatibility_.28default.29, which specifies versions TLSv1, TLSv1.1 and TLSv1.2. And because I find explicitly specifying things more readable I had done just that and written SSLProtocol TLSv1 TLSv1.1 TLSv1.2.

But the SSLProtocol line in the exemplar Apache stanza is actually all -SSLv2 -SSLv3.

The problem was that order matters. To specify things explicitly for Apache I should have written it as TLSv1.2 TLSv1.1 TLSv1. What I actually did had the effect of making Apache only talk to clients that support TLSv1.2.

I corrected /etc/apache2/conf-available/ssl.conf to be as follows and restarted Apache:

SSLProtocol All -SSLv2 -SSLv3
SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA

SSLHonorCipherOrder On
SSLCompression Off

SSLUseStapling On
SSLStaplingResponderTimeout 5
SSLStaplingReturnResponderErrors Off
SSLStaplingCache shmcb:/var/run/ocsp(128000)

With that done, accessibility to clients without TLSv1.2 support was restored. And I’m pleased to note at the time of writing this that searches for the query I referred to before once again include a result from piggott.me.uk.

Thanks are due to Ben Phillips for alerting me to this problem and helping me get to a point where I could replicate it.


  1. Unfortunately part way through this escapade – after changing to the broken SSL configuration and before noticing that it was broken – I switched from AWStats to Piwik for monitoring traffic. As such, though I had noticed that traffic was lower than normal, I had put that down to a difference in what Piwik logs. I was keeping an eye on it but with fairly low priority. If it wasn’t for that switch midway through the affected period, I’d be able to use the data to get an idea of just how many visitors were affected. Of course I could do that with the raw Apache logs, which are still present, but as I said earlier, I’m too busy lazy.
  2. “inferring transportation mode using smartphone sensor data”.