Note: [04/14/14] Today I was contacted by GlobalSign representative Gregory who stumbled over this blog post, and he was so kind to revoke the affected certs free of charge. He also added, in response to my summary below, that there is an option in the Chrome settings to enable revocation checking, and that beginning April 1, 2015, GlobalSign will restrict the maximum validity of then-issued certs to 39 months.

So the Heartbleed bug (CVE-2014-0160) is out, and every administrator using SSL to protect his infrastructure has been wondering the same thing: should I absolutely, positively, without a doubt, replace all certificates and associates keys?

The only reasonable answer is: yes – if you used certificates on a vulnerable machine. Even those in disbelief were quickly proven wrong.

The first thing I did was to patch all impacted OpenSSL instances and restart the services that depend on the OpenSSL library (that includes not only HTTP but also MTA and IMAP, among others). That was the easy part.

What followed was a major pain with my certificate authority and one of its partners.

We’ll reissue your heartbleeded cert… for a fee

I own a couple of domain-validated wildcard certs from AlphaSSL/GlobalSign, which I acquired dirt-cheap from one of its resellers. Since the reseller doesn’t allow me to manage the certs myself, I contacted them directly in the aftermath of heartbleed:

Dear Digitalt support,

I registered a handful of AlphaSSL certificates with you. In light of the just released heartbleed exploit (http://heartbleed.com/) we are in need to reissue the certificates with new private keys. Would that be possible through you or do we have to contact AlphaSSL directly?

Thanks for your help.

The response was quick:

Hi Alex

We can reissue them manuel if you send us new CSR but we do charge 200 DKK per reissue to cover the manual work.

Duh. 200 DDK ~ USD 35 – that’s more than I paid for the cert!

… but we still do recommend that you reissue your cert

In light of the special circumstances of the Heardbleed bug I decided to contact AlphaSSL directly instead.

Dear AlphaSSL support,

The certificate with the order number CE2014xxxxxxxx (*.xxx.com) would need to be reissued due to the heartbleed OpenSSL exploit that was recently discovered. Would that be possible?

Sincerely

Again a quick response, this time directly from GlobalSign support:

Dear Alexander,

Thank you for contacting GlobalSign Support Team!

In line with this OpenSSL known published bug update, the best thing that we can recommend is to: 1) update openssl, 2) generate new key and csr, 3) reissue certificate, 4) revoke the old certificate, 5) install the new certificate on the updated server.

It is an utmost importance to ensure that the OpenSSL is already updated before even starting with the reissue process. We also advice to revoke the old certificate only after it has already been re-issued.

To reissue the certificate, you can refer to the link below as a guide:
https://support.globalsign.com/customer/portal/articles/1223116-reissue-certificate—ssl-certificates

To revoke the certificate, you can refer to the link below as a guide:
https://support.globalsign.com/customer/portal/articles/1251577

You can also refer to the related articles below that maybe of interest about the known and published “HeartBleed” bug for OpenSSL:
http://techcrunch.com/2014/04/07/massive-security-bug-in-openssl-could-effect-a-huge-chunk-of-the-internet/
http://www.theregister.co.uk/2014/04/08/running_openssl_patch_now_to_fix_critical_bug/
http://rehmann.co/projects/heartbeat/

Details and a comprehensive FAQ can be found here: heartbleed.com

Please feel free to contact us if you need further assistance.

Well, at least we seem to agree about the need to rekey my certs. The links above pointing to the “GlobalSign Certificate Center (GCC)” account didn’t work for me – presumably because I didn’t buy the AlphaSSL certs directly through GlobalSign. Thus I replied:

Dear Kenneth,

Thank you for reply and the detailed information. We have updated OpenSSL to fix the heartbleed bug, and we are now the process of reissuing our affected AlphaSSL certificates. We didn’t buy the certificates through the GlobalSign Certificate Center, but through a reseller, who is now saying that a reissue would require manual work and would be charged at ca. US $35 per certificate. Since we bought the certificates at a favorable discount through that reseller, I don’t feel like to argue, but I was hoping, given the circumstances (major OpenSSL exploit, risk of private AlphaSSL cert keys having been exposed), a reissue would be possible without additional cost – which is why I contacted you directly.

Thanks in advance for your assistance.

This time there was no response from GlobalSign. When I still didn’t receive a reply during the next day, I sent them another reply:

Hello Kenneth,

I am not sure if you received my last reply. I was hoping if it was possible to reissue the certificates that I purchased through one of your partners for free given the special circumstances (OpenSSL exploit). I understand that it is my duty under the AlphaSSL subscriber agreement to undertake all reasonable measures to keep the private key secure and that in the event of a possible compromise (the OpenSSL bug) I should report the certificates to you.

It’s my duty to request my certs to be revoked

Indeed, I tried to be a good client and follow the advice stated specifically in the AlphaSSL Subscriber Agreement (PDF):

Protection of Private Key
The subscriber or a subcontractor (e.g. hosting provider) undertakes to take all reasonable measures necessary to maintain control of, keep confidential, and properly protect at all times the private key that corresponds to the public key to be included in the requested AlphaSSL CA AlphaSSL certificate(s) (and any associated access information or device – e.g., password or token).

[…]

Reporting and Revocation
The Subscriber undertakes to promptly cease using a AlphaSSL CA AlphaSSL certificate and its associated private key, and promptly request AlphaSSL CA to revoke the AlphaSSL CA AlphaSSL certificate, in the event that:

• There has been loss, theft, modification, unauthorised disclosure, or other compromise of the private key of the certificate’s subject
• […]

Suddenly no reissue fee, but…

While I was still waiting for a response from GlobalSign, I discovered that my cert reseller updated his website and added a dedicated section to the heartbleed bug with specific instruction how to reissue potentially compromised certificates (send an email with a new CSR, a copy of the completed order email from GlobalSign, and the original reseller order number).

This time there was no mentioning of a manual processing fee. Yee-haw!

After sending the required information I quickly received the newly keyed certificates. Kudos to the reseller for showing extra patience (I screwed up first with the common name in the CSR due to my unforgivable incompetence).

… what about revocation? It ain’t happen.

But that wasn’t all, was it? I received the certs for my new keys, yet the old certs were still considered valid according to GlobalSign’s OCSP responder:

$ openssl ocsp -CApath /etc/ssl/certs/ -issuer AlphaSSL.crt -nonce -CAfile AlphaSSL.crt -url "http://ocsp2.globalsign.com/gsalphag2" -cert myoldcert.crt -header "HOST" "ocsp2.globalsign.com" -resp_text

The response:

OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: 71F686DDE8225C85875486C2C0977EC7729B6A25
    Produced At: Apr 12 09:27:23 2014 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: 80514679462905B325D529520E73F545385E19B6
      Issuer Key Hash: 14EA1955F00E0D32C61F7433B78E661A4C12311E
      Serial Number: 11216B3837CECA9A810B71221FBC8CBB4C2A
    Cert Status: good
    This Update: Apr 12 09:27:23 2014 GMT
[...]

Why is that a problem? As long as the old certs are considered valid, anyone in the possession of the keys could perform a man-in-the-middle attack by presenting the certs to unsuspecting users. (To be honest, the whole revocation mechanism is kinda screwed up since current browsers usually don’t check for revoked certificates, at least no those that are only domain-validated.)

I sent one last e-mail to my cert seller asking for clarification and received a quick response:

Hej Alex

No the old certificates will not be revoked.

O-K. I still considered myself lucky that the manual processing fee got waived – so I didn’t ask for the reason why the certificates wouldn’t be revoked.

Still… under the subscriber agreement, certs ought to be revoked in the event of potential disclosure of the private keys, and according to the GlobalSign Reseller Guide (PDF), it would have been hardly more than a click.

To revoke an SSL Certificate, log into your Partner account, locate the order via the Order History, click Edit, and click Revoke.

Update 04/14: It happened after all.

Thanks to the support from GlobalSign, who contacted me after reading this post, I was able to revoke the certs after all.

$ openssl ocsp -CApath /etc/ssl/certs/ -issuer AlphaSSL.crt -nonce -CAfile AlphaSSL.crt -url "http://ocsp2.globalsign.com/gsalphag2" -cert myoldcert.crt -header "HOST" "ocsp2.globalsign.com" -resp_text

The response now:

OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: 0A3577F2EE7CA9724ABBB12A1E77BD401C1B27BE
    Produced At: Apr 14 19:28:19 2014 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: 80514679462905B325D529520E73F545385E19B6
      Issuer Key Hash: 14EA1955F00E0D32C61F7433B78E661A4C12311E
      Serial Number: 11216B3837CECA9A810B71221FBC8CBB4C2A
    Cert Status: revoked
    Revocation Time: Apr 14 19:27:46 2014 GMT
    Revocation Reason: unspecified (0x0)
    This Update: Apr 14 19:28:19 2014 GMT
[...]

Lessons learned

1. You get what you pay for. Certificates are overpriced, in particular certificates that only involve automated verification (such as domain-verified). I paid considerably less due to a promo offered by a AlphaSSL partner/reseller. As a consequence, I couldn’t rely on direct support from GlobalSign nor could I use their control panel that would have allowed me to perform the rekeying and revoking process myself (Update: A GlobalSign representative contacted me after this post and offered to revoke the certs – thanks!)
2. Cert revocation is borked. Current browsers usually don’t bother with revocation checking (at least for simple domain-verified certs that is), so even if you successfully revoke a compromised cert, a hacker using it in a MITM attack is likely to succeed. Add to it the difficulty in having the certificate authority actually revoke the cert makes the whole thing ludicrous.
3. Long cert life cycles are bad. Based on the assumption that the cert revocation system is borked, a potential theft is even worse for those whose certificates have a long life cycle. The US gov suggests (PDF) that a cert’s lifetime mustn’t exceed three years.
4. Wildcard certs are bad (if you don’t need them). It takes only one server to be compromised to make all other services on the same domain and subdomain vulnerable. The problem is that they all share the same private key. There are some wildcard cert providers who offer unique key pairs for each server/service using the same wildcard domain. But those aren’t bargain certs.