Nathan Dumlao 263787

Post Mortem: A busy week for our Outlook team

Personally, we are huge fans of in-depth blog articles of service failures, because you can really learn something! So we decided to use last weeks problems with our Jira add-in to do the same: Provide you, the interested reader (and/or customer) with a short overview on what happened. Turns out: Not only SAAS services can fail pretty badly but client software as well. We are very sorry if you were affected by these issues, but we’ll double down to make sure this does not happen again!

If you were unlucky, your week may have started with one of the following problems. Luckily, none of these happened for everyone, but it was happening more often than we liked.

Let’s go trough the errors one by one – the first two are actually related so we’ll group them together.

Code signing certificate woes

The first issue which occurred was actually responsible for the first two errors. On Windows, to ensure that the software you run is actually trusted and published by us, all executable files & libraries are signed with our code signing certificate. That allows Windows and Outlook to validate the files and knows it comes from a trusted vendor.

This certificate needs to be renewed once in a while (every two years). This is usually not a critical issue, since we just can order a new certificate and start signing a new version with the new certificate. The old files, signed with the old certificates should just continue to work, since we use something called “trusted timestamping“. Basically, it works like this:

  1. When signing the files with the certificate, the time when the files were signed is added as well
  2. When Windows or Outlook validates the code signing, it uses the stored time to validate the certificate. This means, that even the certificate expired already, it was valid at the time the code was signed, to the validation passes.

In theory, that sounds great, because there is no rush for us to exchange the certificates, since we would just switch this with the next release. In practice, this failed spectacularly 🙂

See, we used code signing certificates from a certain vendor called “StartCom“. Code signing certificates are usually quite expensive, and this was the go-to partner, when you wanted a certificate fast and inexpensive. Unfortunately, this company was sold to a Chinese vendor in 2016, which started a series of unfortunate events – you can read up upon that the Wikipedia article. Long story short: Windows deprecated certificates by this vendor starting September 2017, which somewhat messed up the logic with the timestamping we relied upon.

Suddenly, the planned certificate switch was of much higher priority, because certificate validation started failing for the installer files and the Outlook add-in (see the first two screenshots). We instantly ordered a new EV code signing certificate from a trusted vendor (GlobalSign), which unfortunately took some time to get, since you get the private key mailed on an USB stick:

As you can see, the new certificate finally arrived in our office and started doing it’s job! The new release we are currently rolling out should fix the issues. In case you see the second screen above, unfortunately a reinstall may be necessary. You can find a link to the installer file in your Jira admin panel.

Deprecation of Outlook folder homepage

The second issue was related to a security update, as part of the Outlook October patches (more specifically KB4011164 and others). We are all for security, but breaking many add-ins like ours (or BCM & Dynamics) seems a bit overkill instead of favoring a phased deprecation. As there was no information available on this “fix” prior to the release, we were unable to provide an update before the October update started rolling out.

A little background: We user a folder homepage in Outlook to display our Jira feed. We will be working on another solution to move away from the folder homepages in the future, but this will be part of a larger effort and won’t happen very soon. As a workaround, we re-enabled folder homepages through the official method with our 2.5 update, which is already rolling out. This should hopefully solve the issue until we have a long-term solution ready!

Conclusion

All in all we had quite a few busy weeks, keeping our Outlook add-in up and running – sorry if you were affected by the issue. In case something still does not work correctly, please get back to us via the normal support channels!

The whole affair reconfirms our efforts to move forward with our new Office 365 add-in, which is quickly catching up to our classic, Windows-only add-in in terms of functionality. We hope this will be a worthy, more stable and versatile replacement in the future, check it out if you haven’t already: Office Store.