You're reading the Keybase blog.
There are more posts.

July 19, 2019

Slack Security Incident for Keybase CEO

It was a cold Saturday New York City morning in January 2019. I was packing the car for a family ski vacation when I received a truly horrifying email:

A scary email to receive

That's interesting, I was just in the middle of loading up the Prius. Certainly, I wasn't using Slack from....

GEO IP
...Burnville USA

Oh.

My immediate thoughts, in order:

  • Thankfully we don't put sensitive communications (from financials to hiring to shit-talkin') into Slack. We basically just use a #breaking channel in there in case we have Keybase downtime. Phew. I didn't have to worry about being extorted or embarrassed. And Keybase as a company would almost certainly emerge unscathed.
  • WAIT A SEC. How did this happen? I use strong, secure, distinct, random passwords for all services I log into. Either Slack itself was compromised, my password manager was compromised, or my computers were "rooted" by an attacker.
  • Our weekend was hosed.

At risk of getting the car towed, I dashed an email off to Slack's security team, and after a few back-and-forths, received the standard fare. They did not inform me of the directly related 2015 Security Incident but instead implied that I was messy with my security practices and was to blame.

Though I was more than 90% convinced that Slack had been compromised, as the CEO of a security-focused company, I couldn't take any risks. I had to assume the worst, that my computers were compromised.

In the subsequent days and weeks, I reset all of my passwords, threw away all my computers, bought new computers, factory-reset my phone, rotated all of my Keybase devices (i.e., rotated my "keys"), and reestablished everything from the ground up. It cost Keybase and me a lot of time, money and stress. In the end, I was pretty sure but not 100% convinced that if I had been "rooted", that the attackers couldn't follow me to my new setup. But with these things, you can never know for sure. It's a really scary thing to go through.

And then...

I got the email today that countless other people got. Apparently my account might have been compromised in a previous attack. This corroborates my suspicions - I was never rooted and didn't need to deprovision all that hardware and keys. Still, what's done is done. What's dead can never die. Nonetheless, as the incident comes into focus, things seem worse than previously thought:

The attackers also inserted code that allowed them to capture plaintext passwords as they were entered by users at the time.

Whoa!

Though Slack originally told me 2FA would provide "a bit of extra security," these new data show otherwise. If the attackers inject server code, 2FA or U2F or any Web-based security practice does little.

Also, Slack's announcement seems to say 1% of accounts were still compromised (after 4 years), but we are wondering: how many were compromised then? And what percentage of messages did the compromised accounts have access to? 10%? 50%? Only the hackers know, but it's likely much more than 1%.

And finally, we know the original compromise was in 2015, but I was only notified of a suspicious login in 2019. Were our Dutch friends sifting through our messages for four years before Slack notified us of a suspicious login?

In the end, the damage was limited. More or less, Keybase and I were out:

  • $5000 worth of hardware
  • 60 hours of labor
  • 25 hours of lost sleep
  • 10 additional hours of team effort
  • A first weekend on skis for a potential 2034 Slalom Gold Medalist

What would have been way worse — immeasurably worse — is if our team had used Slack for anything other than what we did use it for, which was discussing outages of our own product. Had my cofounder and I discussed our company's cap table, or business partnerships, or compensation agreements, or ongoing legal matters over Slack; or had our team traded API keys, or security-sensitive matters; or had we controlled mission-critical infrastructure via Slack-powered "bots"; we'd be sweating bullets to this day that our important company secrets were out in the open, about to resurface at the worst possible time.

By contrast, Keybase currently runs all of its mission critical chat applications over Keybase itself. Our people-to-people conversations and our bots. And as always, Keybase messages are end-to-end encrypted, and only our users control their decryption keys. A break-in our of our servers, even one injecting code, cannot yield unencrypted messages or jeopardize message integrity.

Update: for those asking, there is a pretty simple Slack-to-Keybase team importer.

Cheers!

💖 Max Krohn, Keybase CEO
https://keybase.io/max
max@keybase.io



This is a post on the Keybase blog.