Jorge Manrubia

December 22, 2021

A system to encrypt data in bulk for Rails

encryption-splash.png


We just released mass_encryption, a library to encrypt large sets of data using Active Record Encryption. If you have been following along, Rails 7 was recently released and, among many goodies, it supports at-work encryption with Active Record. This new gem addresses a problem many apps will face: perform the initial encryption in existing applications.

Active Record Encryption was an extraction from HEY, and so is this library. We created the first version to encrypt the data entered during the private beta stage, and we eventually extracted it to a private gem that we used to re-encrypt data in HEY several times.

At some point this year, we decided to encrypt sensitive personal data in Basecamp too. There was an objective reason: contents like chats, messages, comments, or todos are sensitive enough to raise the bar. And a subjective one: once you get used to working with encrypted personal data, not doing it feels off.

Now, encrypting data at the scale of Basecamp represented a challenge on its own. We had to encrypt 2 billion records and revamp the version we used for HEY to support this load. We added the "track encryption mode" as part of these improvements, and we used upsert_all, which we extended, to perform updates in bulk.

So after Active Record encryption, console1984 and audits1984, mass_encryption was the last part of the puzzle, and I'm thrilled it's open-source now.

Finally, on a personal note, kudos to Basecamp for the big bet on all this technology and for releasing it so that everyone can use it. No customer was demanding this (try to explain at-work encryption to a non tech-savvy person), and there was no legal or operational need either. The motivation since the first moment was doing the right thing, which makes seeing how everything has crystalized even more special.