This story starts in October of 2019. I was in the Basecamp meetup in Chicago two weeks after joining the company. During an internal presentation, David talked about the need to raise the bar when it came to privacy for the new product the company was working on, codenamed Haystack, now HEY.

We all were busy with all kinds of projects preparing for HEY launch, and I got in charge of the data privacy one. Details at this point were fuzzy, but we knew we wanted some database encryption technology and an auditable Rails console that protected access to sensitive data. 

Before we started working on the technology, Jane and I spent a few weeks doing a thorough analysis of all the personal data we were storing in HEY, categorizing it, and analyzing the systems involved and the protection mechanisms in place.

Regarding database encryption itself, I did a deep dive into the existing solutions in the Rails ecosystem. I learned good things from them all, but I couldn’t find one that matched our vision in terms of features and design. Ultimately, this was going to be a critical piece of HEY, so we decided to develop in-house without finding a perfect match.

We created the first version of the library between February and March of 2020. We didn’t integrate it in HEY, but it served us to learn, discuss and figure out what we wanted. Among other things, we were initially storing keys in the database, and that felt like an unnecessary liability. After some good discussions with Jeremy, I decided to drop that version and start over in May. It was now clear what we wanted, so I could move much faster than the first time. The new version resulted in way less code working much better. 

I pushed the first commit for this new version on the 5th of May. On the 3rd of June, we integrated the new library into HEY and encrypted the database. HEY was in beta and two weeks away from launching officially. We also created the basic console extension to protect and audit console accesses in May (another private gem called console1984). 

On the 10th of June, a security firm audited the library and reported a severe flaw in its deterministic encryption approach. We fixed the problem and re-encrypted all the affected records. This was 5 days before the official launch.

Moving forward to March of 2021, we wanted to get this technology into Rails. While I was writing the pull request, I noticed a little problem. I was explaining how our usage of Marshal when serializing an encrypted payload was safe. Marshal was the fastest and most compact option and, while I was well aware of potential RCE security problems, I thought the only attack vector required gaining direct access to the database. In that exact moment, I realized that the scenario where the initial data wasn't encrypted made these attacks trivial.

So I rewrote the serializer with JSON and, finally, created the Pull Request to add encryption to Active Record. We had now a problem in HEY. We had to make a substantial change to our encryption scheme to use the upstreamed version due to the serialization change. Fully using the upstreamed version was a delicate project, so we sliced it into several deploys, aiming to minimize risks.

We completed the switch to the new scheme this Monday, and started the process of re-encrypting the whole database, which is still ongoing on some huge tables. HEY is now using the vanilla version in Rails 7 everyone else can use. We found some issues in this last deploy that resulted in these fixes in Rails.

Deploying this technology on a live system and having to change encryption schemes several times had a tremendous influence in the design of the library. In terms of difficulty, encrypting new data is trivial. Adding encryption to a live system is not, and changing existing encryption schemes in a live system is even harder. The library supported all these operations while keeping database queries working seamlessly and with thousands of customers using the app. To my knowledge, no other app-level encryption library in Rails or in any other framework supports this.

As an example, this is what we do in HEY to support the old encryption scheme for the transition period while we re-encrypt everything:

config.active_record.encryption.previous = [ { 
  encryptor: PreviousEncryptionScheme::OldEncryptor.new,
  message_serializer: PreviousEncryptionScheme::OldMessageMarshal.new,
  cipher: PreviousEncryptionScheme::OldCipher.new 
} ]

I can tell you that shipping a new encryption technology is nerve-wracking. You can lose data, make queries fail,  allow duplications that should not happen, and, essentially, break any part of the system. While we never ran into big problems in HEY, we haven’t been free of encryption-related incidents either. But because it was challenging and because I had time to think, explore, write and rewrite, this has also been one of the most satisfying projects I’ve ever worked at.

And if you want to incorporate encryption to your Rails 7 app, check the guide!