David Heinemeier Hansson

April 6, 2023

The hardware we need for our cloud exit has arrived

It's been a long time since I last saw a physical piece of hardware used to run our services at 37signals. I vaguely remember doing a tour of our Chicago data center over a decade ago, but somewhere along the line, I just lost interest in the iron itself. Now the interest is back, because hardware is fun again, so let me share my excitement with you!

server-pallets.jpg


These are the two pallets that showed up in our Chicago data center recently. The same day that an identical set arrived in Ashburn, Virginia for our second data center. In total, we received twenty R7625 Dell servers that'll power the bulk of our cloud exit. It's a staggering amount of computing power in a shockingly small footprint.

Here's a diagram of our four cabinets in Chicago (we have another four in Ashburn). As you can tell, there's still a bunch of older hardware dedicated to Basecamp in particular. A good chunk of that will actually get retired, once we're done setting things up. But all the 2U servers marked "kvm" at the bottom of the cabinets are the new ones:

cabinets.png


You can spot the new R7625s at the bottom of the actual racks here, next to the older gear:


Each of these R7625s contain two AMD EPYC 9454 CPUs running at 2.75GHz with 48 cores / 96 threads. That means we're adding almost 4,000 vCPUs to our on-premise fleet! And a ridiculous 7,680 GB of RAM! And 384TB of Gen 4 NVMe storage! Serious horsepower and headroom for years to come. In addition to this, we have another ~six database servers showing up between now and this summer, and then we'll be set.

The contrast to the origin of Basecamp is funny. We launched Basecamp on a single-core Celeron server with just 256MB of RAM back in 2004. Spinning rust at 7,200 RPM. And that was good enough to get the business from part-time to full-time in about a year.

Almost twenty years later, we now have a long lineage of legacy applications (because we promise to keep applications customers depend on running until the end of the internet!), some massive flagship services in Basecamp and HEY, and a mission to get it all running on hardware we own ourselves again.

It's kinda wild to think that it's been less than three months since we decided to scrap Kubernetes and pursue a simpler solution for the cloud exit with Kamal. And that we've already moved half of the cloud applications that need to come home!

Over the next month or so, we plan to bring home both Basecamp Classic (still a multi-million dollar business, even if it hasn't been updated in about 13 years – that SaaS magic!), as well as the grand prize of the cloud exit: HEY! That'll leave us with just Highrise and a small auxiliary service called Portfolio left in the cloud as we start the month of May.

I thought we were already being optimistic when we planned a total cloud exit by the end of summer, but now it seems we'll basically be done by the end of spring instead. Truly a remarkable achievement by the team working on this effort.

The reality of our accelerated timeline has made me even more bullish on cloud exits in general. I imagined getting out of the cloud was going to be as hard as getting in. But that just hasn't proven to be the case. Though perhaps it's helped that we've had that nuts number of $38,000/week in cloud spend as a motivating carrot to get it done quickly!

I seriously hope that other SaaS entrepreneurs looking at their daunting cloud bills are paying attention. Once you've gone cloud, it might seem impossible to contemplate getting out again, but don't believe that for a second.

Modern server hardware is incredible. We've taken huge leaps forward in performance, density, and cost over the last few years. If you haven't run the numbers since cloud became the default in the past decade, you really ought to do so now. The numbers might just shock you as much as they did us.

So the end is now clearly in sight. We've solved all the key technical challenges we needed to address to make the cloud exit happen. We've been running production apps on Kamal for a while now. The path is clear, and I can't wait for those mammoth cloud bills evaporate. I think we're going to find that the napkin math I did for our public calculation of savings will be highly conservative. But we'll see and we'll share.

About David Heinemeier Hansson

Made Basecamp and HEY for the underdogs as co-owner and CTO of 37signals. Created Ruby on Rails. Wrote REWORK, It Doesn't Have to Be Crazy at Work, and REMOTE. Won at Le Mans as a racing driver. Fought the big tech monopolies as an antitrust advocate. Invested in Danish startups.