Avinash Joshi

October 14, 2024

Kamal 2: My upgrade journey

image.png

A quick intro to Kamal: it offers zero-downtime deploys with super simple config. The best part? Kamal works with any containerized web app. It’s convention over configuration - much like Rails! After using various deploy pipelines, Kamal's simplicity wins for most cases, including mine.

When first deploying my Rails app in production, I simply went with Kamal 1.0 - I was blown away by how simple it was to get it up and running. I had been using it for my deployments and it worked like a charm, without any hiccup. I was excited for new enhancements and Kamal 2 was announced at Rails World 2024. Here's what's new:

My upgrade journey

I accidentally eneded up going down the upgrade path when my CI decided to install Kamal 2 (since it wasn't locked to v1.x.x with `gem install kamal`). I took the opportunity to try the upgrade, but hit a brick wall after being get the end-to-end SSL/TLS working with Cloudflare. Before I get into it, here's

A quick detour with my "cloud" setup

I use AWS for my hosting, mainly due to startup credits. But you can think of my setup as a single server instance deploy with a dedicated database server. I run the whole rails app in that one server for now, including Solid Queue and Solid Cable (I recently got rid of redis and will make a blog post on that soon - lots of $$$ savings).

I have two cloud environments - (1) staging (2) production. Below is a diagram of production architecture, and I have the exact same setup for my staging. For deploys, I have a Github action, and simply do `kamal deploy -d staging` or `kamal deploy -d production` based on the environment being deployed to.

Upgrading to Kamal 2

Here were a few things that were important to me
  1. HTTP Auth on my staging environment
  2. End-to-end SSL even with Cloudflare
  3. Being able to deploy from Github and my local machine (i.e., credentials in both GH Secrets & 1Password)

1. HTTP Auth
Kamal 1.0, used traefik, so I simply setup HTTP Auth using a label - see the config/deploy.yml config gist here. However, I realized that since Kamal 2 comes with kamal-proxy, this wasn't an option. I was not sure what to do here. In an effort to find a solution, I ended up tweeting DHH where he reminded that it's built into Rails! So my first task was to move HTTP Auth into Rails. Here's a quick look at the diff of that commit:
diff --git a/app/controllers/application_controller.rb b/app/controllers/application_controller.rb
index e463cb9..9071c35 100644
--- a/app/controllers/application_controller.rb
+++ b/app/controllers/application_controller.rb
@@ -1,2 +1,3 @@
 class ApplicationController < ActionController::Base
+  before_action :http_authenticate
   include Authentication, Authorization
@@ -14,2 +15,11 @@ class ApplicationController < ActionController::Base
 
+  def http_authenticate
+    return unless Rails.env.staging?
+
+    authenticate_or_request_with_http_digest do |username|
+      valid_credentials = Rails.application.credentials.http_auth_credentials || {}
+      valid_credentials[username]
+    end
+  end
+
   def set_time_zone(&block)

diff --git a/config/deploy.staging.yml b/config/deploy.staging.yml
index cfd907d..fe43ad6 100644
--- a/config/deploy.staging.yml
+++ b/config/deploy.staging.yml
@@ -11,4 +11,2 @@ servers:
       traefik.http.routers.websecure.tls.certresolver: letsencrypt
-      traefik.http.routers.websecure.middlewares: auth
-      traefik.http.middlewares.auth.basicauth.users: user1:xxxx,user2:xxxxxx

Deployed with Kamal 1.0 and tested everything worked fine!

2. Ready to upgrade 💪🏼
Once the HTTP Auth for staging was figured out, I was ready to upgrade. The actual "upgrade" was as simple as following steps from Kamal's docs. Just a heads up - before upgrading in production, you must try it out in a different env, say staging! Here's the high level steps I followed:
  1. Upgrade to Kamal gem to v1.9.0 and ensure you can deploy with your existing Kamal v1 config. This has an option to "downgrade" incase something goes bad
  2. Upgrade to Kamal gem v2
  3. Update config/deploy*.yml
    1. Update builders config
    2. Delete traefik configuration
    3. Add "proxy" config
      1. Use port 3000 for proxy, unless you plan to use thruster
      2. set forward_headers to true
  4. In config/production.rb, set config.assume_ssl = true - this was the one that got me and gave errors like "connection refused"
  5. Move from .env to .kamal/secrets - I loved this as it gave me the opportunity to explore something. More about that below.
  6. RUN `kamal upgrade -d staging` and things should go smoothly!

3. Deploy from Github Actions (with GH secrets) and my local machine (with 1Password)
Previously I only had configured deploy to happen from my Github Actions - it was click of a button and worked for me. However, I've always wanted to have the flexibility of deploying from my laptop with credentials stored in 1Password. So with the introduction of Kamal Secrets in v2, I spent some time on what I think is an innovative solution to support both. Here is my staging secret file, which can safely be commited to the git repository.
# file: .kamal/secrets.staging

# If SECRETS_FROM_ENV is set (for e.g., in Github Actions where it's set to true), we'll use the secrets from the environment, else we will load it from 1Password.
SECRETS=$(if [ -n "$SECRETS_FROM_ENV" ]; then echo ""; else kamal secrets fetch --adapter 1password --account cravd --from DEVOPS_STAGING/CREDENTIALS REGISTRY_PASSWORD DATABASE_HOST DATABASE_PASSWORD; fi)

# Secrets from RAILS_MASTER_KEY or the credentials file
RAILS_MASTER_KEY=$(if [ -n "$RAILS_MASTER_KEY" ]; then echo "$RAILS_MASTER_KEY"; else cat config/credentials/staging.key; fi)

# Secrets from either ENV of the above extracted 1Password "SECRETS"
KAMAL_REGISTRY_PASSWORD=$(if [ -n "$SECRETS" ]; then kamal secrets extract REGISTRY_PASSWORD $SECRETS; else echo "$KAMAL_REGISTRY_PASSWORD"; fi)
DATABASE_HOST=$(if [ -n "$SECRETS" ]; then kamal secrets extract DATABASE_HOST $SECRETS; else echo "$DATABASE_HOST"; fi)
DATABASE_PASSWORD=$(if [ -n "$SECRETS" ]; then kamal secrets extract DATABASE_PASSWORD $SECRETS; else echo "$DATABASE_PASSWORD"; fi)

We are LIVE 🎉

It took me a couple of tries and debugging, especially the config.assume_ssl got me good! I even did a `kamal deploy -d staging` at some point. All in all, we are live and kicking with deployments via Kamal v2 - I call the upgrade a HUGE SUCCESS!

Thanks for reading along - would love to connect with fellow Rails enthusiasts, so hit me up!!!

References

About Avinash Joshi

Co-Founder & CTO of Cravd
Email: first name at hey
Social: 𝕏┇LinkedIn┇Website