TIL: Kamal Deployments, Data Migrations, and Lessons Learned

With a new Ruby on Rails version comes new gadgets, tools, and learnings. I've been building out a new Rails 8 app — moving fast, trying to ship ideas quickly — and one of the first things I had to wrangle was Kamal for deployments. It's been a while since I've gone through the motions of deploying apps from scratch, and Kamal already being baked into a freshly generated Rails 8 app felt like the obvious path forward. What followed was a solid few weeks of head-scratching, restarting things that probably shouldn't need restarting, and one AI-assisted detour that sent me in completely the wrong direction. Here's what I learned.

Run kamal setup Before Your First Deploy

My first issue was that the DigitalOcean remote server and container registry I was deploying to didn't have Docker or the necessary packages installed. Instead of doing the sensible thing, I SSH'd into the server and started manually setting up services with Ubuntu snap packages. Predictably, I ran into vague, thoroughly unhelpful errors when I tried to deploy. Turns out the fix was embarrassingly simple — this single command handles dependency setup correctly and would have saved me the entire headache:

kamal setup

Read the docs first. Or at least skim them.

When in Doubt, Restart

This one still bugs me a little, because it's not a real answer — but here we are.

Within the first couple of weeks of using Kamal, I've encountered SSH connection errors and hanging servers more times than I'd like to admit — both through the Kamal deploy process and when trying to connect directly via the command line. The deploy script hangs at various stages, servers become unresponsive, and the more I dug into the debugging, the fewer answers I got. Coming from fintech environments where flakiness like this would be a five-alarm fire, it's a little uncomfortable to just shrug and move on — but right now my goal is to ship and test ideas quickly, not chase down gremlins. Restarting my local machine and the remote server has been the fastest fix, and I've made peace with that for now. I'll circle back when it becomes a real blocker.

If you run into Kamal's deploy lock getting stuck, you can at least clear that cleanly:

kamal lock release

Data Migration Post-Deploy Hook

As the app design keeps evolving, I've needed more robust data migration support beyond standard schema migrations. The data_migrate gem handles this well with minimal setup.

Here's where things went sideways: I leaned on AI to help wire this up, and it confidently told me to add hooks directly into deploy.yml. Vibe coding failed me here. That advice produced a thoroughly unhelpful error:

kamal deploy
  Finished all in 0.0 seconds
  ERROR (Kamal::ConfigurationError): unknown key: hooks

After digging in myself, I found that Kamal (v2.7) handles deploy hooks via shell scripts located in the .kamal/hooks/ directory — with samples already sitting there waiting to be used. Since data_migrate was being introduced in this particular deploy, I added the migration command to the post-deploy hook:

#!/bin/sh
# .kamal/hooks/post-deploy
# This runs on your primary host only (handled by Kamal):
kamal app exec --primary -i "./bin/rails db:migrate:with_data"

Then make it executable:

chmod +x .kamal/hooks/post-deploy

After that, the deploy ran cleanly end-to-end:

Releasing the deploy lock...
  Finished all in 39.3 seconds
Running the post-deploy hook...
  INFO [2a7a7fe1] Running /usr/bin/env .kamal/hooks/post-deploy as snobook@localhost
  INFO [2a7a7fe1] Finished in 8.904 seconds with exit status 0 (successful).

Huzzah! Happy deploying and migrating 🤘