Stanko Krtalic Rusendic

A year at 37signals

2026-08-05T07:26:18Z

A year ago I had my first day at 37signals (the people behind Basecamp, HEY and Fizzy).

This was a dream come true for me.

I wanted to work there ever since I read Rework over a decade ago. The books, podcasts and talks that followed showed me that we also share a similar philosophy towards work and the craft of coding. That just cemented my resolve.

Not to mention that it's the birthplace of Rails! The single piece of software that made me decide to turn my programming hobby into my career.

I had an incredible year, so I want to share how I got here and what I learned that might not be obvious when looking from the outside-in.

Getting the job

I'm not sure how many times I applied for a job here over the years. It must be at least six, maybe more.

Out of those I passed to the final round twice.

The first round is notoriously difficult, which is quite funny because it's just a simple cover letter.

We write a lot internally. Nearly all communication is written and most of it is long-form comments and notes. Being able to write is essential.

I wasn't a good writer. It took me a few attempts to realize that. But I was determined to get better at it, so I worked on it.

The first step to writing better is to read more. Not just technical books, self-help and similar, but stories, essays, opinion pieces. Learn what's out there and expand your vocabulary and literary horizon, then forge your own style from the things you like.

A funny side-effect of reading more is that I started to hate corporate newspeak - it's so bland and dull that I immediately tune out as soon as I start to read or listen to it now. To me it's like communicating in memes.

The second step is to write more. I started writing more often on this blog, which helped with long-form technical writing. But later on I started keeping a diary. Nothing fancy, I'd just jot down what happened every day and some thoughts bouncing around in my head - this helped me in general, not just with writing.

The third step is to learn how to write a cover letter. This took several attempts and it boils down to: be yourself.

I always felt a pressure to make the cover letter sound professional. You know, the things they tell you in school: serious tone, respect, warm but also not too personal.

That's terrible advice. And I realized that only after I started hating corporate newspeak. It makes you sound the same as everyone else, and dull on top of that.

A cover letter should explain why you want to work there. It's your chance to tell your story. It's supposed to be personal, fun and interesting.

The second round is a take-home exercise.

I was never worried about this. For me Rails and the OSS that 37signals puts out has always been a treasure trove to explore and learn from so I was familiar with their style, and I already spent a lot of time internalizing why they structure code like they do.

I could write a whole essay about this, but to keep it short: ergonomics and domain design are important; the code should read like prose.

Lastly there is the interview.

I fumbled the first one. I was unbelievably anxious and over-thought and over-explained every question; the conversation didn't flow. But I got good feedback from it and spent a lot of time improving on it. The second time around I was relaxed and we had a fun chat.

Then, one evening I got an email ominously titled "Some news" telling me that they'd like me to join the company! I was over the moon!

Working here

I have a hazy memory of the first few weeks - they felt like a dream. What I remember the most is reading, and reading a lot.

As soon as I started working on actual tasks a new feeling hit me: doubt. I started to second-guess my own work and this thought formed in my mind that I just lucked out with the hiring process and that someone would soon notice that my work wasn't any good.

This feeling lingered on for months and passed only when I started getting fewer comments on my PRs.

It sounds like a curse, and it felt like it, but now I see that it was also a boon. It motivated me to get better and to learn. I figured that's the only thing that would cure that feeling, and it did.

Then came Fizzy's launch.

I got to work on Fizzy as soon as I joined. My first task was to add FizzyAsk - an LLM-powered chat where you could tell Fizzy what to do or ask it questions about your data.

After adding it I barely ever used it. And to my surprise it was completely scrapped alongside all AI features after a month.

This was the first time I ever saw features get cut - not pivoted, not moved, not put on hold - after they were built and before they were even released.

The reason behind the cut was eye-opening: delightful or nothing.

Imagine an overgrown garden with hundreds of different plants in it - that's what the products of most companies look like to me: an eclectic collection of ever-growing features. The garden is "designed" by adding whatever plants the neighbours added to their gardens this week. Everything is overgrown because nobody has the time to care for so many plants, and even if they do they don't know how. The plants that are cared for are overshadowed by the overgrowth.

By comparison, what I saw with Fizzy was gardening. There was a vision. The garden had just a few flowers, but all of them were curated to work together and arranged to be delightful. If something new didn't work with what was already there it wouldn't make the cut, or we'd remove some of the old flowers to make room for the new ones.

I talked to a few friends about this and some of them had an interesting comment: "Isn't that soul-crushing? You spend so much time building something just to see it go down the drain."

No, it isn't. In fact, it's liberating. Not all ideas are good, but you can't know that ahead of time. Sometimes you have to try them to understand why they don't work. It would be far more soul-draining to have to support an experiment for years than to admit that it wasn't very good and bury it right away.

Whenever I talk about this I think of Buddhist mandalas. These are intricate artworks made of colored sand that can take weeks to complete. Once they are done they are admired for a while and then swept up and thrown into a river. Nothing is permanent.

Later on I was tasked with building a new magic-link-based auth system. Sounds easy, right? It wasn't.

Fizzy was built on two new in-house gems: activerecord-tenanted and beamer. AR-tenanted separated each tenant into their own database, and Beamer replicated those SQLite databases to the whole fleet of servers. The catch was that there had to be a central identity database for sign-in, but each server was the writer for a few tenant databases, so you could never write to that database and then the tenanted database in the same request because the two databases would be on different servers most of the time.

As I was untangling that I realized that I was used to working with the latest release of Ruby and Rails, even with Rails edge. But this was different. We were ahead of Rails edge. We were building what might become Rails edge at some point and dogfooding it ourselves to work out the kinks!

This can sometimes become a challenge. The magic links took two weeks to implement and I spent a good part of those two weeks wondering if it could even be done.

Some of my work on Fizzy did make it into Rails. I added Crockford's Base32 to Fizzy for magic link codes and that became SecureRandom.base32. Later on I added passkeys to Fizzy, and that might also land in Rails as has_passkeys in Active Record.

After the launch we shifted into cycle work. This was the first time I experienced Shape Up.

Shape Up has two modes: cycle work and new-product mode. Cycle work is the thing most people associate with Shape Up - six week cycles with cooldown periods in between. In new product mode there's no fixed cycle, no appetite, no kick-off, no pitch. It's like jazz, things are made up as we go.

After working in both modes I realized that what I thought of as "agile development" was just waterfall with extra steps.

Take scrum as an example: two week sprints, task pointing, standups. It sounds quite similar to Shape Up: six week cycles, appetite, check-ins and hill-charts.

The magic is in what happens behind the scenes.

In every company I ever worked at there was something like a "product roadmap" - a document that laid out each feature that would be added in the fiscal year and in which order they would be worked on. Then scrum gets slapped on top of that. So now you are working in cycles, but there is this Gantt chart called the "roadmap" in the background that suspiciously looks like a waterfall and if anything slips the cascades become a bit wider. Scrum isn't there to get things done, it's there to keep you on schedule.

In Shape Up the plan is laid out six weeks at a time. It isn't set in stone, it can shift; projects are dropped, shortened or extended. The important part is that after the six weeks are over a new plan is laid out that takes into account not only what was done in the last six weeks but also what happened in the world.

Jason Fried had an excellent article about this where he compared it to running like a squirrel. A squirrel runs a little, then stops, looks around, picks a new direction, and then runs some more. As it does that it takes its surroundings into account. It goes in a direction but it regularly re-evaluates. That's agile.

I've seen so many serendipitous situations where one project became the groundwork for another that was just the right fit at the moment. You can't plan this, you have to see and feel the previous work to form a vision of what you could build with it next.

These are just the most eye-opening things I learned.

Last week was a big week for me because the company decided to keep me! After the first year everyone goes through a review that asks one question: knowing what the company knows now, would they hire you again? A huge weight has been lifted off my shoulders even though I knew the review would go well.

I'm grateful I get to work alongside and learn from my amazing colleagues at 37signals.

2026-02-14T14:10:40Z

2025-06-12T14:00:00Z

One of my favorite activities is hiking. I'm fortunate to live close to a mountain where I can go for a hike any time I want - often early in the morning, before work.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A winding road on the way to Medvednica

I love the trek over hills and valleys, through forests and streams, the creaking of trees in the wind, the rustling of leaves, the murmur of streams, and the discussions with friends along the way.

I love the destination - the view, the tranquility, the bean stew with sausage at a hiking lodge, cuddling up to a fireplace in the winter.

But most of all I love the mountain air.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Zagreb, as seen from Sljeme, covered with clouds

It smells different, it’s light, relaxing, refreshing, and clean.

The difference is most striking on the way down - at one point, the air starts to feel heavy, turns dusty, and takes on an odor.

One winter morning, as the trail gave way to pavement and the outskirts of the city, a sharp sulfur smell - almost like gunpowder - hit me and stayed with me all the way home.

Something wasn’t right. The air wasn’t usually that bad. I checked the AQI for Zagreb - it was the fifth worst in the world that day.

I immediately bought a cheap air purifier off IKEA and decided to make an air quality monitor.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Breadboard with the first prototype of my air quality monitor

I made a quick prototype using parts I bought off AliExpress. After ironing out a few kinks, I ended up with something functional - Air Quality Box.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The Air Quality Box

To turn the rat's nest of a prototype into something presentable, I designed a custom PCB that connected all the components and packed them as tightly as possible in a enclosure that I printed. It was as small as I could make it given the parts and air flow constraints needed for everything to work properly.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The tightly packed inside of the Air Quality Box

It integrated with Home Assistant and displayed measurements on its screen. This allowed me to automate my IKEA air purifier, and I could read the air quality right from my desk without any app.

That little boxed served me for four years and taught me a lot about air quality:

Higher CO2 levels in a room cause headaches and drowsiness
Air with very low particulares and 40% humidity feels like mountain air
VOCs and high humidity make the air feel heavy

But it had two major flaws.

First, its design was very industrial - people either loved it or hated it.

Second, the UI was clunky - you couldn’t just glance at it and see what’s wrong, you had to wait for all the measurements to cycle through.

At first, I started working on a V2 which was heavily inspired by IKEA’s Vindriktning. It had a single light bar that told you an overall score of the air quality, and a small screen at the top that showed what needed fixing.

Then I came across the Awair Element. It had everything I wanted - a beautiful enclosure, it shows everything at a glance, it can integrate with Home Assistant. So I just bought one, and I have been very happy with it for the past 6 months.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The Awair Element that I bought

Its default screen shows five bars and a number. The number is the score of the air quality based on all measurements. Each bar represents a measurement: temperature, humidity, CO2, VOC, and particulates. When a value is within the target range, the bar shows a single dot. If it’s too high or too low, it displays two to five dots, depending on how far it has drifted.

I just have one minor annoyance with this interface - you can't tell if a measurement is too low or too high at a glance.

Say the humidity bar shows three dots - is it too humid or too dry? To find out, I have to pull out my phone, open the app, and read the value. Or I can reach around the device, press a button a few times until it displays the humidity measurement, and then cycle back to the score display. It isn't terrible, just annoying.

But I had an idea for how to fix this. I could create an app that lives in the system tray and shows the current air quality score. When clicked, it expands to show all measurements. That way I can quickly check what's wrong - no phone, no fiddling with buttons.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Demo of the system tray app

This wouldn't be my first Linux Desktop app. Years ago, I made one using Qt, QML and Rust, so I knew what I had to do.

There would be two parts to the app - the app itself and the system tray widget. The app discovers air quality monitors on the network, gathers measurements, and handles settings. The system tray widget displays the gathered measurements. The two would communicate with each other via D-Bus.

Over the last few weeks, I've been sharpening my Go skills, so I decided to write this app in Go. And since I use GNOME as my Desktop environment these days, I went with GTK instead of Qt - it fits in better visually.

Which brings me to AI.

Until now, I've used AI as a pair programmer. I’m in the driver’s seat, and when I want to bounce ideas around or look something up, I ask the AI.

To me, that’s the most natural way to work with an AI. It allows me to stay in flow - I can focused on keeping the code clear, modeling the domain, and solving the real problem, while the AI allows me to tap into a vast pool of knowledge in a moment's notice. No more digging through SEO spam for that one useful link - I get exactly what I need, when I need it.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

ChatGPT and NVIM side by side

This approach helped me a lot while learning Go. Whenever I hit a wall, I could ask the AI how it would solve the problem and get a few new things to read up on. In a way, it felt like having a mentor.

This time, I decided to flip things around and let the AI drive while I guide it. I wanted to see if there was a better way to work with AI than what I'd been using. So I installed Claude Code and bought $50 worth of API credits.

(Since this week, if you have a Pro subscription to Claude, Claude Code is included - no API credits needed)

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The initial prompt given to Claude Code

And that's how I - or rather we - made GNOME Desktop Air Monitor.

Demo of GNOME Desktop Air Monitor

I didn't like this way of us working together - for me, the pair programming approach just works better. Letting the AI drive seemed to amplify both of our weaknesses.

First, the AI is very goal-oriented.

It only cares about the outcome - in many ways it behaves like a greedy algorithm, making choices that seem best in the moment, without regard for long-term consequences.

This isn't bad per se. Sometimes this leads to efficient, pragmatic solutions. But more often, it results in spaghetti code - one hack piled on top of another, just to reach the goal. It doesn't care how maintainable, extendable or readable the solution is - that isn't its goal.

To counter that, I had to be very explicit. I had to explain not just what I wanted it to do, but also how I wanted it to do it.

For example, I asked it to create an index screen that shows all devices, then a show screen for individual devices, and finally a settings screen.

The result? One giant app.go file. All three screens were in it. All their logic, all the app setup and window management, everything. There was an App struct that held the state of every button, list and graph that could ever appear, with cryptic names like "button", "settingsButton", "actionList", etc. There were three methods - "showIndex", "showSettings" and "showDevice" - that were nearly identical, each resets the state of every UI element before switching screens.

I had to explicitly ask it: "Do you think it would be better to create a struct for each screen and keep its state there? You could add a reset and a show method there too". It agreed - and refactored the code.

I tried to improve the naming in the same way. But it kept giving me worse and worse suggestions until I gave up and renamed everything manually.

By the end, my prompts had become so detailed that I felt like I was micro-managing someone. And that's just not how I want to work. For me, trust - in someone doing their job and doing it well - is essential. If that trust isn't there, then explaining exactly what needs to be done is often more work than just doing it myself.

I talked to a friend about this and he said something along the lines of "you have to treat it like a junior developer". But that's the problem - it isn't a junior developer. It doesn't learn from its mistakes. You can't explain to it in which situation one approach is better than another. It won't conclude that it produced garbage or that it's painting itself into a corner - as long as the goal was technically achieved.

Second, it doesn't understand the sunken cost fallacy.

The AI will almost never give up on a solution. It will almost always build on top of what already exists. It doesn't question, it doesn't suggest, it just marches on. In that regard, it's similar to a junior developer.

This gets worse and worse as you iterate on your project. At some point you have to step in and explain to it that the current approach just doesn't work, and that it should try something else.

I didn’t really see this as a serious issue until my girlfriend - who can't code - asked ChatGPT to help her build an iOS app. She described what she wanted, and it generated a React Native app that did exactly what she asked. Then she asked it to add live notifications - at the time, an iOS-only feature that required native code to get it working. Instead of switching tools, it piled on hack after hack to make it work. It ultimately failed.

Then I told it to recreate the app in Swift - and it nailed it on the first try.

If I didn't know what it had to do - and didn't tell it explicitly - we'd still be spinning around in circles, trying every hack known to man or machine.

Third, it doesn't generalize.

I often found myself in awe of both its brilliance and its stupidity in the span of five lines of code. It would implement a pattern that felt like the right solution - elegant, compresses the complexity just enough, clean - but only in the narrow context of the problem it was solving at the time. It never generalized the pattern or applied it across the rest of the code. And that’s what’s so frustrating - it has access to vast knowledge, the entire codebase, and limitless memory - you'd expect better.

But enough of the negatives. I also discovered some strengths.

It's brilliant at writing documentation from existing code.

Last week, I started writing a new gem and it crossed my mind to try letting Claude write the documentation for it. To my surprise, it produced well-worded, concise descriptions, complete with good examples and relevant links - even if a few of the links turned out to be dead.

It's decent at writing tests.

I also let it write a few tests, and they turned out surprisingly solid. It covered a lot of edge cases I might’ve missed on a first pass, and the structure was generally clean and readable. Occasionally, it repeated itself more than necessary - but as a starting point, it saved time.

It's excellent for scaffolding.

The most enjoyable way to work with an AI agent, for me, was to let it handle the first pass - generate the rough structure. Then I’d come in and rewrite the code the way I like it. That initial scaffold gave me something to react to, and it helped me move faster.

All in all, I learned that I love programming as much as I love solving problems.

Letting the AI drive helped me see that. It made the coding faster - but it also took the joy out of shaping the code, thinking through the domain, and making things beautiful. That’s the part I don’t want to give up.

Working with AI, as a partner, is what feels best to me. It can help me move faster, think broader, and stay in flow.

My last PHP app: How I Fell for Ruby

2025-05-24T09:00:00Z

Last week I found a backup of the very last PHP app I ever wrote. The backup was from 2009, but the app itself was written in 2007 - 18 years ago!

Back then I was in my late teens and the world was (or at least felt) completely different. Out of nostalgia, I spent some time exploring the source code and even got it to run (for the most part).

This reminded me of how different and quirky the web used to be, and how I discovered Ruby.

What was the app for?

My friends and I used to play Call of Duty 2 together after school - seems like the world hasn't changed that much after all - and we got good at it. So we organized into a team, called a clan, and started playing against other clans for fun.

Every clan we looked up to at the time had its own website where it put up its latest games, demos of those games (a demo is a file that allows you to replay a game and spectate), they had a forum, news, and so on. So we wanted to have our own website, but we were extremely tight on money.

The most that we could spend - all of us together - on a website was around 3€ per month or 35€ per year. To put that into context, 3€ could buy you 2 large pizzas at the time - we were financing this with pocket money.

All hosted website engines were way too expensive for us. We had just enough money to pay for PHP hosting to a local company. For the domain we got a free .su domain from a sketchy registrar. And now we had to make a website.

At the time I used to write small programs to solve my math homework or just to play around with my computer, and I just learned HTML, so I volunteered to make the website.

Looking back, implementing a website with news, a calendar, file uploads and a comment system was very ambitious for a first project. But I didn't know that at the time so I gave it a try and failed. There were too many moving parts which caused too many bugs, and on top of that the file storage logic that I wrote - because I didn't know that databases existed - kept corrupting files forcing me to recreate everything every now and then.

Somehow I came across webSPELL - a CMS for clan websites. That solved most of my problems. Now all I had to do was create a template, add and change a few things. All in all I made five websites using PHP and webSPELL - three different iterations of our website, and two websites for other clans.

The last one I made was the third iteration of our website in 2007. Here it is in most of its former glory.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The website, mostly working

It even had a visitor counter - this was the Google Analytics of the time.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The visitor counter that was in the bottom-right corner

This was written for PHP 4 and I'm running it in a Docker container with PHP 7 so some things - like the news, forum and stats tickers at the top - just broke during the update. I'm honestly amazed that I only had to do a handful of changes to get the app to boot with PHP 7.

Digging through the code made me remember how different the Web was back then.

For starters the app rendered XHTML because HTML 5 didn't exist yet and HTML 4 was difficult to work with.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">

The encoding was set to iso-8859-1 to get support for Croatian diacritics. UTF-8 did exist at the time but a lot of software just didn't support (Windows 98 was still very common) so the best thing I could do was to use iso-8859-1 as it would render the characters correctly for most people.

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

Then the wild part - this whole site is one giant HTML table with an iframe in it.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

DOM of the website

DIVs did exist, but they were much harder to position than table cells - flexbox didn't exist - so they were mostly used as DIViders. And the nice thing about tables is that you can set a fixed width to every cell using the width attribute - this was the only way you could do templates back then.

A template started out as big picture that was sliced into box-shaped sections like a jigsaw puzzle. Then you'd make an HTML table that puts the puzzle back together. You'd make each table cell the exact width and height of your slice and then set the slice image as the background image of that cell. If you had a layout that wouldn't fit into a single table you'd nest HTML tables to get the layout you want.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

An example of how slicing worked

Slicing was an art. A big part of the job was to create slices that could be tiled. This saved on bandwidth which, compared to today's 100+ Mbps speeds, were barely able to provide a meager 2 Mbps. This also meant that you'd often have to slice something in such a way that seams and tiling wasn't obvious.

E.g. the horizontal bars on the top and bottom would get turned into a 1px wide image that would tile horizontally. The vertical bars would be turned in a 1px high tiling image. This also allowed the template to scale to fit any size content and would often reduce a 1MB image down to 1kb or less if you used a GIF.

Another interesting thing - this site was exactly 1011px wide. Most monitors at the time were either 1024px or 1280px wide, so the site would fill most of the screen. Responsiveness just wasn't a thing - the iPhone had just come out and there was literally no way to detect the screen size.

This produces hilarious results on today's monitors. E.g. this is what the site looks like on my ultra-wide.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

What it looks like on a 40" ultra-wide

And on my iPhone 14

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

What it looks like on an iPhone 14

I can barely read the text, and I'll need a toothpick to click on a link.

Ruby, Ruby, Ruby, Ruby

Over the winter break of 2008, I stumbled across a tweet praising Ruby on Rails as an elegant alternative to PHP. The next day, my mom asked me if I could build a website for her real estate business - perfect timing.

I decided to give Rails a try. webSPELL wasn't a good fit for a real estate website, and I was ready for something new.

Something about Ruby just clicked. The code read like a story. Ruby's syntax felt natural and obvious in a way PHP never had. Suddenly programming wasn't just about hacking things together - it was expressive, joyful, and beautiful.

I remember comparing how I used to join first and last names in PHP:

implode(" ", [$first_name, $last_name])

to how it looked in Ruby:

[first_name, last_name].join(" ")

Ruby felt like I was talking to the data. I'd say "Hey Array, join your elements together with this separator". PHP, by contrast, felt like rummaging through a toolbox hoping the right function was in there somewhere.

Simple database queries became poetic:

$user=safe_query("SELECT * FROM ".PREFIX."users WHERE first_name = 'John' LIMIT 1");

vs.

user = User.find_by_first_name("John")

Rails didn’t just simplify the work - it made me fall in love with writing code. After that, there was no turning back. I decided to turn my programming hobby into my career.

I went to college to learn computer science, protocols, and telecommunications. Then my first job was at a digital agency writing Rails apps to help bootstrap businesses - many of which are still around. And I've been fortunate enough to work with Ruby and with Rails on every job I've had since.

That dusty old PHP app didn't age well, but I'm glad I found it.

It brought back memories of a time when building things felt like magic, which reminded me how I stumbled into Ruby. One tiny decision I made one winter ended up shaping my whole career.

I didn't know it then, but I was writing the first lines of a lifelong love story - one that carried me from Call of Duty matches and sliced-up tables to a profession built on joy, clarity, and code that reads like prose.

Building Twice: A clone of Once

2025-05-16T08:00:00Z

When Campfire - the first Once product - came out, a few of us at Ruby Zagreb pitched in to buy a copy.

Prometheus had brought fire from Mount Olympus, and we came to see the blaze. We were finally seeing how the people at 37signals - the birthplace of Rails - build Rails apps. And as a bonus, we could finally ditch our free Slack Channel!

After days of exploring the source code, the time came to set up Campfire. The purchase email included a shell command that looks like this

/bin/bash -c "$(curl -fsSL https://auth.once.com/install/1111-1111-1111-1111)

I ran that, and after solving a few issues, I had a Docker container running Campfire.

This is a super elegant way to distribute apps! But how did it work? I wasn’t sure so I decided to replicate it and see.

Figuring out how Once works

Just like last time, I started to unravel this ball of yarn by following a thread - the shell script.

Since all it does is execute a script it gets from the Internet I went to see what that script was

curl -fsSL https://auth.once.com/install/1111-1111-1111-1111
#!/bin/bash
set -e

echo "Installing the ONCE command..."
echo
sudo true

command_path="https://auth.once.com/install/1111-1111-1111-1111?arch=$(arch)&platform=$(uname)"
curl -s $command_path > .once-tmp
sudo mv .once-tmp /usr/local/bin/once
sudo chmod +x /usr/local/bin/once

if [[ "$OSTYPE" == "darwin"* ]]; then
  # Don't use sudo on macOS; Docker Desktop is likely to be running as the user
  /usr/local/bin/once setup 1111-1111-1111-1111
else
  sudo /usr/local/bin/once setup 1111-1111-1111-1111
fi

This downloads a binary for your OS and CPU architecture, installs it, and then runs it passing in your license key.

As this was a binary I couldn’t just open it up and see it’s code so I resorted to simple reverse engineering using the strings command - it returns all things that look like C strings from a given file.

... 
crypto/tls.masterSecretLabel 
crypto/tls.extendedMasterSecretLabel 
crypto/tls.keyExpansionLabel 
crypto/tls.clientFinishedLabel 
crypto/tls.serverFinishedLabel 
mime/multipart..inittask 
mime/multipart.ErrMessageTooLarge 
mime/multipart.multipartFiles 
mime/multipart.multipartMaxParts 
mime/multipart.emptyParams 
mime/multipart.multipartMaxHeaders 
mime/multipart.quoteEscaper 
net/textproto.errMessageTooLarge 
net/textproto.colon net/textproto.nl 
net/textproto.commonHeader 
net/textproto.commonHeaderOnce 
net..inittask 
net.rfc6724policyTable 
net.confOnce 
net.confVal 
net.netdns 
...

There were a lot of strings that looked like Go packages, functions and methods. I soon found a few references to cobra - a Go package for building CLI tools - so I was sure that I was looking at a compiled Go app.

... 
github.com/spf13/cobra.init 
github.com/spf13/cobra.map.init.0 
github.com/spf13/cobra.GetActiveHelpConfig 
github.com/spf13/cobra.activeHelpEnvVar 
github.com/spf13/cobra.legacyArgs 
github.com/spf13/cobra.(*Command).HasSubCommands 
github.com/spf13/cobra.(*Command).HasParent 
github.com/spf13/cobra.NoArgs 
github.com/spf13/cobra.writePreamble 
...

This was very fortunate because I’ve started using Go at work lately and I knew that it includes a lot of metadata in its binaries (this can be turned off but isn’t by default). So I looked around and found the whole structure of the app, and all the method signatures.

once-cli/ 
├── cmd/ 
│ └── once/ 
│ └── main.go 
│ 
└── internal/ 
    ├── archiver/ 
    │ ├── compressor.go 
    │ └── uncompressor.go 
    │ 
    ├── cmd/ 
    │ ├── auto_update.go 
    │ ├── auto_update_off.go 
    │ ├── auto_update_on.go 
    │ ├── data.go 
    │ ├── data_backup.go 
    │ ├── data_restore.go 
    │ ├── password.go 
    │ ├── password_reset.go 
    │ ├── root.go 
    │ ├── setup.go 
    │ ├── start.go 
    │ ├── status.go 
    │ ├── stop.go 
    │ ├── update.go 
    │ └── util.go 
    │ 
    ├── config/ 
    │ ├── config.go 
    │ ├── environment.go 
    │ └── verifier.go 
    │ 
    ├── docker/ 
    │ └── docker.go 
    │ 
    ├── networking/ 
    │ ├── dns_check.go 
    │ ├── network_check.go 
    │ └── ssl_traffic_check.go 
    │ 
    └── tui/ 
      ├── backup/ 
      │ └── backup.go 
      ├── components/ 
      │ └── task_list.go 
      ├── question/ 
      │ └── question.go 
      ├── restore/ 
      │ └── restore.go 
      ├── runner/ 
      │ └── runner.go 
      ├── setup/ 
      │ ├── installer.go 
      │ ├── registration.go 
      │ └── setup.go 
      └── util.go

And I found three URLs

https://auth.once.com/install/ 
https://auth.once.com/verify/%d 
registry.once.com

The first one was the same as the download URL, the second one obviously verifies something but I didn’t know what. My guess is for the password reset functionality (look at once-cli/internal/cmd/password_reset.go) because it interpolates a number at the end (that's the %d). The third one looks like a private Docker registry.

Can I login to the Docker registry?

After set up, Once creates a config file where it stores how you configured it.

// contents of /root/.config/once/config.json
{
  "token": "1111-1111-1111-1111",
  "product": "campfire",
  "product_name": "Campfire",
  "email_address": "stanko@example.com",
  "ssl_domain": "chat.rubyzg.org",
  "validation_token": "PHONY_VALIDATION_KEY",
  "secret_key_base": "PHONY_SECRET_KEY_BASE",
  "vapid_private_key": "PHONY_PRIVATE_KEY",
  "vapid_public_key": "PHONY_PUBLIC_KEY",
  "storage_location": "/var/once/campfire",
  "cron_hour": 2,
  "once_binary_etag": ""
}

Notice that it stores the license key and my email. Also notice that it calls the license key "token". My guess is that it uses the license key as an auth token with the registry to pull updates. That's a common way to do token auth with Docker Registries. Let's try that.

docker login \
  --username 'stanko@example.com' \
  --password '1111-1111-1111-1111' \
  'https://registry.once.com'

Login Succeeded

Bingo!

Now, where did it get my email, the product, and product name from? These weren't asked during set up. I'm guessing that the install URL is in the binary because it can respond with HTML / text for the install script but it can probably also respond to JSON. Let's try that

curl https://auth.once.com/install/1111-1111-1111-1111.json | jq
{
  "token": "1111-1111-1111-1111",
  "product": "campfire",
  "product_name": "Campfire",
  "email_address": "stanko@example.com",
  "ssl_domain": null,
  "validation_token": "<some-base64-string>"
}

Two for two!

I was curious about the validation token as it was obviously a Base64 string, but decoding it returned complete gibberish - it doesn't even look like unicode - so I assume that it's either completely random or an encrypted payload. Maybe that's what the verify endpoint is for?

My guess is that the token is used to update the "ssl_domain" field with the Once auth server after the install completes or during the DNS check. That would mean that the install endpoint probably also responds to a POST request.

Anyway, I now had a good understanding of how Once works.

How Once works

Once consists of three main parts:

The Auth server
The private Docker Registry
The CLI app

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

An overview of Once

The Auth server handles token validation for the registry, distribution of the CLI binary, license key inspection, and probably the order created webhook from Shopify.

The CLI handles Docker installation and configuration on the client, pulling and running of product Docker images, product container management, automatic-updates, backups, restores, and a few other things.

The private Docker Registry is, well, just a Docker Registry... It stores and distributes Docker images.

The license key you get is used as an access token to the registry. With it the CLI can pull images from the registry. That token probably grants access to a single repository in the registry - just for the product you purchased.

Building Twice

Now the fun part, making a clone of Once.

I decided to call this project "Twice" as a joke on the fact that it's a clone of Once - it's a second implementation. The source code is available on GitHub if you want to explore it, build, modify, or run Twice for yourself.

My main motivation is to learn more about how Once is set up, but a nice side effect is that I'll get a distribution system out of it - in the case that I decide to make an app and sell it. With that in mind, my primary goal is to implement just the setup CLI command. I'll work on everything else later.

Just like with Simpliki, I'll start from the center and work myself out. In the case of Twice the center is the CLI app - specifically the setup command.

As I've been working with Go at work lately, I felt confident in using it for the CLI. This would be a nice exercise to sharpen my Go skills. Otherwise I'd probably go with Rust as I have more experience with it, or I'd give Tebako a try - a new way to package Ruby apps into a single binary.

So I started a new Go project, added cobra to it and followed the getting started guide to create a CLI app with a setup command that accepts one argument - the license key.

go run cmd/twice/main.go setup --help
Install all the necessary dependencies to run a product and then installs the product associated with the given license key

Usage:
  twice setup <license-key> [flags]

Flags:
  -h, --help   help for setup

At this point I got lost in the weeds. Once has this nice TUI (terminal user interface) that walks you through the setup process so, naturally, I wanted to make a nice TUI too.

Twice TUI

I spent a couple of hours working on it, I got quite far, but in the end I realized the TUI didn't matter. It was just slowing me down, so I ditched it and replaced it with good old puts.

Twice CLI

This was waaaaay quicker to implement and allowed me to focus on the actual logic - like figuring out if docker was installed, is it running, writing an API client, interacting with Docker, etc. - instead of rendering a fancy UI.

I like the version with the TUI more - it feels refined - and I think I'll revisit it once I'm more comfortable with Go and Bubble tea (the library that renders the TUI).

As I was building the CLI I started the Auth app and mocked a few endpoints just to get the setup command done. I didn't implement any UI and relied heavily on "bin/rails db:fixtures:load" to give me some data to work with. These endpoints were plain Rails controllers that rendered JSON.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The models I had for Twice

With most of the CLI out of the way, and with some of the Auth app done I turned my attention to the Docker Registry.

There are a few Docker Registries out there:

I liked the idea of having a Registry that's written in Rails so I tried Portus, but I couldn't get it to run locally - there are multiple known issues and the project was archived 2 years ago.

Harbor is a conglomeration of micro services. When I saw that I had to run 4 or more containers just to get the thing going I gave up.

This left me with Registry. It has everything I need - access control via token auth to a configurable auth server. It has no UI or API, but this is a plus as I don't have to think about customers trying to login to the registry UI or interacting with it other than pulling images. And I can make the system stateless, thus less error-prone.

All I had to do to get the registry working is to generate a self-signed certificate

openssl req -x509 -newkey rsa:4096 -nodes \
    -keyout ./auth.key \
    -out ./auth.crt \
    -days 365 \
    -subj "/CN=registry-auth" \
    -addext "basicConstraints=critical,CA:true" \
    -addext "keyUsage=keyCertSign,digitalSignature" \
    -addext "subjectAltName=DNS:registry-auth"

And set a few environment variables that configure it to use token auth, tell it to use my auth server, and to verify the auth tokens using the certificate I just generated

# Use token auth
REGISTRY_AUTH=token
# Use `http://localhost:3000/registry/auth` as the auth server
REGISTRY_AUTH_TOKEN_REALM=http://localhost:3000/registry/auth
# This isn't too important
REGISTRY_AUTH_TOKEN_SERVICE=registry
# This has to match what you return from the auth server and what's in the certificate
REGISTRY_AUTH_TOKEN_ISSUER=registry-auth
# This specifies which certificate to use to verify tokens
REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/auth.pem

That's it. Here is a Docker compose file for it

services:
  registry:
    image: registry:2
    # On Linux
    network_mode: host
    # On Mac
    # ports:
    #   - 5000:5000
    environment:
      - REGISTRY_AUTH=token
      - REGISTRY_AUTH_TOKEN_REALM=http://localhost:3000/registry/auth
      - REGISTRY_AUTH_TOKEN_SERVICE=registry
      - REGISTRY_AUTH_TOKEN_ISSUER=registry-auth
      - REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/auth.pem
      - REGISTRY_STORAGE_DELETE_ENABLED=true
    volumes:
      - ./registry/tmp/registry_data:/var/lib/registry
      - ./registry/certs:/certs:ro

Now the only thing left was to finish the Auth server.

First, I tackled registry token auth. This requires a single endpoint that responds to both POST and GET requests

match "/registry/auth", to: "registry#auth", as: :registry_auth, via: %i[ get post ]

The endpoint receives the users email and token via HTTP basic auth and responds with a JSON payload that looks like this

{
  "token": "signed.jwt.token",
  "expires_in": 300,
  "issued_at": "2025-05-15T16:55:23Z"
}

The token from the response is a JWT that contains a list of access permissions - these define which actions the user can perform.

{
  "iss": "registry-auth", // Has to match the value set in REGISTRY_AUTH_TOKEN_ISSUER
  "sub": "stankoexample.com", // Username of the person logging in
  "aud": "registry", // Has to match REGISTRY_AUTH_TOKEN_SERVICE
  "exp": 1747329069,
  "nbf": 1747328784,
  "iat": 1747328784,
  "jti": "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed",
  "access": [
	{
	  "type": "repository",
	  "name": "campfire",
	  "actions": ["pull"]
	}
  ]
}

The tricky part was to sign the token with the certificate such that the registry would accept it as valid. This took me some time to figure out. To get it to work I had to set two additional JWT headers called KID and X5C, which tell the registry exactly which certificate it should validate the token with. Both headers are derived from the signing certificate itself.

KID = OpenSSL::Digest
  .new("SHA256")
  .update(PRIVATE_KEY.public_key.to_der)
  .hexdigest
  .upcase
  .scan(/.{1,2}/)
  .join(":")
X5C = Base64.strict_encode64(PUBLIC_KEY.to_der)

Putting it all together, the controller looks like this

def auth
  authenticate_or_request_with_http_basic do |email, license_key|
    user = User.find_by(email_address: email)
    deny_access and return if user.blank?

    license = user.licenses.find_by_key(license_key)

    token = user.generate_registry_access_token_to_product(license.product, service: params[:service])
    deny_access and return if token.blank?

    payload = {
      token: token.to_s,
      expires_in: token.duration.to_i,
      issued_at: token.issued_at.iso8601
    }

    render(json: payload, status: :ok)
  end
end

Notice that the token is an object. This provides a nice domain language (DDD domain language) that makes it easier to work with, and reason about, the token. If I want the duration or the issued at time I can ask the token for it; if I want to turn it into a string I can do that too.

At this point I noticed that I didn't implement a way to push images to the registry. To fix that I introduced a new type of user - the Developer. STI made the most sense for this, so I converted the customers table to a users table, added a type column and a few extra columns.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The models in Twice after adding developers

Unlike Customers, Developers login with a password tied to their account and they get full access to the registry.

Second, the install endpoint. This was straight forward

# config.routes.rb
get "/install/:license_key", to: "install#install", as: :install, defaults: { format: :text }

# app/controllers/install_controller.rb
def install
  respond_to do |format|
    format.text
    format.json
  end
end

And then I just had to create two separate view files. One for the script in app/views/install/install.text.erb

#!/usr/bin/env bash
set -e

echo "Installing the TWICE command..."
echo

sudo true

command_path="<%= install_download_url %>?arch=$(uname -m)&platform=$(uname)"
curl -s "$command_path" -o .twice_tmp
sudo mv .twice_tmp /usr/local/bin/twice
sudo chmod +x /usr/local/bin/twice

if [[ "$OSTYPE" == "darwin"* ]]; then
  # Don't use sudo on macOS; Docker Desktop is likely to be running as the user
  /usr/local/bin/twice setup <%= @license.key %>
else
  sudo /usr/local/bin/twice setup <%= @license.key %>
fi

And another in app/views/install/install.json.jbuilder

json.key @license.key

json.owner do
  json.(@license.owner, :id, :email_address)
end

json.product do
  json.(@license.product, :id, :name, :repository)
  json.registry ENV.fetch("REGISTRY_URL", "localhost:5000")
end

Notice that I strayed a bit from 37signals' implementation when it comes to downloading the binary. I decided to separate that out into another controller action because the install endpoint already does too many things in my opinion, and adding file sending to that seems like way too much responsibility for one endpoint.

I also made the decision that the auth server would directly send the file to the client instead of e.g. redirecting to something like S3. This makes it easier to self-host twice, and due to a trick it’s no less performant.

Here is the download action

def download
  platform = sanitize_file_name_part(params[:platform])
  arch = sanitize_file_name_part(params[:arch])

  filename = ["twice", platform, arch].compact.join("-")
  file_path = Pathname.new(Rails.root.join("storage", filename))

  if file_path.exist?
    send_file file_path, type: "application/octet-stream", filename: filename
  else
    head :not_found
  end
end

It assumes that the requested binary will be called "twice-#{platform}-#{arch}", then it goes and checks if such a file exists in the storage directory, and if it does it sends it.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The download endpoint in action

In production this won't actually stream the file from Ruby, instead it sets a special header that tells Thruster to send the file which makes things way more efficient.

Third, I added authentication using the new auth generator in Rails 8, and a UI for managing customers, developers, products and licenses.

Demo of the Auth app UI

And that was it. Twice was done. Here is the demo

Demo of Twice installing an app

Some loose thoughts

When writing about building Simpliki I left out an important part - what I intentionally didn't do.

For Twice, I intentionally didn't implement domain name tracking as I think that it's not important. It can't prevent people from running the docker container how and where they want. It would add complexity and I don't see the benefit of it, so I ditched it.

I do see the utility in all the backup and management commands that the Once CLI has, which that I didn't implement in Twice. I intend to return to that at some point. But they aren't important for distributing apps so I skipped them.

Twice, currently, has the same problem as Once - it doesn't play well with other apps or Kamal. This isn't a problem if you rent a VPS specifically to run a Once app, but if you have a large bare metal machine like me then this is cumbersome. I'd like to change it so that it deploys kamal-proxy and then proxies requests to different apps through it. This would allow it to run multiple apps on the same server.

That's also why the Twice CLI doesn't keep a config file - I'd like to make it so that you can run multiple apps, so the config has to support that. This was extra work that I think isn't important for now. And I'd like to be able to install an app from different Twice instances on the same server. Something like

twice setup 1111-1111-1111-1111@auth.foo.com
twice setup 2222-2222-2222-2222@auth.bar.com

This would make it more versatile - but, again, that's way beyond the scope of creating a clone to learn how Once works.

I liked working with Go. After years of using Rust, having a garbage collector and not having to think about results, unwrapping, lifetimes or borrowing was a breath of fresh air. Though, the way it declares modules still doesn't sit right with me. If you didn't look at the source code, each directory is its own module, and all files in the directory get essentially concatenated together into that module at compile time. It works, and I can't point out any problems with that, but it feels weird.

The folks at 37signals made a really elegant distribution system for their apps. I had a lot of fun figuring out how it works and I learned a lot along the way.

Adding MCP to a Rails app

2025-05-06T09:00:00Z

Last week we had an AI hackathon at work during which we added an MCP server to our Rails app. This was much easier than I expected, the result was both impressive and scary, and I learned a few things about MCP that I didn't expect.

Demo of Claude using our app via MCP to solve a support request

Goal

My team wanted to help our support team by automating some laborious tasks, like cross referencing data, by offloading the bulk of the work to an AI agent. This would make support’s work easier, responses quicker, and in some cases it would free up my team too - an all around win.

To allow the AI to interact with our app - like a person would - we decided to add MCP to it.

What is MCP?

MCP, or Model Context Protocol, is a protocol through which LLMs can interact with different tools and resources to accomplish tasks.

The protocol specifics are fairly dry so I'll gloss over them. What you need to know is that in MCP there exists a server and a client.

A server provides tools, prompts, and resources (I'll focus only on tools). While a client connects to one, or multiple, servers and exposes their tools to an LLM. The client then makes API requests to the Servers to use tools when the LLM tells it to.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Overview of how the MCP client and server interact

In essence - the Client uses the LLM as a brain to plan out how to solve a problem and the Servers as tools to implement the solution.

A tool can be anything from a query, like searching for a user, to a mutation, like creating a coupon code - anything the server is allowed to do on behalf of the LLM.

E.g. You could ask a client to "Find a file called meeting_notes.txt", it would then use the LLM to interpret what that means and plan out how it would do that with the tools it has available. It gets a list of tools from the MCP servers it's connected to.

The LLM might respond with something like "{action: 'tool_call', tool: 'find_file', arguments: ['meeting_notes.txt']}" and the client would go and make a request to an MCP server that exposes a tool called find_file.

The server then responds with something like "{path: '/home/user/meeting_notes.txt', contents: '...'}" and the client passes that on to the LLM which then decides what to do with that information.

MCP and Rails

I used the fast-mcp gem to add MCP support to our app.

bundle add fast-mcp
bin/rails generate fast_mcp:install

This gave me two new directories - app/tools and app/resources - and a class name conflict.

The resources directory contains an ApplicationResource class, but our app already has a different ApplicationResource class in app/models that we use to represent resources from 3rd party REST APIs.

To solve the conflict I decided to namespace everything MCP-related under an MCP namespace.

First I moved the two new directories into an mcp directory

mkdir app/mcp
mv app/tools app/mcp/tools
mv app/resources app/mcp/resources

But now the classes were namespaced with Tools and Resources instead of with MCP because Zeitwerk treats each directory in app/ as a root directory. To reconfigure Zeitwerk I added the following to config/initializers/zeitwerk.rb

# config/initializers/zeitwerk.rb

module MCP; end

Rails.autoloaders.main.tap do |autoloader|
  Dir.glob(Rails.root.join("app/mcp/*")).each do |file|
    next unless File.directory?(file)

    autoloader.push_dir(file, namespace: MCP)
  end
end

That tells Zeitwerk to treat every directory in app/mcp as a root directory with an MCP namespace.

Now app/mcp/resources/application_resource.rb defines MCP::ApplicationResource instead of Resources::ApplicationResource.

With that out of the way I could finally run my server

bin/rails s

The Client

Now that the server was up and running I wanted to see if I could connect to it. The easiest way to do that is through cURL

curl -v http://localhost:3000/mcp/sse
* Host localhost:3000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:3000...
* Connected to localhost (::1) port 3000
* using HTTP/1.x
> GET /mcp/sse HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/8.13.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< Content-Type: text/event-stream
< Cache-Control: no-cache, no-store, must-revalidate
< Connection: keep-alive
< X-Accel-Buffering: no
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET, OPTIONS
< Access-Control-Allow-Headers: Content-Type
< Access-Control-Max-Age: 86400
< Keep-Alive: timeout=600
< Pragma: no-cache
< Expires: 0
* no chunk, no close, no size. Assume close to signal end
<
: SSE connection established

event: endpoint
data: /mcp/messages?

retry: 100

: keep-alive 1

: keep-alive 2

: keep-alive 3

: keep-alive 4

: keep-alive 5

event: message
data: {"jsonrpc":"2.0","method":"ping","id":715239

It seems to be working.

But to actually use an LLM with the server I had to find a proper MCP client. This was way more difficult than I thought it would be, mostly because I'm on Linux.

By far the most well-known MCP client is Claude Desktop, but it isn't available on Linux even though it's an Electron app... I tested a few open-source alternatives but in the end a coworker told me that Zed (the text editor) has an MCP client built-in and it's available on Linux.

paru -Sy zed-preview-bin
zeditor .

(To get MCP support you have to join the Agentic Editing Beta)

To configure Zed I had to give it a shell command with which to start the server. This is a bit odd as I want Zed to connect to an already running server - my app - but MCP, at the moment, doesn't support connecting to remote servers. This is a planned feature that's supposed to release by the end of the year - but I need it now.

Luckily, someone had the same problem as I and created mcp-remote.

With that I configure Zed by giving it the following command

npx mcp-remote@0.0.22 http://localhost:3000/mcp/sse --allow-http --skip-browser-auth

Fast MCP generates a sample tool that, given a User ID and a prefix, returns a greeting for that user. With that in mind I asked

Can you greet a User with ID 636166899?

To my surprise it didn't respond with a greeting but started digging through the codebase

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Claude digging through the code base to figure out how to greet a user

After some digging around I soon discovered that, while the server was running, it wasn't advertising any tools.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

My MCP server advertising 0 tools to the client

The problem was that the initializer for Fast MCP uses the descendants method to load all tools that inherit from ApplicationTool.

server.register_tools(*MCP::ApplicationTool.descendants)
server.register_resources(*MCP::ApplicationResource.descendants

That method returns all classes that inherit from a given class, but a caveat of that method is that it returns only descendants that have been loaded so far.

Foo.descendants
# => []

class Bar < Foo
end

Foo.descendants
# => [Bar]

Since, in development, Rails disabled eager loading no descendants of ApplicationTool are loaded when the initializer runs, and therefore no tools get registered with the server.

There are 2 easy ways around this - either enable eager loading in development (at least for this), or manually register each individual tool.

I opted to go with eager loading as we already had it configured to trigger when an ENV variable is set, we use that in some tests.

This has the downside of requiring a server restart whenever you want to test something, but as you also have to restart your MCP client so that it pulls in the new server changes this wasn't a big deal for me.

# config/environment/development.rb

Rails.application.configure do
  # Eager load all classes if the EAGER_LOAD ENV is set
  config.eager_load = !!ENV['EAGER_LOAD']
  # ...
end

So all I had to do is restart the server with

EAGER_LOAD=1 bin/rails s

I entered the same prompt again, and... it didn't work. It started searching through the codebase again.

After some more digging I concluded that, to test my MCP server I'd have to create a new Profile

= 768px) 520px, 50vw"> = 768px) 520px, 50vw">

Location of the Profile tab in Zed

= 768px) 520px, 50vw"> = 768px) 520px, 50vw">

The Configure Profile section of the Profiles tab in Zed

= 768px) 520px, 50vw"> = 768px) 520px, 50vw">

The new Profile button in Zed

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Naming a new profile in Zed

And then configure that Profile's tools to include only my MCP server's tools.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Configuring tools for the new profile

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Enabling individual tools from different MCP servers

With that configured I tried the prompt again and this time it worked!

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Claude sending a greeting to a User through the MCP server

Building a tool

From here the team and I spent the day building various tools that would allow us to automate common support requests.

I implemented a simple building search tool

module MCP
  class SearchBuildingsTool < ApplicationTool
    tool_name "search_buildings"
    description "Searches for buildings"

    arguments do
      optional(:building_ids).array(:integer).description("IDs of the Buildings to look up")
      optional(:query).maybe(:string).description("The search query")
    end

    def call(building_ids: nil, query: nil,)
      buildings = Building.all

      if building_ids.present?
        buildings = buildings.where(id: building_ids)
      end

      if query.present?
        buildings = buildings.where("name LIKE :query OR handle LIKE :query", query: "%#{query}%")
      end

      buildings
	    .limit(50)
	    .order(name: :desc)
	    .to_json(only: [:id, :name, :handle, :address])
    end
  end
end

that worked quite well

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Demo of the Building search tool

Then I wanted to implement Access Log search. But to search Access Logs you need a building ID. I wasn't sure if the LLM could combine multiple tools by first searching for the building, getting its ID and then searching for the access logs with that ID. And I couldn't find a definitive answer online so I hoped for the best and implemented the following

module MCP
  class SearchAccessLogsTool < ApplicationTool
    tool_name "search_access_logs"
    description "Searches through Access Logs and returns the fist 25 results"

    arguments do
      required(:building_id).filled(:integer).description("ID of the Building that we are searching in")
      optional(:query).maybe(:string).description("The search query")
    end

    def call(building_id: nil, query: nil)
      building = begin
        Building.find(building_id)
      rescue ActiveRecord::RecordNotFound
        return "Building not found"
      end

	  building
	    .access_logs
	    .search({ query: query })
	    .order(logged_at: :desc)
	    .limit(25)
	    .to_json(only: [
	      :id, 
	      :logged_at, 
	      :grantor, 
	      :description,
	      :access_point_id,
	      :status
	    ])
    end
  end
end

To my surprise this worked! It reused the ID it found in the previous response to search for the access logs.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Claude reusing previous result to solve new problems

I was curious if it could actually combine tools so I started a new chat and asked the same question.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Claude combining multiple tools in a single response to solve a problem

The response blew my mind. It could combine tools! To me, this was both incredible and scary. Kind of like seeing primates use tools or develop their own economy.

Takeaways

MCP is a very "loose" protocol. It does allow you to specify what inputs your tool has, which are required and which are optional, but it's not expressive enough specify what the input should look like.

For example, in the Access Logs search I didn't specify what the query string should look like - my mistake - so the LLM just takes its best guess when you ask it something complex like "Can you give me all access logs created in the last hour in the building Crimson?".

I expected it to get all the access logs and then filter them on its own, but that's not what happened. Instead it passed "logged_at:>=2025-05-05T07:36:31" as the query. And, honestly, I can't blame it.

I later refined my tools with very detailed descriptions like this one

module MCP
  class SearchAccessLogsTool < ApplicationTool
    tool_name "search_access_logs"
    description <<~MD
      Searches for Access Logs in a given building and returns
      a JSON string containing an array of up to 25 objects.
      The list is sorted in descending order by the time 
      the access log was logged.
      Inputs:
      - building_id [Integer] (the ID of the building in which to search for access logs)
      - query [String, null] (a plain string that can contain any part ot the name of the grantor, the id of the access log, the description, the id of the associated access point)
      - logged_after [String, null] (An ISO8601 timestamp of the earliest time a log was recorded)
      - logged_before [String, null] (An ISO8601 timestamp of the latest time a log was recorded)
      - statuses [Array[String], null] (A comma-separated list of statuses)
      Output:
      - id [Integer] (The ID of the access log record)
      - logged_at [String] (ISO8601 timestamp of the time the log was recorded)
      - grantor [String, null] (Name of the person that granted this access, a null value means that the system granted the access)
      - description [String] (A description of what happened)
      - access_point_id [Integer] (ID of the access point that was accessed)
      - status [String] (The access point's state after this access)

      Statuses:
      - open - means that the access point was unlocked and that someone opened it
      - closed - means that the access point was unlocked, opened and then closed
      - left_ajar - means that the access point was unlocked, and then left open for more than 1min
      - denied - the access point never unlocked
      
      If the building couldn't be found you'll get a message back stating that.
    MD

    arguments do
      required(:building_id)
        .filled(:integer)
        .description("ID of the Building that we are searching in")
      optional(:query)
        .maybe(:string)
        .description("The search query which can contain any part of the grantors name, id of the record, id of the access point, or part of the description; any other input will be ignored")
      optional(:status)
	    .maybe(:string)
	    .description("Comma-separated list of statuses - valid statuses are open, closed, left_ajar, denied; any other statuses will be ignored")
    end

    def call(**args)
      # ...
    end
  end
end

The more information that I gave the LLM the better the result was. Something to watch out for is that you:

Describe the output (it's as important as describing the input)
Describe error states
List all possible enum values and describe what each value means
Explicitly state what format you expect for an input (what kind of timestamp do you expect, can it be nullable, etc.)
Explicitly state what's not allowed (e.g. additional enum values besides the ones listed)

One more thing. With smaller LLMs, like Llama 3 7b, I had to normalize input values. E.g. I'd tell it that I expect "building" as an enum value but it would pass me "Buildings". Or I'd tell it that a field is a nullable String but it would still pass an empty string when it had no value. But Rails makes it really easy to accommodate for such mistakes.

value.presence&.to_s&.strip&.singularize&.underscore

Conclusion

Integrating our existing app with AI was surprisingly easy. We just had to expose tools and the LLM would figure out how to use them. In 8 hours we managed to set up an MCP server, configure our clients, and expose enough tools to speed up a dozen common support requests, which is amazing.

In this hackathon, being a fully remote team with members in the EU and the US felt like a superpower. The EU team solved all the setup problems, so by the time the US team came online they could jump straight into building tools and testing ideas. For the price of 8 hours we got 14.

I still want to explore how to add permissions and some guard rails so that the LLM can't do too much damage if it goes haywire. I intentionally skipped that during the hackathon as it wasn't important.

This project was a technical success, but I'm not sure if it's the right direction to go in. This integration makes it so that developers get rarely involved in some types of support requests. As a developer, support is less about solving a problem and is more about figuring out customers' pain points and how to make the overall experience better.

I'm not sure how to strike a good balance between automating laborious tasks and surfacing common problems. That's also something I'll have to explore.

Building Simpl息

2025-04-24T06:00:00Z

This week I went for a coffee with a friend. As we talked the topic of hobbies came up. I mentioned that I was having a lot of fun working on a toy project over the past week, and then I showed it to him.

Demo

Him: So you are dabbling in React?

Me: No, this is just Rails.

Him: Which libraries did you use?

Me: None, this is as vanilla as it gets.

Him: What!?

Backstory

For the past 12 months, each month I built an app within a strict time-box of 12 hours.

If the project went overtime, and wasn’t at most 4 hours away from completion, I’d abandon it. If a project failed, and a lot of them did in the beginning, I’d reflect what led to that and tried to fix it in the next one.

My goal is to practice pragmatic decision making and to curb my perfectionism - which has started to show again.

The time-box really helped me make better decisions. At first all my apps failed because I either over-estimated my speed, or under-estimated the complexity and unknowns. But around the third one I got better at both. Then came the hard part - learning to strike a balance between quality work, functionality, and not obsessing over details.

Simpliki

Simpliki is the 12th app I’ve built.

It’s an app for practicing diaphragmatic breathing.

I got the idea for it from my girlfriend. She uses a similar app in her mindfulness practice.

The name is a combination of “simple” and the Japanese word “iki”, which means “breath”.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Google Translate result for translating "breath" to Japanese

You can practice breathing with Simpliki at simpliki.app. And you can peruse the source-code on GitHub.

Starting from the center

The best way to finish a project is to start from the center and work your way from there. I focus on what makes the app work - what makes it what it is - and tackle everything else later. That way I can cut less important bits if I have to.

For Simpliki the center is the breathing exercise - the place where you practice diaphragmatic breathing - without that there is no Simpliki.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Simpliki's center

But I also need a way to get to the breathing exercise - an index screen or something similar.

Personally, when I want to relax and practice mindfulness I don’t like to have too many options to choose from - it takes me out of the experience. So I decided to add a “home” screen with a single button that links to a random exercise.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Simpliki's two centers

My girlfriend likes to practice specific exercises so I added an “Explore” button that leads to an index page with all exercises.

An app can have many centers interconnected with different features. I think of the exercise and the home screen as two separate centers, and the index is just a feature between them.

I learned this from Christopher Alexander's design theory. The most apt analogy I found is in a paper that compares this design principle to patterns of a carpet.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Simpliki's two centers and the features between them

Now that I have an idea what I'm building I'll create a breadboard so that I can work out problems on paper instead of experimenting in code. I skipped this step for Simpliki as it was, well, simple. But the breadboard would look something like this.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The breadboard

This also helps me figure out the domain language - what models I have and what their attributes and methods are called.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The domain

Design

I had a clear vibe I wanted to go for - Endel + Headspace. I like Endel's minimalist and monochrome style - it has a mystical, simple, and concise vibe that I wanted to replicate. And I find the organic, whimsical, animated shapes and play button from Headspace calming - exactly the vibe I want for a mindfulness app.

= 768px) 520px, 50vw"> = 768px) 520px, 50vw">

Endel

= 768px) 520px, 50vw"> = 768px) 520px, 50vw">

Headspace

So I decided to combine the two for Simpliki. I'd use Endel's color-scheme and the animated organic shapes from Headspace.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Simpliki's UI

Starting a new project

I decided to use Tailwind because I like working with it. And I chose SQLite as the database because it's the simplest one to use out of the presets. I honestly don't even need a database for this app, but it's easier to use one than not.

rails new simpliki --css=tailwind

Then in that project I generated my models and a controller

bin/rails g model Exercise name:string

bin/rails g model Exercise::Step exercise:belongs_to action:string duration:integer position:integer

bin/rails g scaffold_controller Exercise

I removed everything that I didn't need, added a "home" action to the controller, a view for that action, and pointed the root route to that action.

Then I created a few fixtures for Exercises and Exercise::Steps.

Over the past 5 years I've completely changed my mind about fixtures. I used to think that they are finicky and convoluted compared to factories. But I've come to like them more after I've seen the downsides of factories over the years - they are extremely slow, it's hard to manage what will get created or you have to define every single record you want to create in every single test.

My favorite advantage of fixtures is:

bin/rails db:fixtures:load

This will take all my fixtures and put them in my current (development or production) database.

Now I don't need to create a way to add or edit Exercises, or add and edit them manually through the console, before I can start working on the show page. I can just load in a few exercises and use them. Best of all, I get to see and use the exact same data that's in my tests - no more trying to figure out what went wrong, I can just open the same Exercise and see.

Then I started working on the views. At first I just added links between the pages, added a header that said "Simpl息" on each page, added a plain HTML "Practice" button on the exercise show page, and created a very simple index page that shows a list of exercises.

The ruby tag

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Simpliki's logo

Then I remembered reading about the HTML ruby element which is used to show the pronunciation of characters. I thought it would be a nice way to show where the app's name comes from.

<h1 id="logo" class="font-serif">
  <span class="mt-4">Simpl</span>
  <ruby>
	  息
	  <rt class="text-gray-500">iki</rt>
  </ruby>
</h1>

It took about a minute to add and style the "iki" - plain HTML is amazing.

Animating SVGs

Next I wanted to animate the play button. I've done similar things before so I was quite confident that I could do it quickly.

Demo of the circle animation

The basic idea is to have an SVG with a single path that consists of 16 points with Bezier curves that form a circle. The number of points determines how organic and uneven the circle will look in the end - 8 is too blocky, 24 is too smooth, 16 is just right.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Basic SVG circle with 16 points and a Bezier curve

Then, every 10ms I change the position of each point slightly and adjust their Bezier curve points to animate the circle.

To make it undulate - move like there are ripples going across it - I can pull or push one point after another in or out, and then adjust the Bezier curve points to make it look hand-drawn.

That's an old computer science trick I learned in college. It sounds complicated, but it's quite straightforward.

There is just one thing you have to know - if you plot x = cos(a) and y = sin(a) where a is any value between 0 and 2PI you'll get a circle of radius 1. Then to get any size circle you just multiply everything by the radius you want.

const NUM_POINTS = 16
const RADIUS = 100

function generatePoints(time) {
  const circlePoints = []
	
  for (let index = 0; index < NUM_POINTS; i++) {
    // Calculate at what angle the point is supposed to be
    const angle = (point.index / NUM_POINTS) * Math.PI * 2
	
    // Convert the angle to coordinates
    const x = RADIUS * Math.cos(angle)
    const y = RADIUS * Math.sin(angle)
	
    circlePoints.push([x, y])
  }

  return circlePoints
}

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The most basic circle

To form waves, I can change the radius for each point using any function that also includes COS and SIN. That way each point moves in its own little circle which, when put all together, looks like a wave. And to make it consistent the SIN and COS have to include the current time.

const NUM_POINTS = 16
const RADIUS = 100
const RIPPLE_FREQ = 0.5 // determines how quickly the waves move
const RIPPLE_AMPLITUDE = 5 // determines how big the waves are

const circlePoints = []

function generatePoints(time) {
  const circlePoints = []
	
  for (let index = 0; index < NUM_POINTS; i++) {
    // Calculate at what angle the point is supposed to be
    const angle = (point.index / NUM_POINTS) * Math.PI * 2

    // Generate a new radius
    // the values were completely arbitrarily chosen
    // you could also just use sin(time) + cos(time)
    // this just looks nice to me
    const rippleY = Math.sin(time * RIPPLE_FREQ + angle)
    const rippleX = Math.cos(time * RIPPLE_FREQ * 0.7 + angle * 1.3)
    const ripple = (rippleY * 0.6 + rippleX * 0.4) * RIPPLE_AMPLITUDE
	
    // Convert the angle to coordinates
    const x = RADIUS * Math.cos(angle) + (ripple * Math.cos(angle))
    const y = RADIUS * Math.sin(angle) + (ripple * Math.sin(angle))
	
    circlePoints.push([x, y])
  }
  
  return circlePoints
}

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Circle with ripples

Then I can just move the bezier curve anchors in the same way to get a hand-drawn look.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Circle with ripples and fixed Bezier curves

The end effect is subtle, but it makes the circle look friendlier.

View transitions

Next I animated the view transitions. This takes maybe 10 minutes but makes the app feel way more modern and responsive.

First I enabled view transitions in Turbo Drive by adding this meta tag to the head section of my layout

<meta name="view-transition" content="same-origin" />

Just adding this one line will already add a fade effect to all page transitions. Then to move individual elements around, like the logo, I have to add it a view-transition-name CSS property with a unique name.

#logo {
  view-transition-name: logo;
}

After that the browser animates the transitions on its own.

Demo of the logo moving with page navigation

For the explore page I wanted to add an animation where the list slides in from the bottom of the page. Luckily this can be done with a few lines of CSS.

@keyframes slide-to-top-from-out-of-view {
  0% {
    transform: translateY(100vh);
    opacity: 0;
  }
  100% {
    transform: translateY(0);
    opacity: 1;
  }
}

#exercises {
  view-transition-name: exercises;
}

html:not([data-same-page-visit])[data-turbo-visit-direction="forward"]::view-transition-new(exercises) {
  animation: 0.25s ease-out slide-to-top-from-out-of-view both;
}

This tells the browser to apply a slide-to-top-from-out-of-view animation for any new exercises list that appear.

Demo of the exercise list sliding in from the bottom of the screen

There is one caveat, View Transitions aren't supported on Firefox. This only works on Chromium and WebKit based browsers. As one of the 2% of global Firefox users I don't mind, the app still works without the transitions.

The breathing animation

Now that I have an animated circle I want to create a breathing animation to help people pace their breath.

The circle should grow when you have to breathe in, and shrink when you have to breathe out.

Demo of the breathing animation

To animate this I need to create three circles - a maximum indicator, a minimum indicator and a progress indicator.

The maximum indicator is static and just has an opacity of 10%. The minimum indicator is mostly static after an initial animation that moves it in place. And the progress indicator constantly shrinks and grows as needed.

But rendering one animated SVG at 60 FPS is already quite taxing, and synchronizing three undulation animations would be a headache. Luckily plain HTML saves the day again with SVG symbols.

In an SVG you can define any set of paths or shapes as a symbol and give it an ID. Then you can render the same symbol in any other SVG in the same HTML with the use tag. You can even change the fill and stroke properties on the use tag to get different versions of the same shape. And when you update the shape in the symbol it immediately reflects in all other SVGs using that symbol.

<!-- Defines a symbol "foo" as a circle -->
<svg xmlns="http://www.w3.org/2000/svg">
  <symbol id="foo" viewBox="0 0 100 100">
    <circle cx="50" cy="50" r="50" />
  </symbol>
</svg>

<!-- Renders a red circle with a black stroke -->
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100">
  <use href="#foo" fill="red" stroke="black" stroke-width="2"></use>
</svg>

<!-- Renders a blue circle -->
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100">
  <use href="#foo" fill="blue"></use>
</svg>

With a symbol I just have to animate one SVG and then I can reuse the same animated shape in any other SVG on the same page. I've created a helper to make things more readable.

circle_id = dom_id(@exercise, :circle)

# Renders an animated circle as a symbol with the id circle_id
animated_circle(symbol: circle_id, class: "hidden") +
# Renders the minimum indicator
animated_circle(use: circle_id, class: "absolute inset-0 z-30 w-full select-none", stroke: "black", stroke_width: 2, name: "exercise-circle-min", data: { exercise_target: "minCircle" }) +
# Renders the progress indicator
animated_circle(use: circle_id, class: "absolute inset-0 z-20 w-full select-none", data: { exercise_target: "progressCircle" }) +
# Renders the maximum indicator
animated_circle(use: circle_id, class: "absolute inset-0 z-10 w-full select-none opacity-10", name: "exercise-circle-max", data: { exercise_target: "maxCircle" })

With that out of the way I now have to animate the progress circle. I read about the Web Animations API a few weeks ago and it seemed like just the right tool for the job.

It was my first time using that API and I think I hit every edge case there is with it.

Long story short, don't use fill forwards and instead manually set your desired state with the onfinish callback. For some reason, fill forwards applied a new CSS rule every time an animation ran. After a few minutes I had a hundred animation style blocks in my Inspection window, all overriding each other. Also, remove any transitions on the animated element as that will make the animation wonky in Firefox (but works fine everywhere else) - the browser will handle the transition for you.

After I figured that out, the rest was straightforward

async function inhale(duration, element) {
  return new Promise(resolve => {
    const animation = this.progressCircleTarget.animate([
      { transform: `scale(0.25)` },
      { transform: `scale(1.0)` }
    ], {
      duration: duration * 1000,
      easing: "linear"
    })

    animation.onfinish = () => {
      this.progressCircleTarget.style.transform = `scale(1.0)`
      resolve()
    }
  }
}

async function exhale(duration, element) {
  return new Promise(resolve => {
    const animation = this.progressCircleTarget.animate([
      { transform: `scale(1.0)` },
      { transform: `scale(0.25)` }
    ], {
      duration: duration * 1000,
      easing: "linear"
    })

    animation.onfinish = () => {
      this.progressCircleTarget.style.transform = `scale(0.25)`
      resolve()
    }
  }
}

Now I just had to pass the exercise steps to the Stimulus controller, and run the animation. This just meant that I had to define a steps value of type Array and serialize the steps as JSON on the HTML element.

Sound

With the animation done I wanted to also add an audio cue that you should breathe in or out.

At first I thought that the simplest way to add sound was to play some sound file - like a gong or a flute - at different playback speeds for different durations, and pitch it down to cue and exhale.

This turned out to be a dead end. I could change the playback speed but at 0.5x playback the sound of a gong starts to sound like Hell's bells.

Then I remembered the Web Audio API with which you can generate sounds. I found a nice tutorial that explains how to make a bell sound, but I was out of my depth here and decided to reach for an LLM so that I don't run out of time.

I created a scaffold class with the interfaces I wanted to have and asked ChatGPT to implement the class so that it generates "a nice and relaxing sound that inspires breathing in a mindfulness app I'm working on, kind of like Endel or Headspace". After a bit of back and forth I had something that I thought was decent.

But it didn't work on my iPhone... Turns out that Apple disables Web Audio playback if your ringer is set to silent, even though the volume is up - odd choice, but ok.

You can listen to the sound at simpliki.app, just pick any exercise and start it. If you are on an iPhone be sure to set your phone OFF silent.

Scroll snapping

As I was working on the breathing animation I noticed that I forgot to record if you are supposed to breathe with your nose or mouth, so I had to go back, add a migration, and update the fixtures.

Then I thought that showing a short description about the exercise would be nice so I added ActionText, and a few fixtures for it.

bin/rails action_text:install

But now that the exercise page could have an overflow and users would have to scroll I wanted the scrollbar to snap if you were close to the top of the screen. That way the exercise circle would always be in focus and you can't accidentally scroll away.

Once again HTML, or rather CSS, saves the day - this time with scroll-snap.

I just added the snap-y and snap-mandatory Tailwind classes to the container, and snap-start to the container of the exercise circle - that's it. Now the browser snaps to the top when you scroll near the top of the page.

Demo of the page snapping to the top when you scroll near the top

Full text search

Since I had four hours left at this point I decided to add full-text search to the index page.

This too was surprisingly easy thanks to an article by Sergio Alvarez. All I needed to do was to create a special kind of virtual table

class AddFullTextSearchToExercises < ActiveRecord::Migration[8.0]
  def change
    create_virtual_table :exercise_full_text_search_vectors, :fts5, %w[name description]
  end
end

Populate it and then query it. I created a concern under the Exercise class that does that. That way the search logic is nicely self-contained.

# frozen_string_literal: true

class Exercise
  module Searchable
    extend ActiveSupport::Concern

    DEFAULT_LIMIT = 25
    VECTOR_TABLE_NAME = FullTextSearchVector.table_name

    included do
      has_one :full_text_search_vector, foreign_key: :rowid, dependent: :destroy

      after_save :create_or_update_full_text_search_vector
    end

    class_methods do
      def search(query)
        return all if query.blank?

        where("#{VECTOR_TABLE_NAME} MATCH ?", escape_fts_match_value(query))
          .joins(:full_text_search_vector)
          .order("bm25(#{VECTOR_TABLE_NAME})")
          .distinct
      end

      def escape_fts_match_value(input)
        input.to_s
          .gsub('"', '""')
          .split(/\s+/)
          .map { |word| %Q("#{word}") }
          .join(" ")
      end

      def create_or_update_all_full_text_search_vectors
        all.find_each(&:create_or_update_full_text_search_vector)
      end
    end

    def create_or_update_full_text_search_vector
      params = {
        id: id,
        name: name,
        description: description.to_plain_text
      }

      sql = if full_text_search_vector
        ActiveRecord::Base.sanitize_sql_array(
          [
            "UPDATE #{VECTOR_TABLE_NAME} SET name = :name, description = :description WHERE rowid = :id",
            params
          ]
        )
      else
        ActiveRecord::Base.sanitize_sql_array(
          [
            "INSERT INTO #{VECTOR_TABLE_NAME} (rowid, name, description) VALUES (:id, :name, :description)",
            params
          ]
        )
      end

      self.class.connection.execute(sql)
    end
  end
end

With that I can do a full-text search across all exercises

Exercise.search("box")

Pagination

Now that I had search I wanted to add pagination. Years ago while exploring what Strada/Turbo Native was before it was released I saw that 37signals has a really neat way of handling pagination.

They have a "pagination" controller that goes and fetches the HTML of the next page, parses it to get the contents of itself from the next page, and then adds that contents to the current page.

No need for special routes or turbo-stream handlers, you just render the paginated page like always and let the controller do the rest. So I tried to recreate that.

First I tried to fetch the next page and parse it using DOMParser to get the next page's document. My first stab worked quite well.

  async #fetchNextPageDocument() {
    const url = this.nextPageLinkTarget.href

    const response = await fetch(url, {
      headers: {
        "Accept": "text/html",
      }
    })

    if (response.ok) {
      return new DOMParser().parseFromString(await response.text(), "text/html")
    } else {
      console.error("Failed to fetch next page:", response.status)
      return null
    }
  }

Then all I had to do was append the children from the new document to the current one

	const doc = await this.#fetchNextPageDocument()

    const newList = doc.querySelector(`[data-${this.identifier}-target="list"]`)

    Array.from(newList.children).forEach(child => {
      this.listTarget.appendChild(child.cloneNode(true));
    });

    const newNextPageLink = doc.querySelector(`[data-${this.identifier}-target="nextPageLink"]`)
    if (newNextPageLink) {
      this.nextPageLinkTarget.replaceWith(newNextPageLink.cloneNode(true))
    } else {
      this.nextPageLinkTarget.remove()
    }

And it worked like a charm. I just render the next page like always and let the controller do the rest.

Demo of the pagination controller loading in more exercises

Auth, create, update & delete

Since I still had time left I decided to implement CRUD for exercises. Here I decided to use the new auth generator. It gave me Users and Sessions, all I now needed to do was add views.

I quickly wrote a nested form Stimulus controller, some CSS to style the inputs and called it a day.

Demo of the sign in process and CRUD actions

Conclusion

I wanted to share how I built Simpliki because everyone I show it to seems to think that it's some SPA made with the latest technology trend when in reality it's just a vanilla Rails app that uses a few browser APIs.

Modern browsers are capable of a lot that just a few years ago required cutting edge technology and all the complexity that comes with it.

But I also wanted to sing the praises of time-boxing your work and working on toy-projects. It helped me develop a healthier way to think and break an old habit.

Adversarial Web Services

2025-03-03T07:00:00Z

My first job was at a digital agency. In addition to app development we also hosted the apps that we'd develop for our clients. This allowed us to avoid fiddly managed hosts, and since we ran everything on a beefy bare-metal server we didn't have to manage dozens of machines at different providers. All in all, this was a good setup.

Then, one Friday morning, I came into the office and went to clock in using an internal app. But the app just timed out. We soon noticed that all internal apps and our client's apps were timing out, our bare-metal server was completely unresponsive, and then came the bombshell - both hard drives in the server had failed.

Luckily we had hourly backups. Or so we thought. We found out the hard way that our backup script had a bug that caused it to create empty backups for the past few months. So we had no viable backup to restore from.

We spent the next 22 hours provisioning a new machine, re-deploying projects to it, recovering databases and files from the old machine and uploading them to the new one. By some miracle we managed to recover everything.

This incident left a sour taste in everybody's mouth. Everything was back to normal, but nobody was confident that it would stay that way for long. We lost confidence in our tools and ourselves. The next week we were told that we’d be abandoning our bare-metal server and would move all our apps to AWS so that we wouldn't have to monitor the hardware and backups were built-in.

Since that Friday, for more than a decade now, I've used AWS at every job I had.

I've spent countless hours working with its services, configuring and misconfiguring them, optimizing them for performance and cost. And in my opinion it just isn't worth it for most people.

AWS makes things easy, not simple

Some people think that AWS-hosted services are somehow magical 1-click solutions that you pay for and forget about. But the truth is far from that.

While you can set up a MySQL/Redis/Kafka instance in just a few easy clicks, after that you have to configure, tweak, monitor and manage that service like you would if you'd host it yourself.

In other words, the complexity is the same, but the setup is easier.

In fact, it's even more complex than running your own MySQL/Postgres/Redis/... server because AWS introduces its own arbitrary limitations on what you can and can't do. And on top of that it also introduces new concepts that you'll have to learn and manage - Floating IPs, Load Balancers, CPU credits, Network credits, IOPS credits, EBS credits, VPCs, Security Groups, IAM roles...

If you don’t learn these things, you will eventually either encounter a problem you can’t fix or receive an exorbitant bill from AWS.

Probably the best example of this is EC2. You can set up an instance in a minute. And you can probably deploy something to it with minimal knowledge of Linux and/or Docker. But what will you do, for example, if you run out of IOPS credits? Or if Docker refuses to start after a mandatory update? Or your instance suddenly changes its IP address?

Setting up a VPS is extremely easy, but if you don't know Linux, networking and some AWS gotchas, but managing and monitoring the instance is still complex.

Think of it as sending an email. It's simple, all you have to do is open up Gmail, write a message, enter a recipient and hit send. But under the hood what is actually happening is a complex exchange between email servers that involves IMAP, DKIM, SPF, DNS, Spam detection, Spoofing detection, reputation management, firewalls, storage, and more.

I hope that some of the acronyms and concepts of the complexity of email servers are foreign to you because that's my whole point.

By buying email from Gmail you are giving Google money so that you don't have to deal with the complexities of running an email server. But at AWS you are giving Amazon money to get your own server. They will set it up for you, but then it's up to you to deal with the complexity of running that server.

Take everything with a grain of salt

One of the biggest mistakes I ever made was to trust AWS marketing at face value. Their marketing often omits important or inconvenient details.

Figuring out what you are really buying requires you to read the product's documentation - not one page, not the pricing page, but all of it - and even then you have to take it with a grain of salt.

Take the incident I wrote about 2 weeks ago. At work we use ElastiCache Redis as a PubSub provider for our WebSockets. One day all our devices started disconnecting randomly. After weeks of investigation it turned out that we hit a CPU cap even though the instance claimed it was at 10% CPU during the incident. This happened because we were using a node type that's capped at 10% CPU which is, conveniently and vaguely, only mentioned in the documentation while the pricing page says that we get 2 full vCPUs.

Or how API Gateway offers WebSockets. But you can't have a connection that lasts longer than 2h, and you can't reliably know if a client has disconnected.

Or how Aurora offers unprecedented speed and infinite storage. But it might corrupt your database. And high load on a read-replica degrades the primary's performance negating the benefit of having the replica.

There are many more examples.

But the gist of the story is that you have to carefully navigate AWS' documentation labyrinth - and it really is a labyrinth - to gather as much information as you can before you commit to a purchase. And even then you aren't 100% sure what you've purchased or how much it will cost you in the end.

Some might say that this is a "skill issue", or just "RTFM". But I think that people who say this are either stuck in an inferiority complex or are just coping.

Imagine if any other business sold stuff like AWS does. Let's take a restaurant as an example.

You go to a restaurant and want to order a pizza. The waiter walks up to you and tells you what pizzas they have. There are a few you like so you ask for the price and the waiter brings you the menu. You see that the pizza you like costs 15€ for a large so you order it. Then a short while later the waiter brings you a single slice of the pizza. You ask him where's the rest of the pizza and he points out that on the back of the menu it says that some pizzas come with limitations mentioned in the "limitation list". Then he brings you the limitation list which states that the pizza you ordered is served per slice.

You'd probably never go to that place ever again.

Sometimes it can feel like racketeering

After using AWS for a while I've noticed that it isn't as elastic as it claims to be, and somehow that feels intentional.

Let's take the WebSocket disconnect incident again. We were using an instance that has 2 vCPUs but can only use 10% consistently and we decided to upgrade to an instance that can use 100% of the CPU available to it.

That’s a 10x increase in compute, so we expected a 10x increase in price - and that’s exactly what we got. The monthly price for our instance went up from about 10€ to about 100€.

But we soon learned that, while we did get 10x the CPU, we got only 1/5 of the bandwidth.

So we upgraded to the first node type that offered the same bandwidth as before and our monthly cost rose to 200€ per month.

That's a 20x increase in price for 10x the compute.

Never mind the exorbitant price for what is essentially a managed EC2 instance that otherwise costs 60€. What bugs me the most here is that we're talking about a virtual machine, not a bare-metal server. You can give it as much CPU, memory and bandwidth as you want.

Nearly all AWS services give you a predefined list of instance types to choose from - this is not just an ElastiCache issue.

I understand why smaller hosting providers make you choose from predefined instance types - they simply don't have the hardware to allow anybody to choose what they need. But a hyperscaler like AWS? One that calls their VPS offering Elastic Cloud Compute?

Another thing I noticed is that some services have conveniently set, completely arbitrary, limitations that seem to exist only to make AWS more money.

For instance, API Gateway imposes a maximum duration on a WebSocket of 2h and will close any idle connections after 10min. But at the same time AWS charges you per WebSocket message in addition to each minute each client was connected, and most applications will need at least 1 message after a reconnect.

If you have 50k clients you'll pay AWS 200€ every month just to keep them connected to a WebSocket 24/7. For comparison, a 60€ EC2 instance can handle more clients than that with no limitations.

To go back to the restaurant analogy. Imagine if you bought a Pizza at a restaurant and the waiter starts bringing you your pizza one slice at a time, because they decided to do so for whatever reason. Then, at the end of the meal, they charge you a delivery fee for every individual slice on top of the price of the pizza.

The pizza would have to be out of this world for me to come back to that restaurant.

When you run into such pricing practices AWS doesn't feel like a partner helping you build stuff anymore; it feels like an adversary trying to squeeze every penny out of you.

Serfs and sovereigns

My biggest gripe with AWS are its proprietary services. Some of them are great and novel while others are OK equivalents of existing open-source services.

When it comes to great and novel services, the first thing that comes to my mind is S3.

You can get S3-compatible object storage at a dozen places today - DigitalOcean Spaces, Backblaze B2, Cloudflare R2, Hetzner's, OVH's and Scaleway's Object Storage, or open-source Garage and MinIO just to name a few - but that wasn't always the case.

A little over 10 years ago S3 reigned supreme when it came to storage. The only real alternative to it was to either store the files locally on your server or using a network attached drive, both of which required some skill to scale and manage - at least compared to S3.

Back then, it was common for people to e.g. use a managed host but still have an AWS account just to get access to S3. Problem was, we were completely dependent on AWS for object storage. If they raised prices we had no real option but to pay. There was no contingency if S3 had an incident you just had to deal with the customer backlash.

When other S3-compatible object storage providers started popping up not only did prices come down but you could suddenly choose a provider that suits your need - no ingress fees, no egress fees, cheaper long-term storage, faster storage, HIPPAA-compliant storage, etc. You could mix and match providers to suit your need, or use multiple as a contingency.

But most importantly, any code I write today that fetches or uploads files to S3 will work just as well anywhere. This means that, if I don't like a price hike or a term change from my current object storage provider, I can just pick up my stuff and move to another provider that offers more favorable terms.

When it comes to S3 - 10 years ago we were basically serfs, today we are sovereign and I don't miss the old days one bit.

There are other great AWS services - like DynamoDB - but I avoid them in favor of open-source alternatives just because I don't want to find myself locked in and having to accept any term change to avoid spending days switching to something else.

Competence

The incident that happened at my first job wasn't caused by our hosting provider, it was caused by us. Our hosting provider at the time gave us all the tools and services we needed to prevent that incident from happening, we just didn't know how to use them. It was our incompetence that caused the incident.

As a knee-jerk reaction we migrated to AWS chasing a promise, or rather a dream, that it will somehow save us from our incompetence - it didn't. All it did was mask that incompetence by making things easy to set up, but our old problems all eventually reoccurred.

Only by learning to run my own server through learning Linux, Docker, networking, monitoring, backups and server security did I manage to avoid making the same mistakes - not just with AWS but at any host.

And that's the take-away here - AWS, or any other managed service provider, won't save you from having to learn how to manage that service. They will give you monitoring, but you still have to know what the metrics mean and what's acceptable for you. They will give you hardware, but you have to know what's enough for you. They will give you firewalls and other networking tools, but you still have to configure them for your use-case.

All you need to learn this stuff is a Virtual Machine or a Raspberry Pi and some time. Set up a small server and try to configure a firewall, install a database, run an app, upgrade the database. If you fail, and you probably will, just re-install and try again.

One of the best decisions I ever made was learning how to self-host the services I use at work. It was difficult at first - as going from 0 to 1 always is - but it made me much more confident and better at my job. I now understand not only how to operate these services but also how to optimize my code to make the most of them. And best of all is that these skills carry over to any hosting provider, project and job.

If you are willing to learn, I think you will be better off renting a few VPS instances or bare-metal machines from the likes of DigitalOcean, Hetzner, OVH, and Scaleway. The predictable pricing, and lack of AWS BS, will save you time and money. Or if you don't want to learn all this, then go with a hosting platform like Heroku or Fly.

Monitoring ActionCable

2025-01-27T10:26:00Z

A few weeks ago I was woken up with the following message "Hey, devices are randomly disconnecting from WebSockets can you come online and help?".

Usually, when a widespread problem like this occurs out of the blue, my first goal is to make the app mostly operational again and my second goal is to fix the cause of the problem. To do that I usually go over each component that could be causing this to check if it's functioning correctly and then drill down on any components that aren't.

Step one, identify which components could be at fault.

We use Rails' ActionCable for WebSockets. I've done a deep dive into ActionCable in a previous article if you'd like to read more about it.

ActionCable, except for the Ruby process itself, depends on Redis (in my case) and on the database(s) that the app uses.

Step two, prioritize.

Since I knew that the devices were randomly disconnecting I assumed that either app containers were restarting for whatever reason, or that something was causing ActionCable's worker pool to lag which then caused missed heartbeats.

This means that the Ruby process is the most likely culprit, then the database, and finally Redis.

Step three, check each components.

I checked Grafana to see what's going on with the app containers and saw that CPU was elevated but wasn't exceeding 80%, while memory didn't go above 70%, and the auto-scaler was adding more containers as needed. Everything was in order given that a reconnect storm was going on.

Then I checked Elastic APM to see what was going on with /cable. But the metrics were phenomenal - sub-millisecond response times and 100% successful responses. Fantastic numbers for a reconnect storm.

Next I checked the database. CPU and memory were at average levels, there were no slow queries. Everything was fine.

Lastly, I checked Redis. To my surprise, here too, everything was fine. The CPU never exceeded 10% and memory was at 5%.

What was going on?

Step four, stabilize the app.

Since Redis and the database were fine, in an Hail Mary attempt, I scaled-out the number of app containers to stop the disconnects. Surprisingly, this worked... This pointed a spotlight at the application code and ActionCable. They must have been the faulty component since the scale-out fixed the issue - but how?

Step five, find the root cause.

At that point I realized that I had no clue what was going on with our WebSockets after they were established. Elastic APM only captures metrics about the HTTP part of the WebSocket connection, but once it was established I had absolutely no insight into what's going on - how many clients there are, what the latency was like, how many messages were being sent...

The best way to figure that out was to introduce some kind of monitoring. But what should I monitor and how should I surface those metrics?

I'm not one to reinvent the wheel, so I checked if some tool for monitoring ActionCable already exists. I found out that Sentry's and AppSignal's APM tools monitor ActionCable's action execution times.

Action Execution Time

Action execution time measures how long it takes for a Channel to process a received message. This metric is important because it can show if you have a slow action which binds up the worker pool for too long and thus lowers throughput and increases latency. It's a nice way to quickly see if everything is alright with ActionCable.

I quickly exposed that metric to Elastic APM using the following code.

# config/initializers/elastic_apm.rb

ActiveSupport::Notifications.subscribe "perform_action.action_cable" do |event|
  transaction = ElasticAPM.start_transaction "#{event.payload[:channel_class]}##{event.payload[:action]}"
  transaction.start(event.time * 1000.0)
  
  ElasticAPM.end_transaction(event.payload[:exception].blank?)
end

That worked!

PubSub latency

I soon noticed that some extra metrics, like the PubSub latency, would be helpful. Knowing the PubSub latency would help me identify problems with the PubSub component of ActionCable. If the latency - the time it takes for a message to be sent to clients since it was broadcast from the app - goes up that means that the PubSub service, in my case Redis, is probably the cause of a problem.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Diagram that shows what PubSub latency measures

To measure that latency I have to know how much time has passed since a message was broadcast to when it was received in the ActionCable server. The easiest way to do that is to broadcast a message with the current time in it, then receive it in some channel and calculate the time difference between the time I received the message and the timestamp in the message.

To send the a message with the current timestamp I can do the following and run it from a cron job every minute

measurement_payload = {
  sent_at: Time.now.to_f
}

ActionCable.server.broadcast("metrics", measurement_payload)

To receive the message and calculate the latency I can do this

stream_from("metrics", proc do |json|
  payload = ActiveSupport::JSON.decode(json)
  sent_at = payload[:sent_at]
  latency = Time.now.to_f - sent_at
  
  Rails.logger.debug "PubSub Latency: #{latency}"
end)

But where can I put this code and how do I expose the latency?

After some experimentation I decided to create a stream from every open channel in ActionCable. This sounds pretty bad, and I'd like to find a better way to do this, but it has some upsides and the downsides can be minimized. The upside is that no measurements are performed if there are no clients. The downside is that each client has its own metrics subscription even though only one subscription per server is needed.

There are many ways to limit the subscription to be one per server and I opted for the simplest one - a mutex. A mutex, aka mutually exclusive lock, allows me to ensure that just one thread is running a piece of code at a time. In Ruby, you'd usually use Mutex#synchronize like so:

# Create the mutex and a counter
mutex = Mutex.new
count = 0

# Defines a proc that prints the value of the counter
# and increments it by 1
work = proc do
  10.times do
    mutex.synchronize do
      sleep 0.1
      puts "count: #{count}"
      count += 1
    end
  end
end

# Spawns 50 threads that all call the proc we just defined
# and then waits for all the threads to finish
50.times.map do
  Thread.new { work.call }
end.to_a.each(&:join)

# Prints the final count after all the threads are done
puts "Final count: #{count}"

If you now run this code from IRB and look at the output you'd see that the count always goes up by one and that the final count is 500

...
count: 487
count: 488
count: 489
count: 490
count: 491
count: 492
count: 493
count: 494
count: 495
count: 496
count: 497
count: 498
count: 499
Final count: 500

But if you remove the mutex, first you'd notice that the code runs much faster, second thing you'd notice is that the output is completely random, sometimes repeats, and change each time you run this

...
count: 487
count: 488
count: 489
count: 490
count: 491
count: 492
count: 493
count: 493
count: 495
count: 496
count: 495
count: 496
count: 494
Final count: 497

This is because, without the mutex, all the threads try to read and update the counter at the same time. Sometimes two threads read the value at the same time - like 493 in the output above - which results in a value being skipped - because both read 493 and updated it to 494.

With the mutex, one thread locks the mutex using the synchronize method, updates the counter ten times, and unlocks the mutex, then another thread lock it again, etc.

That's why with the mutex the code takes longer to complete, each thread runs after the other so the total time is 50 times 10 times 100ms which is 50 seconds. While without the mutex all threads run at the same time so the total time it takes to execute the code is 10 x 100ms or 1 second.

To guarantee that just 1 subscription will measure the latency I can use Mutex#try_lock and Mutex#unlock instead of Mutex#synchronize. Unlike synchronize which blocks and waits until the mutex unlock, try_lock doesn't block and instead returns true if it acquired the lock or false if it didn't. With try_lock I can tell all threads to try an measure the latency, the thread that locks the mutex then performs the measurement and all others return.

MUTEX = Mutex.new

def run_unless_already_running(&block)
  lock_acquired = MUTEX.try_lock
  return unless lock_acquired

  block.call
ensure
  MUTEX.unlock if lock_acquired
end

def collect(payload)
  run_unless_already_running do
    latency = Time.now.to_f - sent_at
  
    Rails.logger.debug "PubSub Latency: #{latency}"
  end
end

stream_from("metrics", proc do |json|
  payload = ActiveSupport::JSON.decode(json)
  collect(payload)
end)

To make things prettier I put all code related to metrics collection into a module

module ActionCable
  module MetricsCollector
    MUTEX = Mutex.new
    STREAM_NAME = "internal.action_cable.metrics"
    COLLECTION_TRESHOLD = 60.seconds

    cattr_accessor :last_collected_at

    class << self
      def measure
        ActionCable.server.broadcast(stream_name, measurement_payload)
      end

      def stream_name = STREAM_NAME

      def measurement_payload
        {
          sent_at: Time.now.to_f
        }
      end

      def collect(payload)
        run_unless_already_running do
          next if measurement_already_collected?

          self.last_collected_at = Time.now
          collect_measurement(payload)
        end
      end

      def measurement_already_collected? = last_collected_at&.after?(collection_trashold.ago)

      def collection_trashold = COLLECTION_TRESHOLD

      private

      def run_unless_already_running(&block)
        lock_acquired = MUTEX.try_lock
        return unless lock_acquired

        block.call
      ensure
        MUTEX.unlock if lock_acquired
      end

      def collect_measurement(payload)
        pub_sub_latency = [0, Time.now.to_f - payload.fetch("sent_at", -Float::INFINITY)].max
        Rails.logger.debug "PubSub latency: #{pub_sub_latency}"
      end
    end
  end
end

Then I changed my cron job to run

bin/rails runner "ActionCable::MetricsCollector.measure"

And added the following to app/channels/application_cable/channel.rb

module ApplicationCable
  class Channel < ActionCable::Channel::Base
    after_subscribe do
      stream_from(
        ::ActionCable::MetricsCollector.stream_name,
        proc do |json|
          payload = ActiveSupport::JSON.decode(json)
          ::ActionCable::MetricsCollector.collect(payload)
        rescue => e
          Rails.logger.error("Failed to collect metrics: #{e}")
        end
      )
    end
  end
end

How do I expose this metric to Elastic APM?

After some thought I decided I wouldn't. I could hack something together, but I'm shoehorning this metric in and later metrics I flat out don't know how to expose in Elastic AMP. So I decided not to bother with this and figure it out later.

Number of connections

Since I'm running an IoT platform I have a somewhat constant number of clients connected at all times. This means that by knowing the number of clients at any given moment I can easily see if there is a problem or if I need to scale out.

Luckily, with the module that I've added to measure the PubSub latency it's easy to expose the number of client. All I have to do is change the collect_measurement method like so

def collect_measurement(payload)
  pub_sub_latency = [0, Time.now.to_f - payload.fetch("sent_at", -Float::INFINITY)].max
  Rails.logger.debug "PubSub latency: #{pub_sub_latency}"

  connection_count = ActionCable.server.connections.length
  Rails.logger.debug "Connection count: #{connection_count}"
end

Without that module I'd probably setup a periodic task like so

module ApplicationCable
  class Channel < ActionCable::Channel::Base
    periodically every: 1.minute do
	  connection_count = ActionCable.server.connections.length
      Rails.logger.debug "Connection count: #{connection_count}"
    end
  end
end

The trick to getting the count is to call ActionCable.server.connections.length from within the server process. If you call it from the Rails console or from a rails runner the count will always return zero as the connections only exist in the server's memory.

Client-Server Latency

Since I'm running my own patch of ActionCable that adds PONG messages in addition to the standard heartbeat PING message, I can easily expose the time it takes for a message to travel from the server over the Internet to a client.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Diagram that shows what Client-Server latency measures

This metric isn't useful for regular web apps as it's non actionable - you can't fix your client's Internet. But it's useful for apps, like mine, where you have control over clients and want to have them connected 24/7.

# Stock Rails 8 doesn't have this notificaiton
ActiveSupport::Notifications.subscribe "client_latency.action_cable" do |_name, _start, _finish, _id, payload|
  Rails.logger.debug("Client latency: #{payload[:value]}")
end

Clients will start disconnecting if this latency exceeds six seconds. If the median latency exceeds a few seconds this indicates a widespread Internet issue (which is more common in the US than I though it would be).

Now that I have all these metrics I have to expose them somehow.

I decided to talk to a coworker from the DevOps team about how they collect and expose metrics from our machines and he told me that they use Prometheus and that he could configure it to collect metrics from ActionCable.

But what is Prometheus?

Prometheus is an open-source monitoring and alerting system - it's a service that collects metrics, allows you to query them, and to configure alerts when they exceed a threshold. It's a bit different from the SaaS monitoring solutions because it scrapes metrics from your application instead of having your application push metrics to it.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Diagram that explains the difference between Prometheus and other APM services

Usually, you'd expose a /metrics endpoint that renders all the metrics the application has collected in a format that Prometheus can parse.

I wasn't sure how to do that at first, but I remember reading an article on Basecamp's dev blog and it mentioned a gem called Yabeda.

Yabeda is a framework for monitoring Ruby applications. Using it you can measure, or report measurements, to various monitoring services like Datadog, NewRelic, Honeybadger, AWS CloudWatch, Prometheus, and many others.

In a nutshell, I can report my measurements to Yabeda and then it handles exposing those metrics to various different monitoring services - like Prometheus - for me.

According to the documentation, before I can use Yabeda I have to configure it which means that I have to define all the metrics that I'm going to expose. Yabeda supports a handful of types of metrics - counters, gauges, and histograms - so let me explain those first.

A counter is a number that can only ever go up by some amount. E.g. the number of ActiveJob jobs processed is a counter because the metric is a number that can only ever increase. But the number of connections to ActionCable isn't a counter since the number goes up when more clients connect and goes down when clients disconnect.

To represent numbers that can go both up and down as a metric we need a gauge. Gauges represent a metric that can be any number at any time.

Finally, a histogram represents a distribution of values. It tells you how many measurements you've got in a certain range called a bucket. Histograms are usually represented by a set of counters and gauges - one counter for each bucket, one counter for the number of measurements, and one gauge for the sum of all measurements. This sounds a bit complicated, but it allows you to calculate the average, median, and other distribution metrics quickly and without having to store every single measurement somewhere. Usually you'd use a histogram to represent time measurements like response times.

Defining these metrics is fairly straight forward

# config/initializers/yabeda.rb

Yabeda.configure do
  group :action_cable do
    histogram :client_server_latency do
      comment "The time it takes for a message to travel via the Internet to a client"
      unit :seconds
      buckets [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    end

    histogram :pubsub_latency do
      comment "The time it takes for a message to travel through the PubSub service"
      unit :seconds
      buckets [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    end

    histogram :action_execution_duration do
      comment "The time it takes to perform an invoked action"
      unit :seconds
      buckets [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    end

    gauge :connection_count, comment: "Number of open WebSocket connections"
  end
end

To report a metric I have to build a chain of method calls consisting of the group names of the metric I want to report, then the metric's name, and finally call measure on that. The measure method accepts either one argument and a block or two arguments. The first argument is a Hash of tags to add to the measurement, I'll explain those later. The second is the measurement itself. Without the second argument, Yabeda will execute the block, measure it's execution time, and report that.

Yabeda.action_cable.client_server_latency.measure({}, 0.123)
# or
Yabeda.action_cable.client_server_latency.measure({}) do
  sleep 0.123 # do some work
end

I can now update all ActiveSupport Notification subscriptions to report to Yabeda, and I can move them all to Yabeda's initializer.

# config/initializers/yabeda.rb

Yabeda.configure do
  group :action_cable do
    histogram :client_server_latency do
      comment "The time it takes for a message to travel via the Internet to a client"
      unit :seconds
      buckets [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    end

    histogram :pub_sub_latency do
      comment "The time it takes for a message to travel through the PubSub service"
      unit :seconds
      buckets [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    end

    histogram :action_execution_duration do
      comment "The time it takes to perform an invoked action"
      unit :seconds
      buckets [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    end

    gauge :connection_count, comment: "Number of open WebSocket connections"
  end
end

ActiveSupport::Notifications.subscribe "perform_action.action_cable" do |event|
  name = "#{event.payload[:channel_class]}##{event.payload[:action]}"
  duration = event.duration / 1000.0 # convert ms to seconds
  
  Yabeda
    .action_cable
    .action_execution_duration
    .measure({ action: action }, duration) # I've used a tag here
end

ActiveSupport::Notifications.subscribe "client_latency.action_cable" do |_name, _start, _finish, _id, payload|
  client_server_latency = payload[:value]
  Yabeda
    .action_cable
    .client_server_latency
    .measure({}, client_server_latency)
end

You might have noticed that I've used a tag when reporting the action execution time. You can think of tags as groups of measurements. Instead of collecting the action execution time of all actions in ActionCable, using tags I'll collect the action execution time of each individual action. That way I can easily pin point which action has the highest duration, but I can still get the average execution time by finding the average of all the tagged measurements.

Without tags I'd have to define dozens of histograms - one for each action. But with tags I can define one histogram and let Yabeda group my measurements for me.

I have to update the metrics collector too

module ActionCable
  module MetricsCollector
    MUTEX = Mutex.new
    STREAM_NAME = "internal.action_cable.metrics"
    COLLECTION_TRESHOLD = 60.seconds

    cattr_accessor :last_collected_at

    class << self
      def measure
        ActionCable.server.broadcast(stream_name, measurement_payload)
      end

      def stream_name = STREAM_NAME

      def measurement_payload
        {
          sent_at: Time.now.to_f
        }
      end

      def collect(payload)
        run_unless_already_running do
          next if measurement_already_collected?

          self.last_collected_at = Time.now
          collect_measurement(payload)
        end
      end

      def measurement_already_collected? = last_collected_at&.after?(collection_trashold.ago)

      def collection_trashold = COLLECTION_TRESHOLD

      private

      def run_unless_already_running(&block)
        lock_acquired = MUTEX.try_lock
        return unless lock_acquired

        block.call
      ensure
        MUTEX.unlock if lock_acquired
      end

      def collect_measurement(payload)
        pub_sub_latency = [0, Time.now.to_f - payload.fetch("sent_at", -Float::INFINITY)].max
        Yabeda.action_cable.pub_sub_latency.measure({}, pub_sub_latency)

		connection_count = ActionCable.server.connections.length
		Yabeda.action_cable.connection_count.set({}, connection_count)
      end
    end
  end
end

Now that my metrics are in Yabeda, I have to somehow expose them to Prometheus. For that I've added the yabeda-prometheus-mmap gem and mounted its route

# config/routes.rb

Rails.application.routes.draw do
  mount Yabeda::Prometheus::Exporter, at: "/metrics"
  # ...
end

There is also yabeda-prometheus, if you are running a single worker server these two are effectively the same. But if you are running Puma with multiple workers, or SolidQueue with multiple workers then the mmap version of the gem will be much faster.

Additionally, when running multiple workers, you probably also want to add "aggregation: :sum" to all gauges. Without that option, if a worker fails and gets restarted, you'll get very wonky metrics because the gauges will get counted twice with the old worker's gauge being stuck at whatever value it was before the worker crashed.

Now, with the route mounted I can connect to my ActionCable server and then open /metrics in a separate browser window to see the output

...
# HELP action_cable_action_execution_duration_seconds Multiprocess metric
# TYPE action_cable_action_execution_duration_seconds histogram
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="+Inf"} 2189
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="0.005"} 101
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="0.01"} 559
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="0.025"} 2044
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="0.05"} 2174
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="0.1"} 2181
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="0.25"} 2189
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="0.5"} 2189
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="1"} 2189
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="10"} 2189
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="2.5"} 2189
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="5"} 2189
action_cable_action_execution_duration_seconds_count{environment="development",action="ChatChannel#post_message"} 2189
action_cable_action_execution_duration_seconds_sum{environment="development",action="ChatChannel#post_message"} 34.04095268249512
...

That's my action execution duration. It works!

My coworker had already configured Prometheus to scrape that endpoint every minute or so and the metrics appeared in Prometheus within a few minutes.

Analyzing the metrics

Now that I had all the metrics somewhere where I could query them I had to make them easy to analyze. I've decided to go with Grafana as we already had an instance to monitor our infrastructure.

Grafana allowed me to take this set of metrics and turn them into graphs that are easy to parse. First, I exposed the number of connections. The graph now clearly shows how many clients were connected at each minute.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The number of connections to ActionCable plotted over time

Next, I plotted the action execution latency. This is a bit more involved just because I wanted to plot the average, median, 95th percentile and 99th percentile.

The average is important for scaling as it's used in Little's law to determine the number of workers necessary to handle the load. The median is a better representation of what most clients experience as it represents the latency of 50% of all clients. The 95th and 99th percentile represent the worst cases - the latency experienced by 95% and 99% of all clients.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The action execution latency of all actions plotted over time

In addition to plotting the overall latency across all actions I've also plotted the latency per action as a heat-map. Here each rectangle represents 1 minute, red rectangles represent high latency and blue represents low latency. If I see a bright red action here I know that it's causing a problem.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A heatmap that breaks down the action execution latency by action over time

I've plotted the client-server latency and PubSub latency the same way as the overall action execution duration.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Client-Server latency plotted over time

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

PubSub latency plotted over time

With all these metrics in Grafana I could now focus on finding and fixing the root cause. As the incident had passed and I didn't have any metrics from it so I'd have to wait for another one.

Step six, fixing the root cause.

The app was now stable so I tried a few things that I guessed could be the cause while I waited to see if this would occur again.

Then, after two weeks, the incident happened again!

I quickly opened Grafana and saw that the PubSub latency was extremely high.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

PubSub latency during the incident

(the 95th and 99th percentile are capped at 10s because that's the largest bucket I've configured)

This means that Redis is the culprit. For some reason It takes ages to deliver messages. As a quick fix, I opened the AWS console and scaled up our Redis instance. Within a few minutes the incident cleared up.

I spent the next week investigating ElastiCache Redis and confirmed that it was the root cause of both incidents. I'll go into more detail on this in a subsequent post, but the gist of it is that AWS is a piece of crap.

Looking back

I would never have found and fixed this without the monitoring that I've set up.

In the following month I've exposed more metrics like the broadcast and transmission latency, and the number of confirmed and rejected subscriptions.

ActiveSupport::Notifications.subscribe "broadcast.action_cable" do |_name, start, finish, _id, _payload|
  Yabeda.action_cable.broadcast_duration.measure({}, finish.to_f - start.to_f)
end

ActiveSupport::Notifications.subscribe "transmit.action_cable" do |_name, start, finish, _id, _payload|
  Yabeda.action_cable.transmission_duration.measure({}, finish.to_f - start.to_f)
end

ActiveSupport::Notifications.subscribe "transmit_subscription_confirmation.action_cable" do |_name, start, finish, _id, _payload|
  Yabeda.action_cable.transmission_duration.measure({}, finish.to_f - start.to_f)
  Yabeda.action_cable.confirmed_subscription_count.increment({}, by: 1)
end

ActiveSupport::Notifications.subscribe "transmit_subscription_rejection.action_cable" do |_name, start, finish, _id, _payload|
  Yabeda.action_cable.transmission_duration.measure({}, finish.to_f - start.to_f)
  Yabeda.action_cable.rejected_subscription_count.increment({}, by: 1)
end

I don't really need those metrics now, but through this incident I've learned that it's better to have a metric before you need it so I exposed everything I could expose with ActiveSupport Notifications.

Additionally, I've added a few more gems - like yabeda-puma-plugin, yabeda-rails and yabeda-activejob - that expose various metrics to help me pinpoint problems in the future.

Software is made by people

2024-06-03T05:05:00Z

The recent Hotwire pop-up drama has shown me that some have forgotten that software is made by people - people with different backgrounds, different values, different skills, and different flaws.

The diversity in opinions and aproaches is what makes modern day software development great. That’s what drives innovation.

There are many tribes of people that like to build software in a certain way, that like to use a certain language, that approach problems a certain way, that prefere certan conventions over others. It’s easy to find a tribe and join it, or to start your own.

When I started programming professionally I’ve aleardy found my tribes and thought that people in other tribes - especially those with diametrically opposite beliefs - were just doing software wrong.

But I was wrong back then.

Today I understand that all tribes in our community formed not because we interpreted something differently, but because we have different values about the same thing - software.

We are like visitors at an art gallery looking at an abstract sculpture - each of us sees something different even though we are looking at the same thing.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Optical illusion that sometimes looks like a rabbit and sometimes like a duck (well, bird)

Where one person sees a rabbit another will see a duck. There is no one right answer. If it’s a rabbit, or a duck, or both, or neither is dependent on you.

HTML-over-the-wire and SPAs, in this context, are two ways of doing the same thing - rendering content in a reactive way. There is no one right answer as to what the best way to do it is. Which one you prefere is up to you.

Why does Rails put the type column first in an index for a polymorphic association?

2024-04-01T12:00:00Z

Last week, I had a discussion with a coworker about how Rails indexes columns used in polymorphic associations. He thought that the order of columns in the index should be flipped - instead of indexing by type and ID, it should index by ID and type - as that way the most restrictive column is first, and therefore the index is more efficient. While I argued that the way that Rails indexes polymorphic associations is very pragmatic while also being efficient.

First off, what are we talking about?

We have an Access Log model in our app. It holds a log of who granted access to which device and when. But the thing granting access can be a Person, another Device or a Schedule.

In Rails, an association that can point to many different types of models is called a "polymorphic associations".

class AccessLog < ApplicationRecord
  belongs_to :device
  belongs_to :grantor, polymorphic: true
end

# Since grantor is polymorphic we can assign any type of model to it
log = AccessLog.create!(device: Device.all.sample, grantor: Person.all.sample)
log.grantor.class # => Person

log.update!(grantor: Device.all.sample)
log.grantor.class # => Device

log.update!(grantor: Schedule.all.sample)
log.grantor.class # => Schedule

# Since the device association isn't polimorphic we can't assign anything
# but a device to it
log.update!(device: Person.all.sample) # => ActiveRecord::AssociationTypeMismatch: Device expected, got Person

A polymorphic and a regular association are implemented differently in the database.

A regular association is just a column that holds an ID that points to a record in the other model's table; usually it also has a foreign key constraint, which ensures that the assigned ID actually exists in the other table.

While a polymorphic association uses two columns, one to hold the type of model it's for and another to hold the ID of the record it's for.

create_table "access_logs", force: :cascade do |t|
  # Regular association - holds the ID of the record from the devices table
  t.bigint "device_id", null: false
  
  # Polymorphic association
  t.string "grantor_type", null: false # holds the model name
  t.bigint "grantor_id", null: false # holds the ID of the record
  
  t.index ["device_id"], name: "index_access_logs_on_device_id"
  t.index ["grantor_type", "grantor_id"], name: "index_access_logs_on_entry"
end

# ensures that every ID in device_id coresponds to an id in the devices table
add_foreign_key "access_logs", "devices"

When you generate an association, Rails will automatically create an index for it. In the case of a polymorphic association, it will create a composite index - that's an index that indexes two or more columns simultaneously - in which it will put the type column first and the ID column last.

Why does it matter which column is first in the index?

This has to do with how databases build indices. By default, MySQL, MariaDB and PostgreSQL build b-tree indices.

An index is similar to a regular book index, but instead of having chapter names and page numbers, it has values from the indexed column and internal record IDs.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Comparison between a book and a naive database index

The problem with this kind of index is that it will grow as the number of distinct values in the indexed column grows.

E.g. if you create an index on a column that holds a person's email address, this kind of index would have one entry for each email, and the database would have to scan through each email until it finds the right one. To speed things up, we can do what dictionaries usually do and create a multi-level index.

In a dictionary, you'd usually have an index that points you to another index. E.g. the main index would tell you that the index for all words starting with R is on page 6. Then the R sub-index would tell you that words starting with RU are on pages 342 to 361. Then you'd go to page 342 and search for the word "Ruby" until you reached page 361.

= 768px) 520px, 50vw"> = 768px) 520px, 50vw">

A multi-level index from a dictionary

= 768px) 520px, 50vw"> = 768px) 520px, 50vw">

A naive multi-level index in a database

B-trees are similar to multi-level indices in dictionaries, though they have extra rules and nuances that aren't important now.

What's important is that a b-tree index can have a sub-index, which in term can have a sub-index, and so on. And the more items each sub-index has, the slower it is to find something.

These indices work for single values such as words and emails, but how would you make an index of two words, or an email and a number?

To create a composite index, most databases just concatenate the values together and build a regular b-tree index. This is a gross simplification of what actually happens, but in essence, that's what they do. Because of that, the order of the columns becomes important.

If you tell the database to build an index over grantor_type first and then grantor_id it would take the two and concatenate them together like so "Person|443". But if you tell it to build an index over grantor_id first and then grantor_type it would concatenate them like so "443|Person".

The order of the columns in the index changes the structure of the index.

When the ID column is first, the index points to fewer rows, so the database has to scan through fewer rows, and therefore it can find a match quicker. That's why it's a general recommendation to put the most specific column first in a composite index - it narrows down the search a lot.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Naive composite database index with the ID column first and type column second

When the type column is first, there are a lot more rows to scan through in the end.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Naive composite database index with the type column first and ID column second

Why would anyone create an index and put the type column first then?

You don't have to follow a multi-level index all the way to the last index entry to find something. E.g. if you are searching for all words that start with R, you can go from the main index to find all sub-indices of words starting with R. Take the start page of the first index and the last page of the last index. Now you know that all words between these two pages start with R, and you didn't have to scan through anything.

In a real app, you'd sometimes want to filter all records that are associated with another record of a certain type. E.g. you might want to fetch all Access Logs that were granted by a Schedule, or all People associated with a Property through an Access Log.

class Property < ApplicationRecord
  has_many :access_logs
  
  has_many :schedule_activations,
    -> { where(grantor_type: "Schedule") },
    class_name: "AccessLog"
    
  has_many :active_people,
    through: :access_logs,
    source: :grantor,
    source_type: "Person"
end

# Calling
Propert.find(42).schedule_activations.count
# Executes the following SQL statement
# ```sql
# SELECT COUNT(*)
# FROM access_logs
# WHERE property_id = 42 
#   AND grantor_type = 'Schedule'
# ```

# Calling
Propert.find(42).active_people.count
# Executes the following SQL statement
# ```sql
# SELECT COUNT(tenants.*)
# FROM tenants INNER JOIN access_logs
#   ON tenants.id = access_logs.grantor_id
# WHERE access_logs.property_id = 42
#   AND access_logs.grantor_type = 'Person'

In that case, the composite index over grantor_id first and then grantor_type is basically useless. The database would have to traverse the whole index to aggregate a range that it then has to scan through.

While the index over grantor_type first and then grantor_id can quickly give you the range that you have to scan through by looking at the first and last entry of a sub-index.

We can test this out.

I created a test MariaDB database, with an access_logs table that I've filled with a million rows. Then I created a composite index over grantor_type and grantor_id that I've named test_index.

When I ask MariaDB to explain how it would go about counting all rows where the grantor_type is Person, it says that it would use test_index to generate the count.

MariaDB [test]> CREATE INDEX test_index ON access_logs (grantor_type, grantor_id);
Query OK, 0 rows affected (1.209 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [test]> EXPLAIN  SELECT COUNT(1) FROM access_logs WHERE grantor_type = 'Person';                                                 
+------+-------------+---------------+------+---------------+------------+---------+-------+--------+--------------------------+
| id   | select_type | table         | type | possible_keys | key        | key_len | ref   | rows   | Extra                    |
+------+-------------+---------------+------+---------------+------------+---------+-------+--------+--------------------------+
|    1 | SIMPLE      | access_logs   | ref  | test_index    | test_index | 183     | const | 498290 | Using where; Using index |
+------+-------------+---------------+------+---------------+------------+---------+-------+--------+--------------------------+
1 row in set (0.002 sec)

MariaDB [test]> SELECT COUNT(1) FROM access_logs WHERE grantor_type = 'Person';
+----------+
| COUNT(1) |
+----------+
|   333400 |
+----------+
1 row in set (0.042 sec)

If I ask the same question, but change the order of columns in the index, then MariaDB says that it can't use any index to generate the count but it still tries to use test_index, and it takes about twice as long to count the rows.

MariaDB [test]> DROP INDEX test_index ON access_logs;
Query OK, 0 rows affected (0.045 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [test]> CREATE INDEX test_index ON access_logs (grantor_id, grantor_type);
Query OK, 0 rows affected (1.146 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [test]> EXPLAIN  SELECT COUNT(1) FROM access_logs WHERE grantor_type = 'Person';
+------+-------------+---------------+-------+---------------+------------+---------+------+--------+--------------------------+
| id   | select_type | table         | type  | possible_keys | key        | key_len | ref  | rows   | Extra                    |
+------+-------------+---------------+-------+---------------+------------+---------+------+--------+--------------------------+
|    1 | SIMPLE      | access_logs   | index | NULL          | test_index | 192     | NULL | 996580 | Using where; Using index |
+------+-------------+---------------+-------+---------------+------------+---------+------+--------+--------------------------+
1 row in set (0.000 sec)

MariaDB [test]> SELECT COUNT(1) FROM access_logs WHERE grantor_type = 'Person';
+----------+
| COUNT(1) |
+----------+
|   333400 |
+----------+
1 row in set (0.095 sec)

These results will vary from database to database, but the principle still stands.

For example, PostgreSQL can sometimes utilize an index even when it isn't optimal. In the case of an index for a polymorphic association, changing the order of the columns has much less impact than in MariaDB. In both cases, PostgreSQL uses the index and produces a result in about the same time.

postgres=# CREATE INDEX test_index ON access_logs (grantor_type, grantor_id);
CREATE INDEX

postgres=# EXPLAIN ANALYZE SELECT COUNT(1) FROM access_logs WHERE grantor_type = 'Person';                                                                      
QUERY PLAN                                                                       -----------------------------------------------------------------------------------------------------------------------------------------------------------------------                                                                            Finalize Aggregate  (cost=6457.79..6457.80 rows=1 width=8) (actual time=12.408..13.780 rows=1 loops=1)                                                ->  Gather  (cost=6457.57..6457.78 rows=2 width=8) (actual time=12.403..13.777 rows=3 loops=1)
    Workers Planned: 2
	Workers Launched: 2
	->  Partial Aggregate  (cost=5457.57..5457.58 rows=1 width=8) (actual time=10.314..10.314 rows=1 loops=3)
	  ->  Parallel Index Only Scan using test_index on access_logs  (cost=0.42..5107.64 rows=139972 width=0) (actual time=0.027..6.566 rows=111133 loops=3)
	    Index Cond: (grantor_type = 'Person'::text)                                      Heap Fetches: 0
Planning Time: 0.145 ms
Execution Time: 13.801 ms
(10 rows)

postgres=# DROP INDEX test_index;
DROP INDEX

postgres=# CREATE INDEX test_index ON door_releases (grantor_id, grantor_type);
CREATE INDEX

postgres=# EXPLAIN ANALYZE SELECT COUNT(1) FROM door_releases WHERE grantor_type = 'Person';
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Finalize Aggregate  (cost=13778.29..13778.30 rows=1 width=8) (actual time=14.570..15.922 rows=1 loops=1)
   ->  Gather  (cost=13778.08..13778.29 rows=2 width=8) (actual time=14.566..15.919 rows=3 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         ->  Partial Aggregate  (cost=12778.08..12778.09 rows=1 width=8) (actual time=12.210..12.210 rows=1 loops=3)
               ->  Parallel Index Only Scan using test_index on door_releases  (cost=0.42..12428.15 rows=139972 width=0) (actual time=0.037..8.808 rows=111133 loops=3)
                     Index Cond: (grantor_type = 'Person'::text)
                     Heap Fetches: 0
 Planning Time: 0.079 ms
 Execution Time: 15.944 ms
(10 rows)

Simply put.

The type column is first in an index for a polymorphic association because then the same index can be used to speed up association look-ups (querying by both type and ID columns) and to filter by the type of the associated record (querying by just the type column).

These are the two most common use cases that you'll encounter in a Rails app. And also the use-cases for which Rails has built-in interfaces for (e.g. the source_type option).

This is more efficient in terms of speed (index creation and updating) and space (the size of the index on disk) than having two or more separate indices, which would be required in some databases if the columns were flipped.

Running Campfire behind traefik

2024-02-06T07:10:00Z

I rent a bare metal server from Scaleway where I host all my apps. For me, that's much more economical than running multiple small VPS boxes. 50€ a month buys me 2TB of disk space, 64GB of RAM and an 4 core / 8 thread CPU on which I can host two or three dozen apps. Compare that to AWS or DigitalOcean where for the same money I get about one quarter of those resources.

All my apps are deployed as Docker containers, most of them using Kamal, in front of which I put traefik as a reverse proxy to direct traffic and provide HTTPS via Let's Encrypt.

Last weekend I wanted to setup a Campfire instance for Ruby Zagreb - my local Ruby user-group - on that machine.

So I ran the official installer and got an error.

#!/usr/bin/bash
sudo once setup 1111-1111-1111-1111 

 ██████╗ ███╗   ██╗ ██████╗███████╗
██╔═══██╗████╗  ██║██╔════╝██╔════╝
██║   ██║██╔██╗ ██║██║     █████╗  
██║   ██║██║╚██╗██║██║     ██╔══╝  
╚██████╔╝██║ ╚████║╚██████╗███████╗
 ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝
                            
Failed to start the software
                            
                                                                                                                                                                                                                                              
The software couldn't be started. Please try again.

I expected the installer to fail since it can't know my setup, and it didn't ask me anything besides the domain of the instance.

Sadly, the email you get with your purchase doesn't explain what the installer does, and I can't check what it's doing since the installer is a binary blob. So I had to snoop around for a bit.

The first thing I found was the installer's error log which said that the container couldn't bind to port 443.

cat .once-error.log
2024/01/27 19:33:11 Error response from daemon: driver failed programming external connectivity on endpoint campfire (e32f1a334f8ac25b9f3df0c283adb3eb3e8a554074f0827d17f8b566d03940a5): Bind for 0.0.0.0:443 failed: port is already allocated

Since traefik is bound to port 80 and 443 this makes perfect sense.

The error looks like it came from Docker so I checked my images, and sure enough there was a new "registry.once.com/campfire:latest" image and a stopped container which I promptly inspected.

Within the container runs a custom proxy called thruster that terminates SSL and handles the sendfile protocol. That's what's bound to port 443 and 80 of the container.

Inside the container is also a Redis instance, which implies that the installer sets at least "sysctl vm.overcommit_memory=1" else Redis wouldn't start.

I also saw that there were ENV variables injected into the container so I looked where those came from and found a config file in "/root/.config/once/config.json".

sudo cat /root/.config/once/config.json | jq
{
  "token": "1111-1111-1111-1111",
  "product": "campfire",
  "product_name": "Campfire",
  "email_address": "phony.email@example.com",
  "ssl_domain": "chat.rubyzg.org",
  "validation_token": "PHONY_VALIDATION_KEY",
  "secret_key_base": "PHONY_SECRET_KEY_BASE",
  "vapid_private_key": "PHONY_PRIVATE_KEY",
  "vapid_public_key": "PHONY_PUBLIC_KEY",
  "storage_location": "/var/once/campfire",
  "cron_hour": 2,
  "once_binary_etag": ""
}

Seems pretty straight forward. This is roughly what the installer does

sudo sysctl vm.overcommit_memory=1

docker run \
	-d \
	--name campfire \
	--restart unless-stopped \
	--env-file campfire/.env \
	-p 443
	-p 80
	-v campfire/storage:/rails/storage \
	registry.once.com/campfire:latest \
	bin/boot;;

Where "campfire/.env" contains the values from "/root/.config/once/config.json" plus a few extras.

# campfire/.env 
SECRET_KEY_BASE=PHONY_SECRET_KEY_BASE
VAPID_PRIVATE_KEY=PHONY_PRIVATE_KEY
VAPID_PUBLIC_KEY=PHONY_PUBLIC_KEY
SSL_DOMAIN=chat.rubyzg.org
DISABLE_SSL=YES

To run Campfire without the installer, behind traefik, all I had to do is

sudo sysctl vm.overcommit_memory=1

docker run \
	-d \
	--name campfire \
	--network apps \
	--restart unless-stopped \
	--env-file campfire/.env \
	--label-file campfire/labels \
	-v campfire/Procfile:/rails/Procfile \
	-v campfire/production.rb:/rails/config/environments/production.rb \
	-v campfire/storage:/rails/storage \
	registry.once.com/campfire:latest \
	bin/boot;;

Put this into "campfire/labels"

# Enable traefik for the container 
traefik.enable=true 

# Convert HTTP traffic to HTTPS 
traefik.http.routers.campfire.entrypoints=websecure
traefik.http.routers.campfire.tls.certresolver=letsencrypt

# Register the container to a domain - CHANGE THIS TO YOUR DOMAIN 
traefik.http.routers.campfire.rule=Host(`chat.rubyzg.org`) 

# Return a service not available screen if Campfire isn't running 
traefik.http.middlewares.campfire.retry.attempts=10 
traefik.http.middlewares.campfire.retry.initialinterval=1s 
traefik.http.services.campfire.loadbalancer.healthcheck.path=/up 
traefik.http.services.campfire.loadbalancer.server.port=80 
traefik.http.services.campfire.loadbalancer.server.scheme=http

Finally, I had to disable thruster and bind Puma to port 80 by updating the "web" entry in the Procfile (you get the source files in the purchase email)

# campfire/Procfile 
web: bin/rails server -b 0.0.0.0 -p 80 
# ...

This will boot the server, it will allow you to setup an admin account, create rooms, add a logo, but you won't be able to post messages because Action Cable can't connect to the server.

Traefik has a bug where it doesn't forward x-forwarded-for headers for WebSocket handshakes.

To fix that I had to update "production.rb" with the exact URL that Action Cable would get requests from

# production.rb
# ...
config.action_cable.url = "wss://#{ENV.fetch("SSL_DOMAIN")}/cable"
config.action_cable.allowed_request_origins = ["https://#{ENV.fetch("SSL_DOMAIN")}"]
# ...

That's it.

It's isn't as straight forward as I'd like it to be, and it would be nice if Kamal and Campfire could play along nicely out of the box. I'd bet there is a large overlap between those communities, and both are 37signals projects.

Deconstructing Action Cable

2024-02-05T13:00:00Z

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Deconstructing Action Cable

I knew what Action Cable was about and roughly how it worked since it was released. But lately I've developed a much deeper understanding of it because I had to extend it and explain its internals to others.

The following is how I explained Action Cable to myself. I took a top-down approach, starting at what I could observe Action Cable doing in my browser and following the code until I found the implementation, which I then deconstructed to understand why it does what it does.

This was a along journey down a deep rabbit hole, but it was worth it. At the end I now understand how and why Action Cable does what it does, and appreciate how simple and elegant it actually is.

From HTTP to WebSocket

Action Cable is a framework that allows your server to push messages to the browser, or any other client.

Usually your server can only respond to requests that a client, like your browser, makes to it. But Action Cable opens a WebSocket between the client and the server through which the server can push messages to the client at any time.

The client can also push messages to the server through that WebSocket. But that's in my opinion less interesting as we could do that without Action Cable.

How does the client connect to Action Cable's WebSocket?

When anyone or anything wants to connect to your Rails app via Action Cable they first have to make a HTTP request to "/cable" and after a bit of back and forth out pops a WebSocket.

But how does a HTTP request become a Websocket?

Back in 1997 HTTP version 1.1 was released. It's very similar to HTTP 1.0 with a few additions, the one that's important for WebSockets is the addition of status 101 aka. Switching protocols and the "Upgrade" header. The intent of status 101 was to enable browsers to gracefully transition from HTTP 1.x to an eventual HTTP 2 or some other protocol that would be incompatible with HTTP 1.

And that's exactly what a browser does when it connects to a WebSocket. It first makes a request with the "Upgrade" header set to "websocket" which tells the server "hey, I want to switch to using WebSockets for this request". If the server supports WebSockets it responds with status 101 indicating that the upgrade was accepted and that, from now on, it will communicate using the WebSocket protocol on that request's connection.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A WebSocket upgrade request

But why do we need the HTTP request in the first place? Can't we just connect directly via the WebSocket protocol since we already know that the server supports it?

The genius of WebSockets lies in the initial HTTP request. Because of that initial HTTP request we have access to all the browser's cookies that are sent over, so we can use them to figure out who wants to open a WebSocket, if they are allowed to do that, and we can remember who the socket is for.

So Action Cable converts HTTP requests to WebSockets?

Yes! It's a WebSocket server, but it also does so much more.

Connections

So someone's connected to our WebSocket, but how do we know who we are talking to and how can we send messages to them?

In Action Cable, each open WebSocket has a corresponding Connection object. The connection object is responsible for keeping track of who the WebSocket is for and figuring out if the client is allowed to open a WebSocket.

When a client makes the initial HTTP request to "/cable", Action Cable will take the request's headers, cookies, URL and create a Connection object from it.

If it exists, the Connection object will be an instance of "ApplicationCable::Connection". If that class doesn't exist it will be an instance of "ActionCable::Connection::Base". You can change the class of the connection by setting "config.action_cable.connection_class = ->{ WhateverClassIWant }".

When a connection is established the "#connect" method on our "ApplicationCable::Connection" object will get called. In that method you'd usually want to figure out who opened the connection, and reject it if they aren't allowed to open it.

module ApplicationCable
  class Connection < ActionCable::Connection::Base
    attr_accessor :current_person
    
    def connect
	  Rails.logger.debug "Someone wants to connect via WebSocket!"
	  set_current_person
	  
	  unless current_person
	    Rails.logger.debug "I don't know who this is. Closing the WebSocket."
	    reject_unauthorized_connection
	  end
	  
	  Rails.logger.debug "It's, #{current_person.name}"
    end

    private

    def set_current_person
      person = 
	    # We can access the param of the HTTP request
	    Person.find_by(token: request.params[:token]) ||
	    # We can access the session of the HTTP request
        Person.find_by(token: request.session[:person_id])
        
      unless person
        # We can access the cookies of the HTTP request
        session = Session.find_by(id: cookies.encrypted[:session_id])
        person = session&.person
      end
	  
	  self.current_person = person
    end
  end
end

There is also a "#disconnect" method that gets called when a WebSocket is closed. It's useful if you want to set something up in "#connect" and then tear it down when the WebSocket closes. E.g. if you'd want to count the number of open connections that a person has, you could do something like this.

module ApplicationCable
  class Connection < ActionCable::Connection::Base
    attr_accessor :current_person
    
    def connect
	  set_current_person
	  reject_unauthorized_connection unless current_person

	  current_person.increment!(:connection_count)
    end

    def disconnect
      current_person&.decrement!(:connection_count)
    end

    private

    # ...
  end
end

Now you know who the WebSocket connection is for, but Rails doesn't. To tell Rails which attr_accessor(s) we'll use for identification we have to change our "attr_accessor" to "identified_by", which is a class method that Action Cable provides.

module ApplicationCable
  class Connection < ActionCable::Connection::Base
    #attr_accessor :current_person # CHANGE THIS
    identified_by :current_person # TO THIS
    
    def connect
	  set_current_person
	  reject_unauthorized_connection unless current_person
    end

    private

    # ...
  end
end

Ok, now we've got a connection and we know who it's for, how do we send a message? Do we have to send some kind of JSON or XML or whatnot?

The Wild WebSocket West

We are used to HTTP in our Rails apps. With it we can transfer text, images, audio, video and more from our server to the browser. But WebSockets are the wild west of the Web.

As the "socket" part implies, a WebSocket is similar to a Unix, TCP or UDP socket in that you can transfer whatever you like over it and in any fashion you'd like. It's up to you to implement a protocol between your server and the browser on top of your WebSocket.

Luckily we don't have to implement anything as Action Cable is not only a WebSocket server but also a protocol.

When you send a message to a client - or vice versa - the the object you are sending will first be turned into a JSON-compatible object like a String, a Number, a Hash, or an Array. Then it will be wrapped inside a Hash under a key called "message" together with some additional things I'll explain in a bit. Then that Hash will be dumped as JSON and sent as an UTF-8 encoded String, byte for byte through the WebSocket. On the other end the message is assembled into a String again and parsed into an object.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Raw message sent via ActionCable to the browser

Why do we need the wrapper Hash?

The wrapper Hash allows us to have different types of messages. The protocol has three types of messages that a client can send to the server:

subscribe
unsubscribe
message

And six kinds of messages that the server can send to the client:

welcome
disconnect
ping
confirm_subscription
reject_subscription
message

The "welcome" message is the simplest of them all. It's just a Hash with a key "type" that holds the value "welcome". All messages sent from the server to a client have a "type" field. This is always the first message that a server sends to any client that connects to it. It's used to indicate to the client that the WebSocket is functional.

{
  "type": "welcome"
}

The "ping" message is used to keep a heartbeat. It's sent to all connected clients every 3 seconds so that the connection doesn't become stale and therefore terminated by some load balance in the middle. And it enables clients to detect when they have disconnected from the server. If the client doesn't receive a heartbeat message within 6 seconds - that's 2 heartbeats - it will try to reconnect. In addition to the "type" field, a ping message will also have a "message" field which holds the Unix timestamp of the moment the message was sent. This timestamp isn't used in the official client implementation.

{
  "type": "ping",
  "message": 1705848059
}

The "disconnect" message instructs a client to disconnect from the server. In addition to the "type" field it has a "reconnect" field which holds a Boolean. If the value of this field is true, the client should immediately try to reconnect after it disconnects. This is useful in case an error occurs and the connection has to be restarted. And in addition to the "reconnect" field there is a "reason" field which holds a String that gives you a brief explanation why your client is being disconnected.

There are four possible disconnect reasons:

unauthorized - sent when "reject_unauthorized_connection" is called
invalid_request - sent when the initial HTTP request was malformed
server_restart - sent when the server is about to restart
remote - send when the client is kicked for whatever reason

{
  "type": "disconnect",
  "reconnect": true,
  "reason": "server_restart"
}

To explain the "message", "subscribe", "confirm_subscription", "reject_subscription" & "unsubscribe" messages I first have to explain Channels.

Channels

What are Channels?

Channels are Action Cable's controllers.

In a regular HTTP request you'd specify which method you want to call on which Controller by setting the request's path and method. E.g. if you'd want to call the "create" method of the "ArticlesController" you'd make a POST request to "/articles" and pass any parameters you'd like the new Article object to have.

class ArticlesController
  # POST /articles
  def create
    if Article.create(params.require(:article).permit(:title, :content))
      redirect_to action: :show, status: :see_other
    else
      render :new, status: :unprocesssable_entity
    end
  end
end

In Action Cable you specify which method you want to call on which Channel. To do that, you first have to subscribe to to a Channel and then you have to send a message to it.

class ChatChannel < ApplicationCable::Channel
  def post_message(data)
	  Chat::Message.create(
	    content: data[:content], 
	    poster: current_person
	  )
  end
end

(Note that we get access to "current_person" because we used "identified_by" in our Connection object)

Now if you want to invoke "ChatChannel#post_message" you first have to subscribe to the "ChatChannel" with a "subscribe" message.

The "subscribe" message has a "command" field instead of a "type" field. All messages sent from the client to the server have a "command" field. In addition to that it also has an "identifier" field which holds a JSON encoded Hash. The identifier Hash has at least a "channel" field which is the class name of the channel. But it can also have other, user defined, values which will become available in the Channel object through the "params" method.

{
  "command": "subscribe",
  "identifier": "{ 
    \"channel\": \"ChatChannel\", 
    \"room_name\": \"Ruby Zagreb\" 
  }"
}

These params can be used however you'd like. E.g.

class ChatChannel < ApplicationCable::Channel
  def post_message(data)
	  current_person
	    .chat_rooms
	    .find_by(name: params[:room_name]) # PARAMS FROM THE SUBSCRIBE MESSAGE
	    &.messages
	    &.create(content: data[:content])
  end
end

When someone subscribes to a Channel the "subscribed" method on the channel will be called. In it you can do various things, one of which is to authorize the subscription. Let's say that you want to reject subscriptions for chat rooms that a person isn't a member of, you could do something like this

class ChatChannel < ApplicationCable::Channel
  def subscribed
    @chat_room = current_person
	  .chat_rooms
	  .find_by(name: params[:room_name])
	  
	reject unless @chat_room
  end
  
  def post_message(data)
	@chat_room.messages.create(content: data[:content])
  end
end

If the "subscribed" method doesn't call "reject" the server will respond with a "confirm_subscription" message.

The "confirm_subscription" message has a "type", and an "identifier" field. The identifier holds the same value that was sent in the original "subscribe" message.

{
  "type": "confirm_subscription",
  "identifier": "{ 
    \"channel\": \"ChatChannel\", 
    \"room_name\": \"Ruby Zagreb\" 
  }"
}

If "reject" was called then the server will respond with a "reject_subscription" message.

The "reject_subscription" message has the same format as the "confirm_subscription" message.

{
  "type": "reject_subscription",
  "identifier": "{ 
    \"channel\": \"ChatChannel\", 
    \"room_name\": \"Ruby Zagreb\" 
  }"
}

Now that the client is subscribed they can invoke an action on the Channel using the regular message type.

The "message" type messages has two variants - one for messages sent from the server to the client, and another for messages sent from the client to the server.

If the client is sending the message, then the message will contain a "command" field with the value "message", an "identifier" and a "data" field. Again, the identifier holds the same value that was sent in the original "subscribe" message. While data can be anything depending on what the client has sent.

{
  "command": "message",
  "identifier": "{ 
    \"channel\": \"ChatChannel\", 
    \"room_name\": \"Ruby Zagreb\" 
  }",
  "data": "Hello, Zagreb!"
}

When the channel receives a message, it will check if it implements a "receive" method and if it does it will invoke it and pass the message's data to it.

Though, if the client sent over a Hash, then the Channel will check if the data Hash contains the key "action" with a String value. If it does, and the Channel implements a method with the same name, then it will be invoked to process the message's data.

{
  "command": "message",
  "identifier": "{ 
    \"channel\": \"ChatChannel\", 
    \"room_name\": \"Ruby Zagreb\" 
  }",
  "data": {
    "action": "post_message",
    "content": "Hello, Zagreb!"
  }
}

I'll explain the server-to-client message format in a moment.

Now we know how to send messages from the client to Action Cable. But how do we send messages from Action Cable to the client?

This is in my opinion the more interesting part. To send something we first have to create a stream to which we can publish messages to.

A stream is a PubSub channel. I guess it's called a stream in Action Cable because, semantically, this PubSub channel acts like a stream of messages for the client.

What is a PubSub channel?

PubSub (short for Publish-subscribe) is a common programming pattern for sending messages between objects. An object can subscribe to messages from, or publish messages to, a channel (sometimes called a topic). When one object publishes a message to a channel, all objects subscribed to that channel receive it. The message can be anything - a String, a Hash, a Number, or another Object.

In Action Cable, our Channel object is the subscriber and our application (controllers, models, jobs, ...) is the publisher.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Illustration of how PubSub works

To create a stream we can use either "stream_from" which creates a PubSub channel using a String identifier that we pass to it (e.g. "chat:1337"). Or “stream_for” which accepts an object (e.g. "Chat.find(1337)"). "stream_for" internally generates a String identifier for the given object and calls "stream_from" with it.

class ChatChannel < ApplicationCable::Channel
  def subscribed
    @chat_room = current_person
	  .chat_rooms
	  .find_by(name: params[:room_name])
	  
	reject unless @chat_room

    # creates a PubSub channel to which we can publish messages to from anywhere
    stream_for @chat_room
    # the above is basicaly the same as
    # stream_from "chat_room:#{@chat_room.id}"
  end
end

Now that we have a stream we can publish messages to it from anywhere using "broadcast_to" which accepts two arguments - the object we are broadcasting to, and the messages to broadcast.

ChatChannel.broacast_to(
  ChatRoom.find(params[:id]), # The same record that we gave to stream_for
  "Hello, Zagreb!" # The message I want to send
)

If you used "stream_from" to create your stream, then you have to use "ActionCable.server.broadcast" instead of "broadcast_to". It also expects two arguments - the stream we are broadcasting to, and the message we are broadcasting.

ActionCable.server.broadcast("chat_room:123", "Hello, Zagreb!")

You can also send a message directly from the Channel object using the "transmit" method.

class ChatChannel < ApplicationCable::Channel
  def subscribed
    @chat_room = current_person
	  .chat_rooms
	  .find_by(name: params[:room_name])
	  
	reject unless @chat_room

    transmit "Welcome back, #{current_person.first_name}!"
  end
end

Ok, but what happens when we broadcast or transmit a message?

Well, it's wrapped in a hash containing two keys - "identifier" and "message". The identifier holds the same value that was sent in the original "subscribe" message. While the message holds whatever you are sending to the client.

{
  "identifier": "{ 
    \"channel\": \"ChatChannel\", 
    \"room_name\": \"Ruby Zagreb\" 
  }",
  "message": "Hello, Zagreb!"
}

Then that Hash is turned into JSON and sent via the WebSocket.

When the client gets that message, it can figure out for which of its subscriptions it's for based on the "identifier", and then it can process the "message" however it likes.

In the browser

Now we have sent a message from the server, but how can we respond to it in a browser?

Action Cable ships with an official JavaScript client that enables you to connect to a server via Action Cable, subscribe to any channel, and receive messages. Just like its server counterpart, it hides the protocol from you and allows you to focus only on the messages.

To connect to a server you have to create a consumer, which is a wrapper around a WebSocket connection to our server.

You can create a consumer that will connect to "/cable" without any extra params just by calling "createConsumer".

import { createConsumer } from "@rails/actioncable"

const consumer = createConsumer()

If you want to pass extra params to the server's Connection object, e.g. for authentication, you'll have to pass the exact URL with params and all to "createConsumer".

import { createConsumer } from "@rails/actioncable"

// Fetches the user's auth token from a meta tag in the DOM
const token = document.querySelector("meta[name=auth_token]")?.content

// Takes the current URL,
// changes it's path to "/ws",
// and adds "?token=#{token}" as the query params.
// The result looks something like "https://example.com/ws?token=123"
const webSocketURL = new URL(window.location.href)
webSocketURL.pathname = "/ws" // usually it's "/cable"
webSocketURL.search = `?token=${token}`

const consumer = createConsumer(webSocketURL.toString())

Now that you have a consumer, you can subscribe with it to any channel you'd like. To subscribe, you have to create a subscription with an identifier.

If you remember from before, an identifier has to have a "channel" field, but can also have any additional fields you want and you'll have access to these fields as "params" in your Channel object.

import { createConsumer } from "@rails/actioncable"

const consumer = createConsumer()

// Subscribe to the ChatChannel 
// and pass { room_name: "Ruby Zagreb" } as params
consumer.subscriptions.create(
  { channel: "ChatChannel", room_name: "Ruby Zagreb" }
)

How do I process an incoming message with this?

Well, when your subscription receives a message it will try to call a "received" method of the subscription object to process the message.

You can create a received method on the subscription yourself, like so

import { createConsumer } from "@rails/actioncable"

const consumer = createConsumer()

// Subscribe to the ChatChannel 
// and pass { room_name: "Ruby Zagreb" } as params
const subscription = consumer.subscriptions.create(
  { channel: "ChatChannel", room_name: "Ruby Zagreb" }
)

subscription.received = function(message) {
  document
    .querySelector("#messages")
    ?.insertAdjacentHTML("beforeend", `<div>${message}</div>`)
}

Though this approach has a problem - if you get a message immediately as you subscribe, but before you create your received method, you'll miss that message.

Action Cable allows you to extend the Subscription object as you create it to avoid this problem. You can pass a second argument while creating a subscription. That argument has to be an object with which the newly created Subscription object will be extended. Extending an object in JavaScript is similar to including a mixin in a class in Ruby.

import { createConsumer } from "@rails/actioncable"

const consumer = createConsumer()

// Subscribe to the ChatChannel 
// and pass { room_name: "Ruby Zagreb" } as params
const subscription = consumer.subscriptions.create(
  { channel: "ChatChannel", room_name: "Ruby Zagreb" },
  {
    received(message) {
	  this.appendMessage(message)
    }
    
    appendMessage(message) {
	  this.messageContainer()?.insertAdjacentHTML(
		"beforeend", 
		`<div>${message}</div>`
	  )
    }
    
    messageContainer() {
	  document.querySelector("#messages")
    }
  }
)

But there are other events that you can process in your Subscription object besides receiving a message.

You can process initialization events, connect and disconnect events, as well as subscription rejections. Each event requires you to define a method to process it. Initialization requires an "initialized" method, rejection a "rejected" method, connect and disconnect a "connected" and "disconnected" method.

import { createConsumer } from "@rails/actioncable"

const consumer = createConsumer()

const subscription = consumer.subscriptions.create(
  { channel: "ChatChannel", room_name: "Ruby Zagreb" },
  {
    // Called right after the Subscription object is created
    initialized() {
      console.log(`A subscription to ${this.identifier} was created`)
    }

    // Called if the subscription was rejected by the server
	rejected() {
	  console.log(`The server rejected the subscription to ${this.identifier}`)
	}

    // Called when the subscription gets confirmed by the server
    connected(data) {
	  // The data object has a single property `reconnected`
	  // which indicates if this was a resubscribe after a disconnect
      console.log(`The server confirmed the subscription to ${this.identifier}! Reconnected: ${data.reconnected}`)
    }

    // Called when the WebSocket closes
	disconnected(data) {
	  // The data object has a single propert `willAttemptReconnect`
	  // which indicates if a reconnect attempt will be made or not
	  console.log(`WebSocket to ${this.consumer.url} closed! Will attempt reconnect: ${data.willAttemptReconnect}`)
	}

    // Called when a message is sent from the server
    received(message) {
	  console.log(`Received message: ${message}`)
    }
  }
)

There are also two actions that you can perform from your subscription - send messages to the server, and unsubscribe.

To unsubscribe just call "unsubscribe".

import { createConsumer } from "@rails/actioncable"

const consumer = createConsumer()

const subscription = consumer.subscriptions.create(
  { channel: "ChatChannel", room_name: "Ruby Zagreb" },
  {
    received(message) {
      console.log(`Received message: ${message}`)
      
      if (message.toLowerCase() === "avada kedavra") this.unsubscribe()
    }
  }
)

And to send a message call the "send" method with the message you want to send.

import { createConsumer } from "@rails/actioncable"

const consumer = createConsumer()

const subscription = consumer.subscriptions.create(
  { channel: "ChatChannel", room_name: "Ruby Zagreb" }
)

subscription.send({ action: "post_message", content: "Hello, Zagreb!" })

Thousands of connections, one server

Now you know how to connect to Action Cable and send messages back and forth. But if you have ever tuned a Rails application you probably know that there is a thread pool that determines how many simulations requests your server can process.

Does that mean that there is a maximum number of connections that Action Cable can handle?

Well, no. As long as your server has memory available it will be able to accept and serve more Action Cable connections. Granted, it will process them slower and slower, but it will work.

How does that work?

Puma, currently the most popular Ruby/Rack application server, is a multi-threaded server. This means that when you make a request to it, your request will be assigned to one thread that will process it and generate a response.

But, you don't want to have too many threads. Each thread running on your server gets a very short slice of time to run its code on the processor. The more threads you have, the more any one thread has to wait to get to run its code on the processor.

So the smarter thing to do is to have a preset number of threads - a thread pool - that can process requests. If a request comes in, and all threads in the pool are busy, the request goes into a queue. It will wait there until a thread becomes available. That's why you want to take as little time as possible to generate a response for a request. The more a request takes to process, the higher the chance that some requests will end up in the queue which means that your app feels slow.

But Action Cable turns requests into WebSockets, which can stay open for days, how don't we run out of threads in the pool?

That's the interesting part. Action Cable processes only the initial HTTP request in Puma's thread pool. After you get a 101 response from the server the request is "hijacked" and put into an event loop. So Puma's thread pool determines the maximum number of WebSocket connections that can be opened at once.

Hijack? Event loop? What?

Rails doesn't integrate directly with an application server (like Puma). Instead it implements Rack's protocol through which it gets requests from the application server and returns responses to it. This enables applications and frameworks like Rails to work with any application server like Puma, WebBrick or Falcon.

Rack is great for request-response interactions like HTTP, but a WebSocket isn't request-response like. It's a bi-directional stream of data.

To support WebSockets and similar stream-like protocols, Rack offers a way to read and write bytes directly to and from a connection initiated by a request - it's called hijacking the socket.

When you hijack a Rack request you get it's underlying socket. You can read and write bytes to the sockets freely. But this means that it's your responsibility to implement whatever protocol you want over that socket.

Action Cable hijacks the socket and passes it to Faye's WebSocket Driver library which implements the WebSocket protocol and it puts that socket into an event loop.

An event loop is a pattern for responding to events. You start an infinite loop in which you wait for some event, when it occurs you process it and the loop starts again.

In Action Cable's case it's waiting for bytes to read from the socket it hijacked. When they are available, it reads them and passes them on to the WebSocket driver.

That's easy enough for one socket. But to wait for any of an infinite number of sockets to have some bytes ready to be read would require an infinite number of threads - one for each socket.

To avoid spawning a thread for each WebSocket, Action Cable uses nio4r which notifies it when a socket has some bytes ready to be read, without spawning any threads. It does so using different functions of the server's operating system's kernel - such as select, epoll and kqueue.

Once it's notified that a socket is ready, it reads its bytes, parses them into a message, and then passes the message to the Connection object, which then puts in in a thread pool to be processed - called the worker pool - to process that message.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Illustration of all the pools that a request goes through

Another thread pool?

Yes. This worker pool allows Action Cable to process multiple incoming messages simultaneously.

You can tweak the size of this worker pool by setting "config.action_cable.worker_pool_size" to the number of threads you'd like to have in that pool.

The more threads in this pool the higher the number of incoming messages that you can process concurrently. More threads also means that your latency will go up (as one threads will have to wait for longer to get access to the processor), your database connection count will go up (as more threads can access the database simultaneously), and your memory consumption will go up as more messages and objects are in memory at the same time.

But there is one more thread pool in Action Cable - the event loop thread pool.

This one is used to dispatch events like sending heart beat messages, subscribing to and unsubscribing from PubSub channels created by stream_from, attaching and detaching hijacked sockets, and triggering periodic timers (to which I'll get to in just a bit).

This thread pool has a fixed size as it's intended to be an event dispatch of sorts where you just schedule a task which the pool only trigger, the execution of the task is mostly done in the worker pool.

Things I wish I knew right away

There are a few things that I learned about Action Cable that I wish I knew earlier.

You can trigger actions and send messages periodically, like every few seconds. This is extremely useful if you have some state that you want to synchronize periodically, or some house keeping you want to do.

In your Channel object you can call a "periodically" class method, give it a method name or a block, and an interval. It will then run that method/block for you with whatever interval you specified.

class StockTickerChannel < ApplicationCable::Channel
  # this will send a message to the client every 2 seconds 
  # as long as they are subscribed
  periodically every: 2.seconds do
    transmit value: @stock.value, timestamp: Time.now.to_i
  end
  
  def subscribed
    @stock = Stock.find_by(symbol: params[:symbol])
	  
	reject unless @stock
  end
end

Some error trackers won't catch errors from your Channel objects unless you explicitly send them. You can do that in your Connection using "rescue_from", just like you would in a Controller.

module ApplicationCable
  class Connection < ActionCable::Connection::Base
    rescue_from Exception do |error|
      MyErrorTracker.capture_exception(error)
    end

    # ...
  end
end

There is a callback on the Connection object for when an action gets invoked on your Channel object. You can register such a callback using "before_command", "after_command" and "around_command", which is extremely useful if you use Current attributes.

module ApplicationCable
  class Connection < ActionCable::Connection::Base
    around_command do
	  Current.set(person: current_person) { yield }
    end

    # ...
  end
end

There are callbacks on the Channel object for when someone subscribes or unsubscribes. There is "before_subscribe", "after_subscribe", "around_subscribe", and "before_unsubscribe", "after_unsubscribe", "around_unsubscribe". These are useful for implementing common behavior through inheritance or mixins without having to call "super" form the "subscribed" or "unsubscribed" method.

module AppearanceTrackable
  extend ActiveSupport::Concern

  included do
	after_subscribe unless: :subscription_rejected? do
	  Current.person.came_online!
	end
	
	after_unsubscribe do
	  Current.person.went_offline!
    end
  end
end

class ChatChannel < ApplicationCable::Channel
  include AppearanceTrackable
  
  def subscribed
    @chat_room = current_person
	  .chat_rooms
	  .find_by(name: params[:room_name])
	  
	reject unless @chat_room

    stream_for @chat_room
  end
end

You probably want to bump Puma's worker timeout if you are debugging your Connection object. Puma will kill any worker that doesn't generate a response within 60 seconds.

This can be annoying if you are trying to debug something with "debugger", "binding.irb" or "binding.pry".

But you can raise that timeout using the "worker_timeout" method in "puma.rb".

# config/puma.rb
require File.expand_path("../config/environment", File.dirname(__FILE__))

# ...

# Kill a worker thread if it didn't generate a responsw in 8 hours
worker_timeout 8 * 3600 if ENV.fetch("RAILS_ENV", "development") == "development"

You have to tweak both Puma's thread and Action Cable's worker counts. Puma's thread count controls how many new WebSocket connections can be created simultaneously. Action Cable's worker pool size determines how many WebSocket messages you can process simultaneously. Both pools will create database connections!

# config/puma.rb
max_threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
min_threads_count = ENV.fetch("RAILS_MIN_THREADS") { max_threads_count }
threads min_threads_count, max_threads_count

# config/application.rb
config.action_cable.worker_pool_size = ENV.fetch("RAILS_MAX_THREADS") { 5 }

In development, sometimes the server can behave a bit wonky. I'm not a 100% sure what's going on but it seems like code reloading can cause rogue threads with stale code. If that happens just run "touch tmp/restart.txt" to quickly restart the server and reset everything.

When a client looses it's Internet connection it will take Action Cable up to 20 min to notice that, close the connection, and trigger callbacks. I wrote about that in my previous article. This can cause you headaches if you want to allow only a certain number of connections per client.

You can use the official JS client outside of the browser, like in Node or Bun. The official client doesn't use the browser's WebSocket object directly. It exports an "adapters" object which has a property called "WebSocket" which holds the object that will be used to establish WebSocket connections. So you can drop-in your own WebSocket object and use it.

You can remotely disconnect any client. There is a "ActionCable::RemoteConnections" class that acts like an Active Record relation. You can query it using its "where" method to get a connection. The where method searches for a connection by its "identified_by" fields. If it finds such a connection it gives you a proxy for that connection on which you can only call "disconnect" which then disconnects the client.

class Person < Applicationrecord
  def ban!
    update!(banned: true)
    
    ActionCable::RemoteConnections
      .where(current_person: self)
      &.disconnect(reconnect: false)
  end
end

There are multiple PubSub adapters available besides Redis. You can use Postgres instead of Redis as a backend (it has some limitations). And you can provide your own if you need.

# config/cable.yml
production:
  adapter: postgres

Final thought

Action Cable is an amazing piece of software. It can seem complex, but that’s because it does a lot. When you look at each piece individually the complexity goes away.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Block overview of everything discussed