Stanko Krtalic Rusendic: Articles

A year at 37signals

2026-08-05T07:26:18Z

A year ago I had my first day at 37signals (the people behind Basecamp, HEY and Fizzy).

This was a dream come true for me.

I wanted to work there ever since I read Rework over a decade ago. The books, podcasts and talks that followed showed me that we also share a similar philosophy towards work and the craft of coding. That just cemented my resolve.

Not to mention that it's the birthplace of Rails! The single piece of software that made me decide to turn my programming hobby into my career.

I had an incredible year, so I want to share how I got here and what I learned that might not be obvious when looking from the outside-in.

Getting the job

I'm not sure how many times I applied for a job here over the years. It must be at least six, maybe more.

Out of those I passed to the final round twice.

The first round is notoriously difficult, which is quite funny because it's just a simple cover letter.

We write a lot internally. Nearly all communication is written and most of it is long-form comments and notes. Being able to write is essential.

I wasn't a good writer. It took me a few attempts to realize that. But I was determined to get better at it, so I worked on it.

The first step to writing better is to read more. Not just technical books, self-help and similar, but stories, essays, opinion pieces. Learn what's out there and expand your vocabulary and literary horizon, then forge your own style from the things you like.

A funny side-effect of reading more is that I started to hate corporate newspeak - it's so bland and dull that I immediately tune out as soon as I start to read or listen to it now. To me it's like communicating in memes.

The second step is to write more. I started writing more often on this blog, which helped with long-form technical writing. But later on I started keeping a diary. Nothing fancy, I'd just jot down what happened every day and some thoughts bouncing around in my head - this helped me in general, not just with writing.

The third step is to learn how to write a cover letter. This took several attempts and it boils down to: be yourself.

I always felt a pressure to make the cover letter sound professional. You know, the things they tell you in school: serious tone, respect, warm but also not too personal.

That's terrible advice. And I realized that only after I started hating corporate newspeak. It makes you sound the same as everyone else, and dull on top of that.

A cover letter should explain why you want to work there. It's your chance to tell your story. It's supposed to be personal, fun and interesting.

The second round is a take-home exercise.

I was never worried about this. For me Rails and the OSS that 37signals puts out has always been a treasure trove to explore and learn from so I was familiar with their style, and I already spent a lot of time internalizing why they structure code like they do.

I could write a whole essay about this, but to keep it short: ergonomics and domain design are important; the code should read like prose.

Lastly there is the interview.

I fumbled the first one. I was unbelievably anxious and over-thought and over-explained every question; the conversation didn't flow. But I got good feedback from it and spent a lot of time improving on it. The second time around I was relaxed and we had a fun chat.

Then, one evening I got an email ominously titled "Some news" telling me that they'd like me to join the company! I was over the moon!

Working here

I have a hazy memory of the first few weeks - they felt like a dream. What I remember the most is reading, and reading a lot.

As soon as I started working on actual tasks a new feeling hit me: doubt. I started to second-guess my own work and this thought formed in my mind that I just lucked out with the hiring process and that someone would soon notice that my work wasn't any good.

This feeling lingered on for months and passed only when I started getting fewer comments on my PRs.

It sounds like a curse, and it felt like it, but now I see that it was also a boon. It motivated me to get better and to learn. I figured that's the only thing that would cure that feeling, and it did.

Then came Fizzy's launch.

I got to work on Fizzy as soon as I joined. My first task was to add FizzyAsk - an LLM-powered chat where you could tell Fizzy what to do or ask it questions about your data.

After adding it I barely ever used it. And to my surprise it was completely scrapped alongside all AI features after a month.

This was the first time I ever saw features get cut - not pivoted, not moved, not put on hold - after they were built and before they were even released.

The reason behind the cut was eye-opening: delightful or nothing.

Imagine an overgrown garden with hundreds of different plants in it - that's what the products of most companies look like to me: an eclectic collection of ever-growing features. The garden is "designed" by adding whatever plants the neighbours added to their gardens this week. Everything is overgrown because nobody has the time to care for so many plants, and even if they do they don't know how. The plants that are cared for are overshadowed by the overgrowth.

By comparison, what I saw with Fizzy was gardening. There was a vision. The garden had just a few flowers, but all of them were curated to work together and arranged to be delightful. If something new didn't work with what was already there it wouldn't make the cut, or we'd remove some of the old flowers to make room for the new ones.

I talked to a few friends about this and some of them had an interesting comment: "Isn't that soul-crushing? You spend so much time building something just to see it go down the drain."

No, it isn't. In fact, it's liberating. Not all ideas are good, but you can't know that ahead of time. Sometimes you have to try them to understand why they don't work. It would be far more soul-draining to have to support an experiment for years than to admit that it wasn't very good and bury it right away.

Whenever I talk about this I think of Buddhist mandalas. These are intricate artworks made of colored sand that can take weeks to complete. Once they are done they are admired for a while and then swept up and thrown into a river. Nothing is permanent.

Later on I was tasked with building a new magic-link-based auth system. Sounds easy, right? It wasn't.

Fizzy was built on two new in-house gems: activerecord-tenanted and beamer. AR-tenanted separated each tenant into their own database, and Beamer replicated those SQLite databases to the whole fleet of servers. The catch was that there had to be a central identity database for sign-in, but each server was the writer for a few tenant databases, so you could never write to that database and then the tenanted database in the same request because the two databases would be on different servers most of the time.

As I was untangling that I realized that I was used to working with the latest release of Ruby and Rails, even with Rails edge. But this was different. We were ahead of Rails edge. We were building what might become Rails edge at some point and dogfooding it ourselves to work out the kinks!

This can sometimes become a challenge. The magic links took two weeks to implement and I spent a good part of those two weeks wondering if it could even be done.

Some of my work on Fizzy did make it into Rails. I added Crockford's Base32 to Fizzy for magic link codes and that became SecureRandom.base32. Later on I added passkeys to Fizzy, and that might also land in Rails as has_passkeys in Active Record.

After the launch we shifted into cycle work. This was the first time I experienced Shape Up.

Shape Up has two modes: cycle work and new-product mode. Cycle work is the thing most people associate with Shape Up - six week cycles with cooldown periods in between. In new product mode there's no fixed cycle, no appetite, no kick-off, no pitch. It's like jazz, things are made up as we go.

After working in both modes I realized that what I thought of as "agile development" was just waterfall with extra steps.

Take scrum as an example: two week sprints, task pointing, standups. It sounds quite similar to Shape Up: six week cycles, appetite, check-ins and hill-charts.

The magic is in what happens behind the scenes.

In every company I ever worked at there was something like a "product roadmap" - a document that laid out each feature that would be added in the fiscal year and in which order they would be worked on. Then scrum gets slapped on top of that. So now you are working in cycles, but there is this Gantt chart called the "roadmap" in the background that suspiciously looks like a waterfall and if anything slips the cascades become a bit wider. Scrum isn't there to get things done, it's there to keep you on schedule.

In Shape Up the plan is laid out six weeks at a time. It isn't set in stone, it can shift; projects are dropped, shortened or extended. The important part is that after the six weeks are over a new plan is laid out that takes into account not only what was done in the last six weeks but also what happened in the world.

Jason Fried had an excellent article about this where he compared it to running like a squirrel. A squirrel runs a little, then stops, looks around, picks a new direction, and then runs some more. As it does that it takes its surroundings into account. It goes in a direction but it regularly re-evaluates. That's agile.

I've seen so many serendipitous situations where one project became the groundwork for another that was just the right fit at the moment. You can't plan this, you have to see and feel the previous work to form a vision of what you could build with it next.

These are just the most eye-opening things I learned.

Last week was a big week for me because the company decided to keep me! After the first year everyone goes through a review that asks one question: knowing what the company knows now, would they hire you again? A huge weight has been lifted off my shoulders even though I knew the review would go well.

I'm grateful I get to work alongside and learn from my amazing colleagues at 37signals.

When I joined I expected a challenge, I wanted to learn and to grow, and that's exactly what I got. I'm glad I managed to rise to it. This really was a dream come true.

Clean air & AI

2025-06-12T14:00:00Z

One of my favorite activities is hiking. I'm fortunate to live close to a mountain where I can go for a hike any time I want - often early in the morning, before work.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A winding road on the way to Medvednica

I love the trek over hills and valleys, through forests and streams, the creaking of trees in the wind, the rustling of leaves, the murmur of streams, and the discussions with friends along the way.

I love the destination - the view, the tranquility, the bean stew with sausage at a hiking lodge, cuddling up to a fireplace in the winter.

But most of all I love the mountain air.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Zagreb, as seen from Sljeme, covered with clouds

It smells different, it’s light, relaxing, refreshing, and clean.

The difference is most striking on the way down - at one point, the air starts to feel heavy, turns dusty, and takes on an odor.

One winter morning, as the trail gave way to pavement and the outskirts of the city, a sharp sulfur smell - almost like gunpowder - hit me and stayed with me all the way home.

Something wasn’t right. The air wasn’t usually that bad. I checked the AQI for Zagreb - it was the fifth worst in the world that day.

I immediately bought a cheap air purifier off IKEA and decided to make an air quality monitor.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Breadboard with the first prototype of my air quality monitor

I made a quick prototype using parts I bought off AliExpress. After ironing out a few kinks, I ended up with something functional - Air Quality Box.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The Air Quality Box

To turn the rat's nest of a prototype into something presentable, I designed a custom PCB that connected all the components and packed them as tightly as possible in a enclosure that I printed. It was as small as I could make it given the parts and air flow constraints needed for everything to work properly.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The tightly packed inside of the Air Quality Box

It integrated with Home Assistant and displayed measurements on its screen. This allowed me to automate my IKEA air purifier, and I could read the air quality right from my desk without any app.

That little boxed served me for four years and taught me a lot about air quality:

Higher CO2 levels in a room cause headaches and drowsiness
Air with very low particulares and 40% humidity feels like mountain air
VOCs and high humidity make the air feel heavy

But it had two major flaws.

First, its design was very industrial - people either loved it or hated it.

Second, the UI was clunky - you couldn’t just glance at it and see what’s wrong, you had to wait for all the measurements to cycle through.

At first, I started working on a V2 which was heavily inspired by IKEA’s Vindriktning. It had a single light bar that told you an overall score of the air quality, and a small screen at the top that showed what needed fixing.

Then I came across the Awair Element. It had everything I wanted - a beautiful enclosure, it shows everything at a glance, it can integrate with Home Assistant. So I just bought one, and I have been very happy with it for the past 6 months.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The Awair Element that I bought

Its default screen shows five bars and a number. The number is the score of the air quality based on all measurements. Each bar represents a measurement: temperature, humidity, CO2, VOC, and particulates. When a value is within the target range, the bar shows a single dot. If it’s too high or too low, it displays two to five dots, depending on how far it has drifted.

I just have one minor annoyance with this interface - you can't tell if a measurement is too low or too high at a glance.

Say the humidity bar shows three dots - is it too humid or too dry? To find out, I have to pull out my phone, open the app, and read the value. Or I can reach around the device, press a button a few times until it displays the humidity measurement, and then cycle back to the score display. It isn't terrible, just annoying.

But I had an idea for how to fix this. I could create an app that lives in the system tray and shows the current air quality score. When clicked, it expands to show all measurements. That way I can quickly check what's wrong - no phone, no fiddling with buttons.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Demo of the system tray app

This wouldn't be my first Linux Desktop app. Years ago, I made one using Qt, QML and Rust, so I knew what I had to do.

There would be two parts to the app - the app itself and the system tray widget. The app discovers air quality monitors on the network, gathers measurements, and handles settings. The system tray widget displays the gathered measurements. The two would communicate with each other via D-Bus.

Over the last few weeks, I've been sharpening my Go skills, so I decided to write this app in Go. And since I use GNOME as my Desktop environment these days, I went with GTK instead of Qt - it fits in better visually.

Which brings me to AI.

Until now, I've used AI as a pair programmer. I’m in the driver’s seat, and when I want to bounce ideas around or look something up, I ask the AI.

To me, that’s the most natural way to work with an AI. It allows me to stay in flow - I can focused on keeping the code clear, modeling the domain, and solving the real problem, while the AI allows me to tap into a vast pool of knowledge in a moment's notice. No more digging through SEO spam for that one useful link - I get exactly what I need, when I need it.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

ChatGPT and NVIM side by side

This approach helped me a lot while learning Go. Whenever I hit a wall, I could ask the AI how it would solve the problem and get a few new things to read up on. In a way, it felt like having a mentor.

This time, I decided to flip things around and let the AI drive while I guide it. I wanted to see if there was a better way to work with AI than what I'd been using. So I installed Claude Code and bought $50 worth of API credits.

(Since this week, if you have a Pro subscription to Claude, Claude Code is included - no API credits needed)

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The initial prompt given to Claude Code

And that's how I - or rather we - made GNOME Desktop Air Monitor.

Demo of GNOME Desktop Air Monitor

I didn't like this way of us working together - for me, the pair programming approach just works better. Letting the AI drive seemed to amplify both of our weaknesses.

First, the AI is very goal-oriented.

It only cares about the outcome - in many ways it behaves like a greedy algorithm, making choices that seem best in the moment, without regard for long-term consequences.

This isn't bad per se. Sometimes this leads to efficient, pragmatic solutions. But more often, it results in spaghetti code - one hack piled on top of another, just to reach the goal. It doesn't care how maintainable, extendable or readable the solution is - that isn't its goal.

To counter that, I had to be very explicit. I had to explain not just what I wanted it to do, but also how I wanted it to do it.

For example, I asked it to create an index screen that shows all devices, then a show screen for individual devices, and finally a settings screen.

The result? One giant app.go file. All three screens were in it. All their logic, all the app setup and window management, everything. There was an App struct that held the state of every button, list and graph that could ever appear, with cryptic names like "button", "settingsButton", "actionList", etc. There were three methods - "showIndex", "showSettings" and "showDevice" - that were nearly identical, each resets the state of every UI element before switching screens.

I had to explicitly ask it: "Do you think it would be better to create a struct for each screen and keep its state there? You could add a reset and a show method there too". It agreed - and refactored the code.

I tried to improve the naming in the same way. But it kept giving me worse and worse suggestions until I gave up and renamed everything manually.

By the end, my prompts had become so detailed that I felt like I was micro-managing someone. And that's just not how I want to work. For me, trust - in someone doing their job and doing it well - is essential. If that trust isn't there, then explaining exactly what needs to be done is often more work than just doing it myself.

I talked to a friend about this and he said something along the lines of "you have to treat it like a junior developer". But that's the problem - it isn't a junior developer. It doesn't learn from its mistakes. You can't explain to it in which situation one approach is better than another. It won't conclude that it produced garbage or that it's painting itself into a corner - as long as the goal was technically achieved.

Second, it doesn't understand the sunken cost fallacy.

The AI will almost never give up on a solution. It will almost always build on top of what already exists. It doesn't question, it doesn't suggest, it just marches on. In that regard, it's similar to a junior developer.

This gets worse and worse as you iterate on your project. At some point you have to step in and explain to it that the current approach just doesn't work, and that it should try something else.

I didn’t really see this as a serious issue until my girlfriend - who can't code - asked ChatGPT to help her build an iOS app. She described what she wanted, and it generated a React Native app that did exactly what she asked. Then she asked it to add live notifications - at the time, an iOS-only feature that required native code to get it working. Instead of switching tools, it piled on hack after hack to make it work. It ultimately failed.

Then I told it to recreate the app in Swift - and it nailed it on the first try.

If I didn't know what it had to do - and didn't tell it explicitly - we'd still be spinning around in circles, trying every hack known to man or machine.

Third, it doesn't generalize.

I often found myself in awe of both its brilliance and its stupidity in the span of five lines of code. It would implement a pattern that felt like the right solution - elegant, compresses the complexity just enough, clean - but only in the narrow context of the problem it was solving at the time. It never generalized the pattern or applied it across the rest of the code. And that’s what’s so frustrating - it has access to vast knowledge, the entire codebase, and limitless memory - you'd expect better.

But enough of the negatives. I also discovered some strengths.

It's brilliant at writing documentation from existing code.

Last week, I started writing a new gem and it crossed my mind to try letting Claude write the documentation for it. To my surprise, it produced well-worded, concise descriptions, complete with good examples and relevant links - even if a few of the links turned out to be dead.

It's decent at writing tests.

I also let it write a few tests, and they turned out surprisingly solid. It covered a lot of edge cases I might’ve missed on a first pass, and the structure was generally clean and readable. Occasionally, it repeated itself more than necessary - but as a starting point, it saved time.

It's excellent for scaffolding.

The most enjoyable way to work with an AI agent, for me, was to let it handle the first pass - generate the rough structure. Then I’d come in and rewrite the code the way I like it. That initial scaffold gave me something to react to, and it helped me move faster.

All in all, I learned that I love programming as much as I love solving problems.

Letting the AI drive helped me see that. It made the coding faster - but it also took the joy out of shaping the code, thinking through the domain, and making things beautiful. That’s the part I don’t want to give up.

Working with AI, as a partner, is what feels best to me. It can help me move faster, think broader, and stay in flow.

My last PHP app: How I Fell for Ruby

2025-05-24T09:00:00Z

Last week I found a backup of the very last PHP app I ever wrote. The backup was from 2009, but the app itself was written in 2007 - 18 years ago!

Back then I was in my late teens and the world was (or at least felt) completely different. Out of nostalgia, I spent some time exploring the source code and even got it to run (for the most part).

This reminded me of how different and quirky the web used to be, and how I discovered Ruby.

What was the app for?

My friends and I used to play Call of Duty 2 together after school - seems like the world hasn't changed that much after all - and we got good at it. So we organized into a team, called a clan, and started playing against other clans for fun.

Every clan we looked up to at the time had its own website where it put up its latest games, demos of those games (a demo is a file that allows you to replay a game and spectate), they had a forum, news, and so on. So we wanted to have our own website, but we were extremely tight on money.

The most that we could spend - all of us together - on a website was around 3€ per month or 35€ per year. To put that into context, 3€ could buy you 2 large pizzas at the time - we were financing this with pocket money.

All hosted website engines were way too expensive for us. We had just enough money to pay for PHP hosting to a local company. For the domain we got a free .su domain from a sketchy registrar. And now we had to make a website.

At the time I used to write small programs to solve my math homework or just to play around with my computer, and I just learned HTML, so I volunteered to make the website.

Looking back, implementing a website with news, a calendar, file uploads and a comment system was very ambitious for a first project. But I didn't know that at the time so I gave it a try and failed. There were too many moving parts which caused too many bugs, and on top of that the file storage logic that I wrote - because I didn't know that databases existed - kept corrupting files forcing me to recreate everything every now and then.

Somehow I came across webSPELL - a CMS for clan websites. That solved most of my problems. Now all I had to do was create a template, add and change a few things. All in all I made five websites using PHP and webSPELL - three different iterations of our website, and two websites for other clans.

The last one I made was the third iteration of our website in 2007. Here it is in most of its former glory.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The website, mostly working

It even had a visitor counter - this was the Google Analytics of the time.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The visitor counter that was in the bottom-right corner

This was written for PHP 4 and I'm running it in a Docker container with PHP 7 so some things - like the news, forum and stats tickers at the top - just broke during the update. I'm honestly amazed that I only had to do a handful of changes to get the app to boot with PHP 7.

Digging through the code made me remember how different the Web was back then.

For starters the app rendered XHTML because HTML 5 didn't exist yet and HTML 4 was difficult to work with.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">

The encoding was set to iso-8859-1 to get support for Croatian diacritics. UTF-8 did exist at the time but a lot of software just didn't support (Windows 98 was still very common) so the best thing I could do was to use iso-8859-1 as it would render the characters correctly for most people.

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

Then the wild part - this whole site is one giant HTML table with an iframe in it.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

DOM of the website

DIVs did exist, but they were much harder to position than table cells - flexbox didn't exist - so they were mostly used as DIViders. And the nice thing about tables is that you can set a fixed width to every cell using the width attribute - this was the only way you could do templates back then.

A template started out as big picture that was sliced into box-shaped sections like a jigsaw puzzle. Then you'd make an HTML table that puts the puzzle back together. You'd make each table cell the exact width and height of your slice and then set the slice image as the background image of that cell. If you had a layout that wouldn't fit into a single table you'd nest HTML tables to get the layout you want.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

An example of how slicing worked

Slicing was an art. A big part of the job was to create slices that could be tiled. This saved on bandwidth which, compared to today's 100+ Mbps speeds, were barely able to provide a meager 2 Mbps. This also meant that you'd often have to slice something in such a way that seams and tiling wasn't obvious.

E.g. the horizontal bars on the top and bottom would get turned into a 1px wide image that would tile horizontally. The vertical bars would be turned in a 1px high tiling image. This also allowed the template to scale to fit any size content and would often reduce a 1MB image down to 1kb or less if you used a GIF.

Another interesting thing - this site was exactly 1011px wide. Most monitors at the time were either 1024px or 1280px wide, so the site would fill most of the screen. Responsiveness just wasn't a thing - the iPhone had just come out and there was literally no way to detect the screen size.

This produces hilarious results on today's monitors. E.g. this is what the site looks like on my ultra-wide.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

What it looks like on a 40" ultra-wide

And on my iPhone 14

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

What it looks like on an iPhone 14

I can barely read the text, and I'll need a toothpick to click on a link.

Ruby, Ruby, Ruby, Ruby

Over the winter break of 2008, I stumbled across a tweet praising Ruby on Rails as an elegant alternative to PHP. The next day, my mom asked me if I could build a website for her real estate business - perfect timing.

I decided to give Rails a try. webSPELL wasn't a good fit for a real estate website, and I was ready for something new.

Something about Ruby just clicked. The code read like a story. Ruby's syntax felt natural and obvious in a way PHP never had. Suddenly programming wasn't just about hacking things together - it was expressive, joyful, and beautiful.

I remember comparing how I used to join first and last names in PHP:

implode(" ", [$first_name, $last_name])

to how it looked in Ruby:

[first_name, last_name].join(" ")

Ruby felt like I was talking to the data. I'd say "Hey Array, join your elements together with this separator". PHP, by contrast, felt like rummaging through a toolbox hoping the right function was in there somewhere.

Simple database queries became poetic:

$user=safe_query("SELECT * FROM ".PREFIX."users WHERE first_name = 'John' LIMIT 1");

vs.

user = User.find_by_first_name("John")

Rails didn’t just simplify the work - it made me fall in love with writing code. After that, there was no turning back. I decided to turn my programming hobby into my career.

I went to college to learn computer science, protocols, and telecommunications. Then my first job was at a digital agency writing Rails apps to help bootstrap businesses - many of which are still around. And I've been fortunate enough to work with Ruby and with Rails on every job I've had since.

That dusty old PHP app didn't age well, but I'm glad I found it.

It brought back memories of a time when building things felt like magic, which reminded me how I stumbled into Ruby. One tiny decision I made one winter ended up shaping my whole career.

I didn't know it then, but I was writing the first lines of a lifelong love story - one that carried me from Call of Duty matches and sliced-up tables to a profession built on joy, clarity, and code that reads like prose.

Building Twice: A clone of Once

2025-05-16T08:00:00Z

When Campfire - the first Once product - came out, a few of us at Ruby Zagreb pitched in to buy a copy.

Prometheus had brought fire from Mount Olympus, and we came to see the blaze. We were finally seeing how the people at 37signals - the birthplace of Rails - build Rails apps. And as a bonus, we could finally ditch our free Slack Channel!

After days of exploring the source code, the time came to set up Campfire. The purchase email included a shell command that looks like this

/bin/bash -c "$(curl -fsSL https://auth.once.com/install/1111-1111-1111-1111)

I ran that, and after solving a few issues, I had a Docker container running Campfire.

This is a super elegant way to distribute apps! But how did it work? I wasn’t sure so I decided to replicate it and see.

Figuring out how Once works

Just like last time, I started to unravel this ball of yarn by following a thread - the shell script.

Since all it does is execute a script it gets from the Internet I went to see what that script was

curl -fsSL https://auth.once.com/install/1111-1111-1111-1111
#!/bin/bash
set -e

echo "Installing the ONCE command..."
echo
sudo true

command_path="https://auth.once.com/install/1111-1111-1111-1111?arch=$(arch)&platform=$(uname)"
curl -s $command_path > .once-tmp
sudo mv .once-tmp /usr/local/bin/once
sudo chmod +x /usr/local/bin/once

if [[ "$OSTYPE" == "darwin"* ]]; then
  # Don't use sudo on macOS; Docker Desktop is likely to be running as the user
  /usr/local/bin/once setup 1111-1111-1111-1111
else
  sudo /usr/local/bin/once setup 1111-1111-1111-1111
fi

This downloads a binary for your OS and CPU architecture, installs it, and then runs it passing in your license key.

As this was a binary I couldn’t just open it up and see it’s code so I resorted to simple reverse engineering using the strings command - it returns all things that look like C strings from a given file.

... 
crypto/tls.masterSecretLabel 
crypto/tls.extendedMasterSecretLabel 
crypto/tls.keyExpansionLabel 
crypto/tls.clientFinishedLabel 
crypto/tls.serverFinishedLabel 
mime/multipart..inittask 
mime/multipart.ErrMessageTooLarge 
mime/multipart.multipartFiles 
mime/multipart.multipartMaxParts 
mime/multipart.emptyParams 
mime/multipart.multipartMaxHeaders 
mime/multipart.quoteEscaper 
net/textproto.errMessageTooLarge 
net/textproto.colon net/textproto.nl 
net/textproto.commonHeader 
net/textproto.commonHeaderOnce 
net..inittask 
net.rfc6724policyTable 
net.confOnce 
net.confVal 
net.netdns 
...

There were a lot of strings that looked like Go packages, functions and methods. I soon found a few references to cobra - a Go package for building CLI tools - so I was sure that I was looking at a compiled Go app.

... 
github.com/spf13/cobra.init 
github.com/spf13/cobra.map.init.0 
github.com/spf13/cobra.GetActiveHelpConfig 
github.com/spf13/cobra.activeHelpEnvVar 
github.com/spf13/cobra.legacyArgs 
github.com/spf13/cobra.(*Command).HasSubCommands 
github.com/spf13/cobra.(*Command).HasParent 
github.com/spf13/cobra.NoArgs 
github.com/spf13/cobra.writePreamble 
...

This was very fortunate because I’ve started using Go at work lately and I knew that it includes a lot of metadata in its binaries (this can be turned off but isn’t by default). So I looked around and found the whole structure of the app, and all the method signatures.

once-cli/ 
├── cmd/ 
│ └── once/ 
│ └── main.go 
│ 
└── internal/ 
    ├── archiver/ 
    │ ├── compressor.go 
    │ └── uncompressor.go 
    │ 
    ├── cmd/ 
    │ ├── auto_update.go 
    │ ├── auto_update_off.go 
    │ ├── auto_update_on.go 
    │ ├── data.go 
    │ ├── data_backup.go 
    │ ├── data_restore.go 
    │ ├── password.go 
    │ ├── password_reset.go 
    │ ├── root.go 
    │ ├── setup.go 
    │ ├── start.go 
    │ ├── status.go 
    │ ├── stop.go 
    │ ├── update.go 
    │ └── util.go 
    │ 
    ├── config/ 
    │ ├── config.go 
    │ ├── environment.go 
    │ └── verifier.go 
    │ 
    ├── docker/ 
    │ └── docker.go 
    │ 
    ├── networking/ 
    │ ├── dns_check.go 
    │ ├── network_check.go 
    │ └── ssl_traffic_check.go 
    │ 
    └── tui/ 
      ├── backup/ 
      │ └── backup.go 
      ├── components/ 
      │ └── task_list.go 
      ├── question/ 
      │ └── question.go 
      ├── restore/ 
      │ └── restore.go 
      ├── runner/ 
      │ └── runner.go 
      ├── setup/ 
      │ ├── installer.go 
      │ ├── registration.go 
      │ └── setup.go 
      └── util.go

And I found three URLs

https://auth.once.com/install/ 
https://auth.once.com/verify/%d 
registry.once.com

The first one was the same as the download URL, the second one obviously verifies something but I didn’t know what. My guess is for the password reset functionality (look at once-cli/internal/cmd/password_reset.go) because it interpolates a number at the end (that's the %d). The third one looks like a private Docker registry.

Can I login to the Docker registry?

After set up, Once creates a config file where it stores how you configured it.

// contents of /root/.config/once/config.json
{
  "token": "1111-1111-1111-1111",
  "product": "campfire",
  "product_name": "Campfire",
  "email_address": "stanko@example.com",
  "ssl_domain": "chat.rubyzg.org",
  "validation_token": "PHONY_VALIDATION_KEY",
  "secret_key_base": "PHONY_SECRET_KEY_BASE",
  "vapid_private_key": "PHONY_PRIVATE_KEY",
  "vapid_public_key": "PHONY_PUBLIC_KEY",
  "storage_location": "/var/once/campfire",
  "cron_hour": 2,
  "once_binary_etag": ""
}

Notice that it stores the license key and my email. Also notice that it calls the license key "token". My guess is that it uses the license key as an auth token with the registry to pull updates. That's a common way to do token auth with Docker Registries. Let's try that.

docker login \
  --username 'stanko@example.com' \
  --password '1111-1111-1111-1111' \
  'https://registry.once.com'

Login Succeeded

Bingo!

Now, where did it get my email, the product, and product name from? These weren't asked during set up. I'm guessing that the install URL is in the binary because it can respond with HTML / text for the install script but it can probably also respond to JSON. Let's try that

curl https://auth.once.com/install/1111-1111-1111-1111.json | jq
{
  "token": "1111-1111-1111-1111",
  "product": "campfire",
  "product_name": "Campfire",
  "email_address": "stanko@example.com",
  "ssl_domain": null,
  "validation_token": "<some-base64-string>"
}

Two for two!

I was curious about the validation token as it was obviously a Base64 string, but decoding it returned complete gibberish - it doesn't even look like unicode - so I assume that it's either completely random or an encrypted payload. Maybe that's what the verify endpoint is for?

My guess is that the token is used to update the "ssl_domain" field with the Once auth server after the install completes or during the DNS check. That would mean that the install endpoint probably also responds to a POST request.

Anyway, I now had a good understanding of how Once works.

How Once works

Once consists of three main parts:

The Auth server
The private Docker Registry
The CLI app

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

An overview of Once

The Auth server handles token validation for the registry, distribution of the CLI binary, license key inspection, and probably the order created webhook from Shopify.

The CLI handles Docker installation and configuration on the client, pulling and running of product Docker images, product container management, automatic-updates, backups, restores, and a few other things.

The private Docker Registry is, well, just a Docker Registry... It stores and distributes Docker images.

The license key you get is used as an access token to the registry. With it the CLI can pull images from the registry. That token probably grants access to a single repository in the registry - just for the product you purchased.

Building Twice

Now the fun part, making a clone of Once.

I decided to call this project "Twice" as a joke on the fact that it's a clone of Once - it's a second implementation. The source code is available on GitHub if you want to explore it, build, modify, or run Twice for yourself.

My main motivation is to learn more about how Once is set up, but a nice side effect is that I'll get a distribution system out of it - in the case that I decide to make an app and sell it. With that in mind, my primary goal is to implement just the setup CLI command. I'll work on everything else later.

Just like with Simpliki, I'll start from the center and work myself out. In the case of Twice the center is the CLI app - specifically the setup command.

As I've been working with Go at work lately, I felt confident in using it for the CLI. This would be a nice exercise to sharpen my Go skills. Otherwise I'd probably go with Rust as I have more experience with it, or I'd give Tebako a try - a new way to package Ruby apps into a single binary.

So I started a new Go project, added cobra to it and followed the getting started guide to create a CLI app with a setup command that accepts one argument - the license key.

go run cmd/twice/main.go setup --help
Install all the necessary dependencies to run a product and then installs the product associated with the given license key

Usage:
  twice setup <license-key> [flags]

Flags:
  -h, --help   help for setup

At this point I got lost in the weeds. Once has this nice TUI (terminal user interface) that walks you through the setup process so, naturally, I wanted to make a nice TUI too.

Twice TUI

I spent a couple of hours working on it, I got quite far, but in the end I realized the TUI didn't matter. It was just slowing me down, so I ditched it and replaced it with good old puts.

Twice CLI

This was waaaaay quicker to implement and allowed me to focus on the actual logic - like figuring out if docker was installed, is it running, writing an API client, interacting with Docker, etc. - instead of rendering a fancy UI.

I like the version with the TUI more - it feels refined - and I think I'll revisit it once I'm more comfortable with Go and Bubble tea (the library that renders the TUI).

As I was building the CLI I started the Auth app and mocked a few endpoints just to get the setup command done. I didn't implement any UI and relied heavily on "bin/rails db:fixtures:load" to give me some data to work with. These endpoints were plain Rails controllers that rendered JSON.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The models I had for Twice

With most of the CLI out of the way, and with some of the Auth app done I turned my attention to the Docker Registry.

There are a few Docker Registries out there:

I liked the idea of having a Registry that's written in Rails so I tried Portus, but I couldn't get it to run locally - there are multiple known issues and the project was archived 2 years ago.

Harbor is a conglomeration of micro services. When I saw that I had to run 4 or more containers just to get the thing going I gave up.

This left me with Registry. It has everything I need - access control via token auth to a configurable auth server. It has no UI or API, but this is a plus as I don't have to think about customers trying to login to the registry UI or interacting with it other than pulling images. And I can make the system stateless, thus less error-prone.

All I had to do to get the registry working is to generate a self-signed certificate

openssl req -x509 -newkey rsa:4096 -nodes \
    -keyout ./auth.key \
    -out ./auth.crt \
    -days 365 \
    -subj "/CN=registry-auth" \
    -addext "basicConstraints=critical,CA:true" \
    -addext "keyUsage=keyCertSign,digitalSignature" \
    -addext "subjectAltName=DNS:registry-auth"

And set a few environment variables that configure it to use token auth, tell it to use my auth server, and to verify the auth tokens using the certificate I just generated

# Use token auth
REGISTRY_AUTH=token
# Use `http://localhost:3000/registry/auth` as the auth server
REGISTRY_AUTH_TOKEN_REALM=http://localhost:3000/registry/auth
# This isn't too important
REGISTRY_AUTH_TOKEN_SERVICE=registry
# This has to match what you return from the auth server and what's in the certificate
REGISTRY_AUTH_TOKEN_ISSUER=registry-auth
# This specifies which certificate to use to verify tokens
REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/auth.pem

That's it. Here is a Docker compose file for it

services:
  registry:
    image: registry:2
    # On Linux
    network_mode: host
    # On Mac
    # ports:
    #   - 5000:5000
    environment:
      - REGISTRY_AUTH=token
      - REGISTRY_AUTH_TOKEN_REALM=http://localhost:3000/registry/auth
      - REGISTRY_AUTH_TOKEN_SERVICE=registry
      - REGISTRY_AUTH_TOKEN_ISSUER=registry-auth
      - REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/auth.pem
      - REGISTRY_STORAGE_DELETE_ENABLED=true
    volumes:
      - ./registry/tmp/registry_data:/var/lib/registry
      - ./registry/certs:/certs:ro

Now the only thing left was to finish the Auth server.

First, I tackled registry token auth. This requires a single endpoint that responds to both POST and GET requests

match "/registry/auth", to: "registry#auth", as: :registry_auth, via: %i[ get post ]

The endpoint receives the users email and token via HTTP basic auth and responds with a JSON payload that looks like this

{
  "token": "signed.jwt.token",
  "expires_in": 300,
  "issued_at": "2025-05-15T16:55:23Z"
}

The token from the response is a JWT that contains a list of access permissions - these define which actions the user can perform.

{
  "iss": "registry-auth", // Has to match the value set in REGISTRY_AUTH_TOKEN_ISSUER
  "sub": "stankoexample.com", // Username of the person logging in
  "aud": "registry", // Has to match REGISTRY_AUTH_TOKEN_SERVICE
  "exp": 1747329069,
  "nbf": 1747328784,
  "iat": 1747328784,
  "jti": "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed",
  "access": [
	{
	  "type": "repository",
	  "name": "campfire",
	  "actions": ["pull"]
	}
  ]
}

The tricky part was to sign the token with the certificate such that the registry would accept it as valid. This took me some time to figure out. To get it to work I had to set two additional JWT headers called KID and X5C, which tell the registry exactly which certificate it should validate the token with. Both headers are derived from the signing certificate itself.

KID = OpenSSL::Digest
  .new("SHA256")
  .update(PRIVATE_KEY.public_key.to_der)
  .hexdigest
  .upcase
  .scan(/.{1,2}/)
  .join(":")
X5C = Base64.strict_encode64(PUBLIC_KEY.to_der)

Putting it all together, the controller looks like this

def auth
  authenticate_or_request_with_http_basic do |email, license_key|
    user = User.find_by(email_address: email)
    deny_access and return if user.blank?

    license = user.licenses.find_by_key(license_key)

    token = user.generate_registry_access_token_to_product(license.product, service: params[:service])
    deny_access and return if token.blank?

    payload = {
      token: token.to_s,
      expires_in: token.duration.to_i,
      issued_at: token.issued_at.iso8601
    }

    render(json: payload, status: :ok)
  end
end

Notice that the token is an object. This provides a nice domain language (DDD domain language) that makes it easier to work with, and reason about, the token. If I want the duration or the issued at time I can ask the token for it; if I want to turn it into a string I can do that too.

At this point I noticed that I didn't implement a way to push images to the registry. To fix that I introduced a new type of user - the Developer. STI made the most sense for this, so I converted the customers table to a users table, added a type column and a few extra columns.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The models in Twice after adding developers

Unlike Customers, Developers login with a password tied to their account and they get full access to the registry.

Second, the install endpoint. This was straight forward

# config.routes.rb
get "/install/:license_key", to: "install#install", as: :install, defaults: { format: :text }

# app/controllers/install_controller.rb
def install
  respond_to do |format|
    format.text
    format.json
  end
end

And then I just had to create two separate view files. One for the script in app/views/install/install.text.erb

#!/usr/bin/env bash
set -e

echo "Installing the TWICE command..."
echo

sudo true

command_path="<%= install_download_url %>?arch=$(uname -m)&platform=$(uname)"
curl -s "$command_path" -o .twice_tmp
sudo mv .twice_tmp /usr/local/bin/twice
sudo chmod +x /usr/local/bin/twice

if [[ "$OSTYPE" == "darwin"* ]]; then
  # Don't use sudo on macOS; Docker Desktop is likely to be running as the user
  /usr/local/bin/twice setup <%= @license.key %>
else
  sudo /usr/local/bin/twice setup <%= @license.key %>
fi

And another in app/views/install/install.json.jbuilder

json.key @license.key

json.owner do
  json.(@license.owner, :id, :email_address)
end

json.product do
  json.(@license.product, :id, :name, :repository)
  json.registry ENV.fetch("REGISTRY_URL", "localhost:5000")
end

Notice that I strayed a bit from 37signals' implementation when it comes to downloading the binary. I decided to separate that out into another controller action because the install endpoint already does too many things in my opinion, and adding file sending to that seems like way too much responsibility for one endpoint.

I also made the decision that the auth server would directly send the file to the client instead of e.g. redirecting to something like S3. This makes it easier to self-host twice, and due to a trick it’s no less performant.

Here is the download action

def download
  platform = sanitize_file_name_part(params[:platform])
  arch = sanitize_file_name_part(params[:arch])

  filename = ["twice", platform, arch].compact.join("-")
  file_path = Pathname.new(Rails.root.join("storage", filename))

  if file_path.exist?
    send_file file_path, type: "application/octet-stream", filename: filename
  else
    head :not_found
  end
end

It assumes that the requested binary will be called "twice-#{platform}-#{arch}", then it goes and checks if such a file exists in the storage directory, and if it does it sends it.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The download endpoint in action

In production this won't actually stream the file from Ruby, instead it sets a special header that tells Thruster to send the file which makes things way more efficient.

Third, I added authentication using the new auth generator in Rails 8, and a UI for managing customers, developers, products and licenses.

Demo of the Auth app UI

And that was it. Twice was done. Here is the demo

Demo of Twice installing an app

Some loose thoughts

When writing about building Simpliki I left out an important part - what I intentionally didn't do.

For Twice, I intentionally didn't implement domain name tracking as I think that it's not important. It can't prevent people from running the docker container how and where they want. It would add complexity and I don't see the benefit of it, so I ditched it.

I do see the utility in all the backup and management commands that the Once CLI has, which that I didn't implement in Twice. I intend to return to that at some point. But they aren't important for distributing apps so I skipped them.

Twice, currently, has the same problem as Once - it doesn't play well with other apps or Kamal. This isn't a problem if you rent a VPS specifically to run a Once app, but if you have a large bare metal machine like me then this is cumbersome. I'd like to change it so that it deploys kamal-proxy and then proxies requests to different apps through it. This would allow it to run multiple apps on the same server.

That's also why the Twice CLI doesn't keep a config file - I'd like to make it so that you can run multiple apps, so the config has to support that. This was extra work that I think isn't important for now. And I'd like to be able to install an app from different Twice instances on the same server. Something like

twice setup 1111-1111-1111-1111@auth.foo.com
twice setup 2222-2222-2222-2222@auth.bar.com

This would make it more versatile - but, again, that's way beyond the scope of creating a clone to learn how Once works.

I liked working with Go. After years of using Rust, having a garbage collector and not having to think about results, unwrapping, lifetimes or borrowing was a breath of fresh air. Though, the way it declares modules still doesn't sit right with me. If you didn't look at the source code, each directory is its own module, and all files in the directory get essentially concatenated together into that module at compile time. It works, and I can't point out any problems with that, but it feels weird.

The folks at 37signals made a really elegant distribution system for their apps. I had a lot of fun figuring out how it works and I learned a lot along the way.

Adding MCP to a Rails app

2025-05-06T09:00:00Z

Last week we had an AI hackathon at work during which we added an MCP server to our Rails app. This was much easier than I expected, the result was both impressive and scary, and I learned a few things about MCP that I didn't expect.

Demo of Claude using our app via MCP to solve a support request

Goal

My team wanted to help our support team by automating some laborious tasks, like cross referencing data, by offloading the bulk of the work to an AI agent. This would make support’s work easier, responses quicker, and in some cases it would free up my team too - an all around win.

To allow the AI to interact with our app - like a person would - we decided to add MCP to it.

What is MCP?

MCP, or Model Context Protocol, is a protocol through which LLMs can interact with different tools and resources to accomplish tasks.

The protocol specifics are fairly dry so I'll gloss over them. What you need to know is that in MCP there exists a server and a client.

A server provides tools, prompts, and resources (I'll focus only on tools). While a client connects to one, or multiple, servers and exposes their tools to an LLM. The client then makes API requests to the Servers to use tools when the LLM tells it to.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Overview of how the MCP client and server interact

In essence - the Client uses the LLM as a brain to plan out how to solve a problem and the Servers as tools to implement the solution.

A tool can be anything from a query, like searching for a user, to a mutation, like creating a coupon code - anything the server is allowed to do on behalf of the LLM.

E.g. You could ask a client to "Find a file called meeting_notes.txt", it would then use the LLM to interpret what that means and plan out how it would do that with the tools it has available. It gets a list of tools from the MCP servers it's connected to.

The LLM might respond with something like "{action: 'tool_call', tool: 'find_file', arguments: ['meeting_notes.txt']}" and the client would go and make a request to an MCP server that exposes a tool called find_file.

The server then responds with something like "{path: '/home/user/meeting_notes.txt', contents: '...'}" and the client passes that on to the LLM which then decides what to do with that information.

MCP and Rails

I used the fast-mcp gem to add MCP support to our app.

bundle add fast-mcp
bin/rails generate fast_mcp:install

This gave me two new directories - app/tools and app/resources - and a class name conflict.

The resources directory contains an ApplicationResource class, but our app already has a different ApplicationResource class in app/models that we use to represent resources from 3rd party REST APIs.

To solve the conflict I decided to namespace everything MCP-related under an MCP namespace.

First I moved the two new directories into an mcp directory

mkdir app/mcp
mv app/tools app/mcp/tools
mv app/resources app/mcp/resources

But now the classes were namespaced with Tools and Resources instead of with MCP because Zeitwerk treats each directory in app/ as a root directory. To reconfigure Zeitwerk I added the following to config/initializers/zeitwerk.rb

# config/initializers/zeitwerk.rb

module MCP; end

Rails.autoloaders.main.tap do |autoloader|
  Dir.glob(Rails.root.join("app/mcp/*")).each do |file|
    next unless File.directory?(file)

    autoloader.push_dir(file, namespace: MCP)
  end
end

That tells Zeitwerk to treat every directory in app/mcp as a root directory with an MCP namespace.

Now app/mcp/resources/application_resource.rb defines MCP::ApplicationResource instead of Resources::ApplicationResource.

With that out of the way I could finally run my server

bin/rails s

The Client

Now that the server was up and running I wanted to see if I could connect to it. The easiest way to do that is through cURL

curl -v http://localhost:3000/mcp/sse
* Host localhost:3000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:3000...
* Connected to localhost (::1) port 3000
* using HTTP/1.x
> GET /mcp/sse HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/8.13.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< Content-Type: text/event-stream
< Cache-Control: no-cache, no-store, must-revalidate
< Connection: keep-alive
< X-Accel-Buffering: no
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET, OPTIONS
< Access-Control-Allow-Headers: Content-Type
< Access-Control-Max-Age: 86400
< Keep-Alive: timeout=600
< Pragma: no-cache
< Expires: 0
* no chunk, no close, no size. Assume close to signal end
<
: SSE connection established

event: endpoint
data: /mcp/messages?

retry: 100

: keep-alive 1

: keep-alive 2

: keep-alive 3

: keep-alive 4

: keep-alive 5

event: message
data: {"jsonrpc":"2.0","method":"ping","id":715239

It seems to be working.

But to actually use an LLM with the server I had to find a proper MCP client. This was way more difficult than I thought it would be, mostly because I'm on Linux.

By far the most well-known MCP client is Claude Desktop, but it isn't available on Linux even though it's an Electron app... I tested a few open-source alternatives but in the end a coworker told me that Zed (the text editor) has an MCP client built-in and it's available on Linux.

paru -Sy zed-preview-bin
zeditor .

(To get MCP support you have to join the Agentic Editing Beta)

To configure Zed I had to give it a shell command with which to start the server. This is a bit odd as I want Zed to connect to an already running server - my app - but MCP, at the moment, doesn't support connecting to remote servers. This is a planned feature that's supposed to release by the end of the year - but I need it now.

Luckily, someone had the same problem as I and created mcp-remote.

With that I configure Zed by giving it the following command

npx mcp-remote@0.0.22 http://localhost:3000/mcp/sse --allow-http --skip-browser-auth

Fast MCP generates a sample tool that, given a User ID and a prefix, returns a greeting for that user. With that in mind I asked

Can you greet a User with ID 636166899?

To my surprise it didn't respond with a greeting but started digging through the codebase

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Claude digging through the code base to figure out how to greet a user

After some digging around I soon discovered that, while the server was running, it wasn't advertising any tools.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

My MCP server advertising 0 tools to the client

The problem was that the initializer for Fast MCP uses the descendants method to load all tools that inherit from ApplicationTool.

server.register_tools(*MCP::ApplicationTool.descendants)
server.register_resources(*MCP::ApplicationResource.descendants

That method returns all classes that inherit from a given class, but a caveat of that method is that it returns only descendants that have been loaded so far.

Foo.descendants
# => []

class Bar < Foo
end

Foo.descendants
# => [Bar]

Since, in development, Rails disabled eager loading no descendants of ApplicationTool are loaded when the initializer runs, and therefore no tools get registered with the server.

There are 2 easy ways around this - either enable eager loading in development (at least for this), or manually register each individual tool.

I opted to go with eager loading as we already had it configured to trigger when an ENV variable is set, we use that in some tests.

This has the downside of requiring a server restart whenever you want to test something, but as you also have to restart your MCP client so that it pulls in the new server changes this wasn't a big deal for me.

# config/environment/development.rb

Rails.application.configure do
  # Eager load all classes if the EAGER_LOAD ENV is set
  config.eager_load = !!ENV['EAGER_LOAD']
  # ...
end

So all I had to do is restart the server with

EAGER_LOAD=1 bin/rails s

I entered the same prompt again, and... it didn't work. It started searching through the codebase again.

After some more digging I concluded that, to test my MCP server I'd have to create a new Profile

= 768px) 520px, 50vw"> = 768px) 520px, 50vw">

Location of the Profile tab in Zed

= 768px) 520px, 50vw"> = 768px) 520px, 50vw">

The Configure Profile section of the Profiles tab in Zed

= 768px) 520px, 50vw"> = 768px) 520px, 50vw">

The new Profile button in Zed

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Naming a new profile in Zed

And then configure that Profile's tools to include only my MCP server's tools.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Configuring tools for the new profile

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Enabling individual tools from different MCP servers

With that configured I tried the prompt again and this time it worked!

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Claude sending a greeting to a User through the MCP server

Building a tool

From here the team and I spent the day building various tools that would allow us to automate common support requests.

I implemented a simple building search tool

module MCP
  class SearchBuildingsTool < ApplicationTool
    tool_name "search_buildings"
    description "Searches for buildings"

    arguments do
      optional(:building_ids).array(:integer).description("IDs of the Buildings to look up")
      optional(:query).maybe(:string).description("The search query")
    end

    def call(building_ids: nil, query: nil,)
      buildings = Building.all

      if building_ids.present?
        buildings = buildings.where(id: building_ids)
      end

      if query.present?
        buildings = buildings.where("name LIKE :query OR handle LIKE :query", query: "%#{query}%")
      end

      buildings
	    .limit(50)
	    .order(name: :desc)
	    .to_json(only: [:id, :name, :handle, :address])
    end
  end
end

that worked quite well

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Demo of the Building search tool

Then I wanted to implement Access Log search. But to search Access Logs you need a building ID. I wasn't sure if the LLM could combine multiple tools by first searching for the building, getting its ID and then searching for the access logs with that ID. And I couldn't find a definitive answer online so I hoped for the best and implemented the following

module MCP
  class SearchAccessLogsTool < ApplicationTool
    tool_name "search_access_logs"
    description "Searches through Access Logs and returns the fist 25 results"

    arguments do
      required(:building_id).filled(:integer).description("ID of the Building that we are searching in")
      optional(:query).maybe(:string).description("The search query")
    end

    def call(building_id: nil, query: nil)
      building = begin
        Building.find(building_id)
      rescue ActiveRecord::RecordNotFound
        return "Building not found"
      end

	  building
	    .access_logs
	    .search({ query: query })
	    .order(logged_at: :desc)
	    .limit(25)
	    .to_json(only: [
	      :id, 
	      :logged_at, 
	      :grantor, 
	      :description,
	      :access_point_id,
	      :status
	    ])
    end
  end
end

To my surprise this worked! It reused the ID it found in the previous response to search for the access logs.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Claude reusing previous result to solve new problems

I was curious if it could actually combine tools so I started a new chat and asked the same question.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Claude combining multiple tools in a single response to solve a problem

The response blew my mind. It could combine tools! To me, this was both incredible and scary. Kind of like seeing primates use tools or develop their own economy.

Takeaways

MCP is a very "loose" protocol. It does allow you to specify what inputs your tool has, which are required and which are optional, but it's not expressive enough specify what the input should look like.

For example, in the Access Logs search I didn't specify what the query string should look like - my mistake - so the LLM just takes its best guess when you ask it something complex like "Can you give me all access logs created in the last hour in the building Crimson?".

I expected it to get all the access logs and then filter them on its own, but that's not what happened. Instead it passed "logged_at:>=2025-05-05T07:36:31" as the query. And, honestly, I can't blame it.

I later refined my tools with very detailed descriptions like this one

module MCP
  class SearchAccessLogsTool < ApplicationTool
    tool_name "search_access_logs"
    description <<~MD
      Searches for Access Logs in a given building and returns
      a JSON string containing an array of up to 25 objects.
      The list is sorted in descending order by the time 
      the access log was logged.
      Inputs:
      - building_id [Integer] (the ID of the building in which to search for access logs)
      - query [String, null] (a plain string that can contain any part ot the name of the grantor, the id of the access log, the description, the id of the associated access point)
      - logged_after [String, null] (An ISO8601 timestamp of the earliest time a log was recorded)
      - logged_before [String, null] (An ISO8601 timestamp of the latest time a log was recorded)
      - statuses [Array[String], null] (A comma-separated list of statuses)
      Output:
      - id [Integer] (The ID of the access log record)
      - logged_at [String] (ISO8601 timestamp of the time the log was recorded)
      - grantor [String, null] (Name of the person that granted this access, a null value means that the system granted the access)
      - description [String] (A description of what happened)
      - access_point_id [Integer] (ID of the access point that was accessed)
      - status [String] (The access point's state after this access)

      Statuses:
      - open - means that the access point was unlocked and that someone opened it
      - closed - means that the access point was unlocked, opened and then closed
      - left_ajar - means that the access point was unlocked, and then left open for more than 1min
      - denied - the access point never unlocked
      
      If the building couldn't be found you'll get a message back stating that.
    MD

    arguments do
      required(:building_id)
        .filled(:integer)
        .description("ID of the Building that we are searching in")
      optional(:query)
        .maybe(:string)
        .description("The search query which can contain any part of the grantors name, id of the record, id of the access point, or part of the description; any other input will be ignored")
      optional(:status)
	    .maybe(:string)
	    .description("Comma-separated list of statuses - valid statuses are open, closed, left_ajar, denied; any other statuses will be ignored")
    end

    def call(**args)
      # ...
    end
  end
end

The more information that I gave the LLM the better the result was. Something to watch out for is that you:

Describe the output (it's as important as describing the input)
Describe error states
List all possible enum values and describe what each value means
Explicitly state what format you expect for an input (what kind of timestamp do you expect, can it be nullable, etc.)
Explicitly state what's not allowed (e.g. additional enum values besides the ones listed)

One more thing. With smaller LLMs, like Llama 3 7b, I had to normalize input values. E.g. I'd tell it that I expect "building" as an enum value but it would pass me "Buildings". Or I'd tell it that a field is a nullable String but it would still pass an empty string when it had no value. But Rails makes it really easy to accommodate for such mistakes.

value.presence&.to_s&.strip&.singularize&.underscore

Conclusion

Integrating our existing app with AI was surprisingly easy. We just had to expose tools and the LLM would figure out how to use them. In 8 hours we managed to set up an MCP server, configure our clients, and expose enough tools to speed up a dozen common support requests, which is amazing.

In this hackathon, being a fully remote team with members in the EU and the US felt like a superpower. The EU team solved all the setup problems, so by the time the US team came online they could jump straight into building tools and testing ideas. For the price of 8 hours we got 14.

I still want to explore how to add permissions and some guard rails so that the LLM can't do too much damage if it goes haywire. I intentionally skipped that during the hackathon as it wasn't important.

This project was a technical success, but I'm not sure if it's the right direction to go in. This integration makes it so that developers get rarely involved in some types of support requests. As a developer, support is less about solving a problem and is more about figuring out customers' pain points and how to make the overall experience better.

I'm not sure how to strike a good balance between automating laborious tasks and surfacing common problems. That's also something I'll have to explore.

Building Simpl息

2025-04-24T06:00:00Z

This week I went for a coffee with a friend. As we talked the topic of hobbies came up. I mentioned that I was having a lot of fun working on a toy project over the past week, and then I showed it to him.

Demo

Him: So you are dabbling in React?

Me: No, this is just Rails.

Him: Which libraries did you use?

Me: None, this is as vanilla as it gets.

Him: What!?

Backstory

For the past 12 months, each month I built an app within a strict time-box of 12 hours.

If the project went overtime, and wasn’t at most 4 hours away from completion, I’d abandon it. If a project failed, and a lot of them did in the beginning, I’d reflect what led to that and tried to fix it in the next one.

My goal is to practice pragmatic decision making and to curb my perfectionism - which has started to show again.

The time-box really helped me make better decisions. At first all my apps failed because I either over-estimated my speed, or under-estimated the complexity and unknowns. But around the third one I got better at both. Then came the hard part - learning to strike a balance between quality work, functionality, and not obsessing over details.

Simpliki

Simpliki is the 12th app I’ve built.

It’s an app for practicing diaphragmatic breathing.

I got the idea for it from my girlfriend. She uses a similar app in her mindfulness practice.

The name is a combination of “simple” and the Japanese word “iki”, which means “breath”.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Google Translate result for translating "breath" to Japanese

You can practice breathing with Simpliki at simpliki.app. And you can peruse the source-code on GitHub.

Starting from the center

The best way to finish a project is to start from the center and work your way from there. I focus on what makes the app work - what makes it what it is - and tackle everything else later. That way I can cut less important bits if I have to.

For Simpliki the center is the breathing exercise - the place where you practice diaphragmatic breathing - without that there is no Simpliki.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Simpliki's center

But I also need a way to get to the breathing exercise - an index screen or something similar.

Personally, when I want to relax and practice mindfulness I don’t like to have too many options to choose from - it takes me out of the experience. So I decided to add a “home” screen with a single button that links to a random exercise.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Simpliki's two centers

My girlfriend likes to practice specific exercises so I added an “Explore” button that leads to an index page with all exercises.

An app can have many centers interconnected with different features. I think of the exercise and the home screen as two separate centers, and the index is just a feature between them.

I learned this from Christopher Alexander's design theory. The most apt analogy I found is in a paper that compares this design principle to patterns of a carpet.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Simpliki's two centers and the features between them

Now that I have an idea what I'm building I'll create a breadboard so that I can work out problems on paper instead of experimenting in code. I skipped this step for Simpliki as it was, well, simple. But the breadboard would look something like this.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The breadboard

This also helps me figure out the domain language - what models I have and what their attributes and methods are called.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The domain

Design

I had a clear vibe I wanted to go for - Endel + Headspace. I like Endel's minimalist and monochrome style - it has a mystical, simple, and concise vibe that I wanted to replicate. And I find the organic, whimsical, animated shapes and play button from Headspace calming - exactly the vibe I want for a mindfulness app.

= 768px) 520px, 50vw"> = 768px) 520px, 50vw">

Endel

= 768px) 520px, 50vw"> = 768px) 520px, 50vw">

Headspace

So I decided to combine the two for Simpliki. I'd use Endel's color-scheme and the animated organic shapes from Headspace.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Simpliki's UI

Starting a new project

I decided to use Tailwind because I like working with it. And I chose SQLite as the database because it's the simplest one to use out of the presets. I honestly don't even need a database for this app, but it's easier to use one than not.

rails new simpliki --css=tailwind

Then in that project I generated my models and a controller

bin/rails g model Exercise name:string

bin/rails g model Exercise::Step exercise:belongs_to action:string duration:integer position:integer

bin/rails g scaffold_controller Exercise

I removed everything that I didn't need, added a "home" action to the controller, a view for that action, and pointed the root route to that action.

Then I created a few fixtures for Exercises and Exercise::Steps.

Over the past 5 years I've completely changed my mind about fixtures. I used to think that they are finicky and convoluted compared to factories. But I've come to like them more after I've seen the downsides of factories over the years - they are extremely slow, it's hard to manage what will get created or you have to define every single record you want to create in every single test.

My favorite advantage of fixtures is:

bin/rails db:fixtures:load

This will take all my fixtures and put them in my current (development or production) database.

Now I don't need to create a way to add or edit Exercises, or add and edit them manually through the console, before I can start working on the show page. I can just load in a few exercises and use them. Best of all, I get to see and use the exact same data that's in my tests - no more trying to figure out what went wrong, I can just open the same Exercise and see.

Then I started working on the views. At first I just added links between the pages, added a header that said "Simpl息" on each page, added a plain HTML "Practice" button on the exercise show page, and created a very simple index page that shows a list of exercises.

The ruby tag

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Simpliki's logo

Then I remembered reading about the HTML ruby element which is used to show the pronunciation of characters. I thought it would be a nice way to show where the app's name comes from.

<h1 id="logo" class="font-serif">
  <span class="mt-4">Simpl</span>
  <ruby>
	  息
	  <rt class="text-gray-500">iki</rt>
  </ruby>
</h1>

It took about a minute to add and style the "iki" - plain HTML is amazing.

Animating SVGs

Next I wanted to animate the play button. I've done similar things before so I was quite confident that I could do it quickly.

Demo of the circle animation

The basic idea is to have an SVG with a single path that consists of 16 points with Bezier curves that form a circle. The number of points determines how organic and uneven the circle will look in the end - 8 is too blocky, 24 is too smooth, 16 is just right.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Basic SVG circle with 16 points and a Bezier curve

Then, every 10ms I change the position of each point slightly and adjust their Bezier curve points to animate the circle.

To make it undulate - move like there are ripples going across it - I can pull or push one point after another in or out, and then adjust the Bezier curve points to make it look hand-drawn.

That's an old computer science trick I learned in college. It sounds complicated, but it's quite straightforward.

There is just one thing you have to know - if you plot x = cos(a) and y = sin(a) where a is any value between 0 and 2PI you'll get a circle of radius 1. Then to get any size circle you just multiply everything by the radius you want.

const NUM_POINTS = 16
const RADIUS = 100

function generatePoints(time) {
  const circlePoints = []
	
  for (let index = 0; index < NUM_POINTS; i++) {
    // Calculate at what angle the point is supposed to be
    const angle = (point.index / NUM_POINTS) * Math.PI * 2
	
    // Convert the angle to coordinates
    const x = RADIUS * Math.cos(angle)
    const y = RADIUS * Math.sin(angle)
	
    circlePoints.push([x, y])
  }

  return circlePoints
}

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The most basic circle

To form waves, I can change the radius for each point using any function that also includes COS and SIN. That way each point moves in its own little circle which, when put all together, looks like a wave. And to make it consistent the SIN and COS have to include the current time.

const NUM_POINTS = 16
const RADIUS = 100
const RIPPLE_FREQ = 0.5 // determines how quickly the waves move
const RIPPLE_AMPLITUDE = 5 // determines how big the waves are

const circlePoints = []

function generatePoints(time) {
  const circlePoints = []
	
  for (let index = 0; index < NUM_POINTS; i++) {
    // Calculate at what angle the point is supposed to be
    const angle = (point.index / NUM_POINTS) * Math.PI * 2

    // Generate a new radius
    // the values were completely arbitrarily chosen
    // you could also just use sin(time) + cos(time)
    // this just looks nice to me
    const rippleY = Math.sin(time * RIPPLE_FREQ + angle)
    const rippleX = Math.cos(time * RIPPLE_FREQ * 0.7 + angle * 1.3)
    const ripple = (rippleY * 0.6 + rippleX * 0.4) * RIPPLE_AMPLITUDE
	
    // Convert the angle to coordinates
    const x = RADIUS * Math.cos(angle) + (ripple * Math.cos(angle))
    const y = RADIUS * Math.sin(angle) + (ripple * Math.sin(angle))
	
    circlePoints.push([x, y])
  }
  
  return circlePoints
}

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Circle with ripples

Then I can just move the bezier curve anchors in the same way to get a hand-drawn look.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Circle with ripples and fixed Bezier curves

The end effect is subtle, but it makes the circle look friendlier.

View transitions

Next I animated the view transitions. This takes maybe 10 minutes but makes the app feel way more modern and responsive.

First I enabled view transitions in Turbo Drive by adding this meta tag to the head section of my layout

<meta name="view-transition" content="same-origin" />

Just adding this one line will already add a fade effect to all page transitions. Then to move individual elements around, like the logo, I have to add it a view-transition-name CSS property with a unique name.

#logo {
  view-transition-name: logo;
}

After that the browser animates the transitions on its own.

Demo of the logo moving with page navigation

For the explore page I wanted to add an animation where the list slides in from the bottom of the page. Luckily this can be done with a few lines of CSS.

@keyframes slide-to-top-from-out-of-view {
  0% {
    transform: translateY(100vh);
    opacity: 0;
  }
  100% {
    transform: translateY(0);
    opacity: 1;
  }
}

#exercises {
  view-transition-name: exercises;
}

html:not([data-same-page-visit])[data-turbo-visit-direction="forward"]::view-transition-new(exercises) {
  animation: 0.25s ease-out slide-to-top-from-out-of-view both;
}

This tells the browser to apply a slide-to-top-from-out-of-view animation for any new exercises list that appear.

Demo of the exercise list sliding in from the bottom of the screen

There is one caveat, View Transitions aren't supported on Firefox. This only works on Chromium and WebKit based browsers. As one of the 2% of global Firefox users I don't mind, the app still works without the transitions.

The breathing animation

Now that I have an animated circle I want to create a breathing animation to help people pace their breath.

The circle should grow when you have to breathe in, and shrink when you have to breathe out.

Demo of the breathing animation

To animate this I need to create three circles - a maximum indicator, a minimum indicator and a progress indicator.

The maximum indicator is static and just has an opacity of 10%. The minimum indicator is mostly static after an initial animation that moves it in place. And the progress indicator constantly shrinks and grows as needed.

But rendering one animated SVG at 60 FPS is already quite taxing, and synchronizing three undulation animations would be a headache. Luckily plain HTML saves the day again with SVG symbols.

In an SVG you can define any set of paths or shapes as a symbol and give it an ID. Then you can render the same symbol in any other SVG in the same HTML with the use tag. You can even change the fill and stroke properties on the use tag to get different versions of the same shape. And when you update the shape in the symbol it immediately reflects in all other SVGs using that symbol.

<!-- Defines a symbol "foo" as a circle -->
<svg xmlns="http://www.w3.org/2000/svg">
  <symbol id="foo" viewBox="0 0 100 100">
    <circle cx="50" cy="50" r="50" />
  </symbol>
</svg>

<!-- Renders a red circle with a black stroke -->
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100">
  <use href="#foo" fill="red" stroke="black" stroke-width="2"></use>
</svg>

<!-- Renders a blue circle -->
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100">
  <use href="#foo" fill="blue"></use>
</svg>

With a symbol I just have to animate one SVG and then I can reuse the same animated shape in any other SVG on the same page. I've created a helper to make things more readable.

circle_id = dom_id(@exercise, :circle)

# Renders an animated circle as a symbol with the id circle_id
animated_circle(symbol: circle_id, class: "hidden") +
# Renders the minimum indicator
animated_circle(use: circle_id, class: "absolute inset-0 z-30 w-full select-none", stroke: "black", stroke_width: 2, name: "exercise-circle-min", data: { exercise_target: "minCircle" }) +
# Renders the progress indicator
animated_circle(use: circle_id, class: "absolute inset-0 z-20 w-full select-none", data: { exercise_target: "progressCircle" }) +
# Renders the maximum indicator
animated_circle(use: circle_id, class: "absolute inset-0 z-10 w-full select-none opacity-10", name: "exercise-circle-max", data: { exercise_target: "maxCircle" })

With that out of the way I now have to animate the progress circle. I read about the Web Animations API a few weeks ago and it seemed like just the right tool for the job.

It was my first time using that API and I think I hit every edge case there is with it.

Long story short, don't use fill forwards and instead manually set your desired state with the onfinish callback. For some reason, fill forwards applied a new CSS rule every time an animation ran. After a few minutes I had a hundred animation style blocks in my Inspection window, all overriding each other. Also, remove any transitions on the animated element as that will make the animation wonky in Firefox (but works fine everywhere else) - the browser will handle the transition for you.

After I figured that out, the rest was straightforward

async function inhale(duration, element) {
  return new Promise(resolve => {
    const animation = this.progressCircleTarget.animate([
      { transform: `scale(0.25)` },
      { transform: `scale(1.0)` }
    ], {
      duration: duration * 1000,
      easing: "linear"
    })

    animation.onfinish = () => {
      this.progressCircleTarget.style.transform = `scale(1.0)`
      resolve()
    }
  }
}

async function exhale(duration, element) {
  return new Promise(resolve => {
    const animation = this.progressCircleTarget.animate([
      { transform: `scale(1.0)` },
      { transform: `scale(0.25)` }
    ], {
      duration: duration * 1000,
      easing: "linear"
    })

    animation.onfinish = () => {
      this.progressCircleTarget.style.transform = `scale(0.25)`
      resolve()
    }
  }
}

Now I just had to pass the exercise steps to the Stimulus controller, and run the animation. This just meant that I had to define a steps value of type Array and serialize the steps as JSON on the HTML element.

Sound

With the animation done I wanted to also add an audio cue that you should breathe in or out.

At first I thought that the simplest way to add sound was to play some sound file - like a gong or a flute - at different playback speeds for different durations, and pitch it down to cue and exhale.

This turned out to be a dead end. I could change the playback speed but at 0.5x playback the sound of a gong starts to sound like Hell's bells.

Then I remembered the Web Audio API with which you can generate sounds. I found a nice tutorial that explains how to make a bell sound, but I was out of my depth here and decided to reach for an LLM so that I don't run out of time.

I created a scaffold class with the interfaces I wanted to have and asked ChatGPT to implement the class so that it generates "a nice and relaxing sound that inspires breathing in a mindfulness app I'm working on, kind of like Endel or Headspace". After a bit of back and forth I had something that I thought was decent.

But it didn't work on my iPhone... Turns out that Apple disables Web Audio playback if your ringer is set to silent, even though the volume is up - odd choice, but ok.

You can listen to the sound at simpliki.app, just pick any exercise and start it. If you are on an iPhone be sure to set your phone OFF silent.

Scroll snapping

As I was working on the breathing animation I noticed that I forgot to record if you are supposed to breathe with your nose or mouth, so I had to go back, add a migration, and update the fixtures.

Then I thought that showing a short description about the exercise would be nice so I added ActionText, and a few fixtures for it.

bin/rails action_text:install

But now that the exercise page could have an overflow and users would have to scroll I wanted the scrollbar to snap if you were close to the top of the screen. That way the exercise circle would always be in focus and you can't accidentally scroll away.

Once again HTML, or rather CSS, saves the day - this time with scroll-snap.

I just added the snap-y and snap-mandatory Tailwind classes to the container, and snap-start to the container of the exercise circle - that's it. Now the browser snaps to the top when you scroll near the top of the page.

Demo of the page snapping to the top when you scroll near the top

Full text search

Since I had four hours left at this point I decided to add full-text search to the index page.

This too was surprisingly easy thanks to an article by Sergio Alvarez. All I needed to do was to create a special kind of virtual table

class AddFullTextSearchToExercises < ActiveRecord::Migration[8.0]
  def change
    create_virtual_table :exercise_full_text_search_vectors, :fts5, %w[name description]
  end
end

Populate it and then query it. I created a concern under the Exercise class that does that. That way the search logic is nicely self-contained.

# frozen_string_literal: true

class Exercise
  module Searchable
    extend ActiveSupport::Concern

    DEFAULT_LIMIT = 25
    VECTOR_TABLE_NAME = FullTextSearchVector.table_name

    included do
      has_one :full_text_search_vector, foreign_key: :rowid, dependent: :destroy

      after_save :create_or_update_full_text_search_vector
    end

    class_methods do
      def search(query)
        return all if query.blank?

        where("#{VECTOR_TABLE_NAME} MATCH ?", escape_fts_match_value(query))
          .joins(:full_text_search_vector)
          .order("bm25(#{VECTOR_TABLE_NAME})")
          .distinct
      end

      def escape_fts_match_value(input)
        input.to_s
          .gsub('"', '""')
          .split(/\s+/)
          .map { |word| %Q("#{word}") }
          .join(" ")
      end

      def create_or_update_all_full_text_search_vectors
        all.find_each(&:create_or_update_full_text_search_vector)
      end
    end

    def create_or_update_full_text_search_vector
      params = {
        id: id,
        name: name,
        description: description.to_plain_text
      }

      sql = if full_text_search_vector
        ActiveRecord::Base.sanitize_sql_array(
          [
            "UPDATE #{VECTOR_TABLE_NAME} SET name = :name, description = :description WHERE rowid = :id",
            params
          ]
        )
      else
        ActiveRecord::Base.sanitize_sql_array(
          [
            "INSERT INTO #{VECTOR_TABLE_NAME} (rowid, name, description) VALUES (:id, :name, :description)",
            params
          ]
        )
      end

      self.class.connection.execute(sql)
    end
  end
end

With that I can do a full-text search across all exercises

Exercise.search("box")

Pagination

Now that I had search I wanted to add pagination. Years ago while exploring what Strada/Turbo Native was before it was released I saw that 37signals has a really neat way of handling pagination.

They have a "pagination" controller that goes and fetches the HTML of the next page, parses it to get the contents of itself from the next page, and then adds that contents to the current page.

No need for special routes or turbo-stream handlers, you just render the paginated page like always and let the controller do the rest. So I tried to recreate that.

First I tried to fetch the next page and parse it using DOMParser to get the next page's document. My first stab worked quite well.

  async #fetchNextPageDocument() {
    const url = this.nextPageLinkTarget.href

    const response = await fetch(url, {
      headers: {
        "Accept": "text/html",
      }
    })

    if (response.ok) {
      return new DOMParser().parseFromString(await response.text(), "text/html")
    } else {
      console.error("Failed to fetch next page:", response.status)
      return null
    }
  }

Then all I had to do was append the children from the new document to the current one

	const doc = await this.#fetchNextPageDocument()

    const newList = doc.querySelector(`[data-${this.identifier}-target="list"]`)

    Array.from(newList.children).forEach(child => {
      this.listTarget.appendChild(child.cloneNode(true));
    });

    const newNextPageLink = doc.querySelector(`[data-${this.identifier}-target="nextPageLink"]`)
    if (newNextPageLink) {
      this.nextPageLinkTarget.replaceWith(newNextPageLink.cloneNode(true))
    } else {
      this.nextPageLinkTarget.remove()
    }

And it worked like a charm. I just render the next page like always and let the controller do the rest.

Demo of the pagination controller loading in more exercises

Auth, create, update & delete

Since I still had time left I decided to implement CRUD for exercises. Here I decided to use the new auth generator. It gave me Users and Sessions, all I now needed to do was add views.

I quickly wrote a nested form Stimulus controller, some CSS to style the inputs and called it a day.

Demo of the sign in process and CRUD actions

Conclusion

I wanted to share how I built Simpliki because everyone I show it to seems to think that it's some SPA made with the latest technology trend when in reality it's just a vanilla Rails app that uses a few browser APIs.

Modern browsers are capable of a lot that just a few years ago required cutting edge technology and all the complexity that comes with it.

But I also wanted to sing the praises of time-boxing your work and working on toy-projects. It helped me develop a healthier way to think and break an old habit.

Adversarial Web Services

2025-03-03T07:00:00Z

My first job was at a digital agency. In addition to app development we also hosted the apps that we'd develop for our clients. This allowed us to avoid fiddly managed hosts, and since we ran everything on a beefy bare-metal server we didn't have to manage dozens of machines at different providers. All in all, this was a good setup.

Then, one Friday morning, I came into the office and went to clock in using an internal app. But the app just timed out. We soon noticed that all internal apps and our client's apps were timing out, our bare-metal server was completely unresponsive, and then came the bombshell - both hard drives in the server had failed.

Luckily we had hourly backups. Or so we thought. We found out the hard way that our backup script had a bug that caused it to create empty backups for the past few months. So we had no viable backup to restore from.

We spent the next 22 hours provisioning a new machine, re-deploying projects to it, recovering databases and files from the old machine and uploading them to the new one. By some miracle we managed to recover everything.

This incident left a sour taste in everybody's mouth. Everything was back to normal, but nobody was confident that it would stay that way for long. We lost confidence in our tools and ourselves. The next week we were told that we’d be abandoning our bare-metal server and would move all our apps to AWS so that we wouldn't have to monitor the hardware and backups were built-in.

Since that Friday, for more than a decade now, I've used AWS at every job I had.

I've spent countless hours working with its services, configuring and misconfiguring them, optimizing them for performance and cost. And in my opinion it just isn't worth it for most people.

AWS makes things easy, not simple

Some people think that AWS-hosted services are somehow magical 1-click solutions that you pay for and forget about. But the truth is far from that.

While you can set up a MySQL/Redis/Kafka instance in just a few easy clicks, after that you have to configure, tweak, monitor and manage that service like you would if you'd host it yourself.

In other words, the complexity is the same, but the setup is easier.

In fact, it's even more complex than running your own MySQL/Postgres/Redis/... server because AWS introduces its own arbitrary limitations on what you can and can't do. And on top of that it also introduces new concepts that you'll have to learn and manage - Floating IPs, Load Balancers, CPU credits, Network credits, IOPS credits, EBS credits, VPCs, Security Groups, IAM roles...

If you don’t learn these things, you will eventually either encounter a problem you can’t fix or receive an exorbitant bill from AWS.

Probably the best example of this is EC2. You can set up an instance in a minute. And you can probably deploy something to it with minimal knowledge of Linux and/or Docker. But what will you do, for example, if you run out of IOPS credits? Or if Docker refuses to start after a mandatory update? Or your instance suddenly changes its IP address?

Setting up a VPS is extremely easy, but if you don't know Linux, networking and some AWS gotchas, but managing and monitoring the instance is still complex.

Think of it as sending an email. It's simple, all you have to do is open up Gmail, write a message, enter a recipient and hit send. But under the hood what is actually happening is a complex exchange between email servers that involves IMAP, DKIM, SPF, DNS, Spam detection, Spoofing detection, reputation management, firewalls, storage, and more.

I hope that some of the acronyms and concepts of the complexity of email servers are foreign to you because that's my whole point.

By buying email from Gmail you are giving Google money so that you don't have to deal with the complexities of running an email server. But at AWS you are giving Amazon money to get your own server. They will set it up for you, but then it's up to you to deal with the complexity of running that server.

Take everything with a grain of salt

One of the biggest mistakes I ever made was to trust AWS marketing at face value. Their marketing often omits important or inconvenient details.

Figuring out what you are really buying requires you to read the product's documentation - not one page, not the pricing page, but all of it - and even then you have to take it with a grain of salt.

Take the incident I wrote about 2 weeks ago. At work we use ElastiCache Redis as a PubSub provider for our WebSockets. One day all our devices started disconnecting randomly. After weeks of investigation it turned out that we hit a CPU cap even though the instance claimed it was at 10% CPU during the incident. This happened because we were using a node type that's capped at 10% CPU which is, conveniently and vaguely, only mentioned in the documentation while the pricing page says that we get 2 full vCPUs.

Or how API Gateway offers WebSockets. But you can't have a connection that lasts longer than 2h, and you can't reliably know if a client has disconnected.

Or how Aurora offers unprecedented speed and infinite storage. But it might corrupt your database. And high load on a read-replica degrades the primary's performance negating the benefit of having the replica.

There are many more examples.

But the gist of the story is that you have to carefully navigate AWS' documentation labyrinth - and it really is a labyrinth - to gather as much information as you can before you commit to a purchase. And even then you aren't 100% sure what you've purchased or how much it will cost you in the end.

Some might say that this is a "skill issue", or just "RTFM". But I think that people who say this are either stuck in an inferiority complex or are just coping.

Imagine if any other business sold stuff like AWS does. Let's take a restaurant as an example.

You go to a restaurant and want to order a pizza. The waiter walks up to you and tells you what pizzas they have. There are a few you like so you ask for the price and the waiter brings you the menu. You see that the pizza you like costs 15€ for a large so you order it. Then a short while later the waiter brings you a single slice of the pizza. You ask him where's the rest of the pizza and he points out that on the back of the menu it says that some pizzas come with limitations mentioned in the "limitation list". Then he brings you the limitation list which states that the pizza you ordered is served per slice.

You'd probably never go to that place ever again.

Sometimes it can feel like racketeering

After using AWS for a while I've noticed that it isn't as elastic as it claims to be, and somehow that feels intentional.

Let's take the WebSocket disconnect incident again. We were using an instance that has 2 vCPUs but can only use 10% consistently and we decided to upgrade to an instance that can use 100% of the CPU available to it.

That’s a 10x increase in compute, so we expected a 10x increase in price - and that’s exactly what we got. The monthly price for our instance went up from about 10€ to about 100€.

But we soon learned that, while we did get 10x the CPU, we got only 1/5 of the bandwidth.

So we upgraded to the first node type that offered the same bandwidth as before and our monthly cost rose to 200€ per month.

That's a 20x increase in price for 10x the compute.

Never mind the exorbitant price for what is essentially a managed EC2 instance that otherwise costs 60€. What bugs me the most here is that we're talking about a virtual machine, not a bare-metal server. You can give it as much CPU, memory and bandwidth as you want.

Nearly all AWS services give you a predefined list of instance types to choose from - this is not just an ElastiCache issue.

I understand why smaller hosting providers make you choose from predefined instance types - they simply don't have the hardware to allow anybody to choose what they need. But a hyperscaler like AWS? One that calls their VPS offering Elastic Cloud Compute?

Another thing I noticed is that some services have conveniently set, completely arbitrary, limitations that seem to exist only to make AWS more money.

For instance, API Gateway imposes a maximum duration on a WebSocket of 2h and will close any idle connections after 10min. But at the same time AWS charges you per WebSocket message in addition to each minute each client was connected, and most applications will need at least 1 message after a reconnect.

If you have 50k clients you'll pay AWS 200€ every month just to keep them connected to a WebSocket 24/7. For comparison, a 60€ EC2 instance can handle more clients than that with no limitations.

To go back to the restaurant analogy. Imagine if you bought a Pizza at a restaurant and the waiter starts bringing you your pizza one slice at a time, because they decided to do so for whatever reason. Then, at the end of the meal, they charge you a delivery fee for every individual slice on top of the price of the pizza.

The pizza would have to be out of this world for me to come back to that restaurant.

When you run into such pricing practices AWS doesn't feel like a partner helping you build stuff anymore; it feels like an adversary trying to squeeze every penny out of you.

Serfs and sovereigns

My biggest gripe with AWS are its proprietary services. Some of them are great and novel while others are OK equivalents of existing open-source services.

When it comes to great and novel services, the first thing that comes to my mind is S3.

You can get S3-compatible object storage at a dozen places today - DigitalOcean Spaces, Backblaze B2, Cloudflare R2, Hetzner's, OVH's and Scaleway's Object Storage, or open-source Garage and MinIO just to name a few - but that wasn't always the case.

A little over 10 years ago S3 reigned supreme when it came to storage. The only real alternative to it was to either store the files locally on your server or using a network attached drive, both of which required some skill to scale and manage - at least compared to S3.

Back then, it was common for people to e.g. use a managed host but still have an AWS account just to get access to S3. Problem was, we were completely dependent on AWS for object storage. If they raised prices we had no real option but to pay. There was no contingency if S3 had an incident you just had to deal with the customer backlash.

When other S3-compatible object storage providers started popping up not only did prices come down but you could suddenly choose a provider that suits your need - no ingress fees, no egress fees, cheaper long-term storage, faster storage, HIPPAA-compliant storage, etc. You could mix and match providers to suit your need, or use multiple as a contingency.

But most importantly, any code I write today that fetches or uploads files to S3 will work just as well anywhere. This means that, if I don't like a price hike or a term change from my current object storage provider, I can just pick up my stuff and move to another provider that offers more favorable terms.

When it comes to S3 - 10 years ago we were basically serfs, today we are sovereign and I don't miss the old days one bit.

There are other great AWS services - like DynamoDB - but I avoid them in favor of open-source alternatives just because I don't want to find myself locked in and having to accept any term change to avoid spending days switching to something else.

Competence

The incident that happened at my first job wasn't caused by our hosting provider, it was caused by us. Our hosting provider at the time gave us all the tools and services we needed to prevent that incident from happening, we just didn't know how to use them. It was our incompetence that caused the incident.

As a knee-jerk reaction we migrated to AWS chasing a promise, or rather a dream, that it will somehow save us from our incompetence - it didn't. All it did was mask that incompetence by making things easy to set up, but our old problems all eventually reoccurred.

Only by learning to run my own server through learning Linux, Docker, networking, monitoring, backups and server security did I manage to avoid making the same mistakes - not just with AWS but at any host.

And that's the take-away here - AWS, or any other managed service provider, won't save you from having to learn how to manage that service. They will give you monitoring, but you still have to know what the metrics mean and what's acceptable for you. They will give you hardware, but you have to know what's enough for you. They will give you firewalls and other networking tools, but you still have to configure them for your use-case.

All you need to learn this stuff is a Virtual Machine or a Raspberry Pi and some time. Set up a small server and try to configure a firewall, install a database, run an app, upgrade the database. If you fail, and you probably will, just re-install and try again.

One of the best decisions I ever made was learning how to self-host the services I use at work. It was difficult at first - as going from 0 to 1 always is - but it made me much more confident and better at my job. I now understand not only how to operate these services but also how to optimize my code to make the most of them. And best of all is that these skills carry over to any hosting provider, project and job.

If you are willing to learn, I think you will be better off renting a few VPS instances or bare-metal machines from the likes of DigitalOcean, Hetzner, OVH, and Scaleway. The predictable pricing, and lack of AWS BS, will save you time and money. Or if you don't want to learn all this, then go with a hosting platform like Heroku or Fly.

Monitoring ActionCable

2025-01-27T10:26:00Z

A few weeks ago I was woken up with the following message "Hey, devices are randomly disconnecting from WebSockets can you come online and help?".

Usually, when a widespread problem like this occurs out of the blue, my first goal is to make the app mostly operational again and my second goal is to fix the cause of the problem. To do that I usually go over each component that could be causing this to check if it's functioning correctly and then drill down on any components that aren't.

Step one, identify which components could be at fault.

We use Rails' ActionCable for WebSockets. I've done a deep dive into ActionCable in a previous article if you'd like to read more about it.

ActionCable, except for the Ruby process itself, depends on Redis (in my case) and on the database(s) that the app uses.

Step two, prioritize.

Since I knew that the devices were randomly disconnecting I assumed that either app containers were restarting for whatever reason, or that something was causing ActionCable's worker pool to lag which then caused missed heartbeats.

This means that the Ruby process is the most likely culprit, then the database, and finally Redis.

Step three, check each components.

I checked Grafana to see what's going on with the app containers and saw that CPU was elevated but wasn't exceeding 80%, while memory didn't go above 70%, and the auto-scaler was adding more containers as needed. Everything was in order given that a reconnect storm was going on.

Then I checked Elastic APM to see what was going on with /cable. But the metrics were phenomenal - sub-millisecond response times and 100% successful responses. Fantastic numbers for a reconnect storm.

Next I checked the database. CPU and memory were at average levels, there were no slow queries. Everything was fine.

Lastly, I checked Redis. To my surprise, here too, everything was fine. The CPU never exceeded 10% and memory was at 5%.

What was going on?

Step four, stabilize the app.

Since Redis and the database were fine, in an Hail Mary attempt, I scaled-out the number of app containers to stop the disconnects. Surprisingly, this worked... This pointed a spotlight at the application code and ActionCable. They must have been the faulty component since the scale-out fixed the issue - but how?

Step five, find the root cause.

At that point I realized that I had no clue what was going on with our WebSockets after they were established. Elastic APM only captures metrics about the HTTP part of the WebSocket connection, but once it was established I had absolutely no insight into what's going on - how many clients there are, what the latency was like, how many messages were being sent...

The best way to figure that out was to introduce some kind of monitoring. But what should I monitor and how should I surface those metrics?

I'm not one to reinvent the wheel, so I checked if some tool for monitoring ActionCable already exists. I found out that Sentry's and AppSignal's APM tools monitor ActionCable's action execution times.

Action Execution Time

Action execution time measures how long it takes for a Channel to process a received message. This metric is important because it can show if you have a slow action which binds up the worker pool for too long and thus lowers throughput and increases latency. It's a nice way to quickly see if everything is alright with ActionCable.

I quickly exposed that metric to Elastic APM using the following code.

# config/initializers/elastic_apm.rb

ActiveSupport::Notifications.subscribe "perform_action.action_cable" do |event|
  transaction = ElasticAPM.start_transaction "#{event.payload[:channel_class]}##{event.payload[:action]}"
  transaction.start(event.time * 1000.0)
  
  ElasticAPM.end_transaction(event.payload[:exception].blank?)
end

That worked!

PubSub latency

I soon noticed that some extra metrics, like the PubSub latency, would be helpful. Knowing the PubSub latency would help me identify problems with the PubSub component of ActionCable. If the latency - the time it takes for a message to be sent to clients since it was broadcast from the app - goes up that means that the PubSub service, in my case Redis, is probably the cause of a problem.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Diagram that shows what PubSub latency measures

To measure that latency I have to know how much time has passed since a message was broadcast to when it was received in the ActionCable server. The easiest way to do that is to broadcast a message with the current time in it, then receive it in some channel and calculate the time difference between the time I received the message and the timestamp in the message.

To send the a message with the current timestamp I can do the following and run it from a cron job every minute

measurement_payload = {
  sent_at: Time.now.to_f
}

ActionCable.server.broadcast("metrics", measurement_payload)

To receive the message and calculate the latency I can do this

stream_from("metrics", proc do |json|
  payload = ActiveSupport::JSON.decode(json)
  sent_at = payload[:sent_at]
  latency = Time.now.to_f - sent_at
  
  Rails.logger.debug "PubSub Latency: #{latency}"
end)

But where can I put this code and how do I expose the latency?

After some experimentation I decided to create a stream from every open channel in ActionCable. This sounds pretty bad, and I'd like to find a better way to do this, but it has some upsides and the downsides can be minimized. The upside is that no measurements are performed if there are no clients. The downside is that each client has its own metrics subscription even though only one subscription per server is needed.

There are many ways to limit the subscription to be one per server and I opted for the simplest one - a mutex. A mutex, aka mutually exclusive lock, allows me to ensure that just one thread is running a piece of code at a time. In Ruby, you'd usually use Mutex#synchronize like so:

# Create the mutex and a counter
mutex = Mutex.new
count = 0

# Defines a proc that prints the value of the counter
# and increments it by 1
work = proc do
  10.times do
    mutex.synchronize do
      sleep 0.1
      puts "count: #{count}"
      count += 1
    end
  end
end

# Spawns 50 threads that all call the proc we just defined
# and then waits for all the threads to finish
50.times.map do
  Thread.new { work.call }
end.to_a.each(&:join)

# Prints the final count after all the threads are done
puts "Final count: #{count}"

If you now run this code from IRB and look at the output you'd see that the count always goes up by one and that the final count is 500

...
count: 487
count: 488
count: 489
count: 490
count: 491
count: 492
count: 493
count: 494
count: 495
count: 496
count: 497
count: 498
count: 499
Final count: 500

But if you remove the mutex, first you'd notice that the code runs much faster, second thing you'd notice is that the output is completely random, sometimes repeats, and change each time you run this

...
count: 487
count: 488
count: 489
count: 490
count: 491
count: 492
count: 493
count: 493
count: 495
count: 496
count: 495
count: 496
count: 494
Final count: 497

This is because, without the mutex, all the threads try to read and update the counter at the same time. Sometimes two threads read the value at the same time - like 493 in the output above - which results in a value being skipped - because both read 493 and updated it to 494.

With the mutex, one thread locks the mutex using the synchronize method, updates the counter ten times, and unlocks the mutex, then another thread lock it again, etc.

That's why with the mutex the code takes longer to complete, each thread runs after the other so the total time is 50 times 10 times 100ms which is 50 seconds. While without the mutex all threads run at the same time so the total time it takes to execute the code is 10 x 100ms or 1 second.

To guarantee that just 1 subscription will measure the latency I can use Mutex#try_lock and Mutex#unlock instead of Mutex#synchronize. Unlike synchronize which blocks and waits until the mutex unlock, try_lock doesn't block and instead returns true if it acquired the lock or false if it didn't. With try_lock I can tell all threads to try an measure the latency, the thread that locks the mutex then performs the measurement and all others return.

MUTEX = Mutex.new

def run_unless_already_running(&block)
  lock_acquired = MUTEX.try_lock
  return unless lock_acquired

  block.call
ensure
  MUTEX.unlock if lock_acquired
end

def collect(payload)
  run_unless_already_running do
    latency = Time.now.to_f - sent_at
  
    Rails.logger.debug "PubSub Latency: #{latency}"
  end
end

stream_from("metrics", proc do |json|
  payload = ActiveSupport::JSON.decode(json)
  collect(payload)
end)

To make things prettier I put all code related to metrics collection into a module

module ActionCable
  module MetricsCollector
    MUTEX = Mutex.new
    STREAM_NAME = "internal.action_cable.metrics"
    COLLECTION_TRESHOLD = 60.seconds

    cattr_accessor :last_collected_at

    class << self
      def measure
        ActionCable.server.broadcast(stream_name, measurement_payload)
      end

      def stream_name = STREAM_NAME

      def measurement_payload
        {
          sent_at: Time.now.to_f
        }
      end

      def collect(payload)
        run_unless_already_running do
          next if measurement_already_collected?

          self.last_collected_at = Time.now
          collect_measurement(payload)
        end
      end

      def measurement_already_collected? = last_collected_at&.after?(collection_trashold.ago)

      def collection_trashold = COLLECTION_TRESHOLD

      private

      def run_unless_already_running(&block)
        lock_acquired = MUTEX.try_lock
        return unless lock_acquired

        block.call
      ensure
        MUTEX.unlock if lock_acquired
      end

      def collect_measurement(payload)
        pub_sub_latency = [0, Time.now.to_f - payload.fetch("sent_at", -Float::INFINITY)].max
        Rails.logger.debug "PubSub latency: #{pub_sub_latency}"
      end
    end
  end
end

Then I changed my cron job to run

bin/rails runner "ActionCable::MetricsCollector.measure"

And added the following to app/channels/application_cable/channel.rb

module ApplicationCable
  class Channel < ActionCable::Channel::Base
    after_subscribe do
      stream_from(
        ::ActionCable::MetricsCollector.stream_name,
        proc do |json|
          payload = ActiveSupport::JSON.decode(json)
          ::ActionCable::MetricsCollector.collect(payload)
        rescue => e
          Rails.logger.error("Failed to collect metrics: #{e}")
        end
      )
    end
  end
end

How do I expose this metric to Elastic APM?

After some thought I decided I wouldn't. I could hack something together, but I'm shoehorning this metric in and later metrics I flat out don't know how to expose in Elastic AMP. So I decided not to bother with this and figure it out later.

Number of connections

Since I'm running an IoT platform I have a somewhat constant number of clients connected at all times. This means that by knowing the number of clients at any given moment I can easily see if there is a problem or if I need to scale out.

Luckily, with the module that I've added to measure the PubSub latency it's easy to expose the number of client. All I have to do is change the collect_measurement method like so

def collect_measurement(payload)
  pub_sub_latency = [0, Time.now.to_f - payload.fetch("sent_at", -Float::INFINITY)].max
  Rails.logger.debug "PubSub latency: #{pub_sub_latency}"

  connection_count = ActionCable.server.connections.length
  Rails.logger.debug "Connection count: #{connection_count}"
end

Without that module I'd probably setup a periodic task like so

module ApplicationCable
  class Channel < ActionCable::Channel::Base
    periodically every: 1.minute do
	  connection_count = ActionCable.server.connections.length
      Rails.logger.debug "Connection count: #{connection_count}"
    end
  end
end

The trick to getting the count is to call ActionCable.server.connections.length from within the server process. If you call it from the Rails console or from a rails runner the count will always return zero as the connections only exist in the server's memory.

Client-Server Latency

Since I'm running my own patch of ActionCable that adds PONG messages in addition to the standard heartbeat PING message, I can easily expose the time it takes for a message to travel from the server over the Internet to a client.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Diagram that shows what Client-Server latency measures

This metric isn't useful for regular web apps as it's non actionable - you can't fix your client's Internet. But it's useful for apps, like mine, where you have control over clients and want to have them connected 24/7.

# Stock Rails 8 doesn't have this notificaiton
ActiveSupport::Notifications.subscribe "client_latency.action_cable" do |_name, _start, _finish, _id, payload|
  Rails.logger.debug("Client latency: #{payload[:value]}")
end

Clients will start disconnecting if this latency exceeds six seconds. If the median latency exceeds a few seconds this indicates a widespread Internet issue (which is more common in the US than I though it would be).

Now that I have all these metrics I have to expose them somehow.

I decided to talk to a coworker from the DevOps team about how they collect and expose metrics from our machines and he told me that they use Prometheus and that he could configure it to collect metrics from ActionCable.

But what is Prometheus?

Prometheus is an open-source monitoring and alerting system - it's a service that collects metrics, allows you to query them, and to configure alerts when they exceed a threshold. It's a bit different from the SaaS monitoring solutions because it scrapes metrics from your application instead of having your application push metrics to it.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Diagram that explains the difference between Prometheus and other APM services

Usually, you'd expose a /metrics endpoint that renders all the metrics the application has collected in a format that Prometheus can parse.

I wasn't sure how to do that at first, but I remember reading an article on Basecamp's dev blog and it mentioned a gem called Yabeda.

Yabeda is a framework for monitoring Ruby applications. Using it you can measure, or report measurements, to various monitoring services like Datadog, NewRelic, Honeybadger, AWS CloudWatch, Prometheus, and many others.

In a nutshell, I can report my measurements to Yabeda and then it handles exposing those metrics to various different monitoring services - like Prometheus - for me.

According to the documentation, before I can use Yabeda I have to configure it which means that I have to define all the metrics that I'm going to expose. Yabeda supports a handful of types of metrics - counters, gauges, and histograms - so let me explain those first.

A counter is a number that can only ever go up by some amount. E.g. the number of ActiveJob jobs processed is a counter because the metric is a number that can only ever increase. But the number of connections to ActionCable isn't a counter since the number goes up when more clients connect and goes down when clients disconnect.

To represent numbers that can go both up and down as a metric we need a gauge. Gauges represent a metric that can be any number at any time.

Finally, a histogram represents a distribution of values. It tells you how many measurements you've got in a certain range called a bucket. Histograms are usually represented by a set of counters and gauges - one counter for each bucket, one counter for the number of measurements, and one gauge for the sum of all measurements. This sounds a bit complicated, but it allows you to calculate the average, median, and other distribution metrics quickly and without having to store every single measurement somewhere. Usually you'd use a histogram to represent time measurements like response times.

Defining these metrics is fairly straight forward

# config/initializers/yabeda.rb

Yabeda.configure do
  group :action_cable do
    histogram :client_server_latency do
      comment "The time it takes for a message to travel via the Internet to a client"
      unit :seconds
      buckets [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    end

    histogram :pubsub_latency do
      comment "The time it takes for a message to travel through the PubSub service"
      unit :seconds
      buckets [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    end

    histogram :action_execution_duration do
      comment "The time it takes to perform an invoked action"
      unit :seconds
      buckets [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    end

    gauge :connection_count, comment: "Number of open WebSocket connections"
  end
end

To report a metric I have to build a chain of method calls consisting of the group names of the metric I want to report, then the metric's name, and finally call measure on that. The measure method accepts either one argument and a block or two arguments. The first argument is a Hash of tags to add to the measurement, I'll explain those later. The second is the measurement itself. Without the second argument, Yabeda will execute the block, measure it's execution time, and report that.

Yabeda.action_cable.client_server_latency.measure({}, 0.123)
# or
Yabeda.action_cable.client_server_latency.measure({}) do
  sleep 0.123 # do some work
end

I can now update all ActiveSupport Notification subscriptions to report to Yabeda, and I can move them all to Yabeda's initializer.

# config/initializers/yabeda.rb

Yabeda.configure do
  group :action_cable do
    histogram :client_server_latency do
      comment "The time it takes for a message to travel via the Internet to a client"
      unit :seconds
      buckets [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    end

    histogram :pub_sub_latency do
      comment "The time it takes for a message to travel through the PubSub service"
      unit :seconds
      buckets [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    end

    histogram :action_execution_duration do
      comment "The time it takes to perform an invoked action"
      unit :seconds
      buckets [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    end

    gauge :connection_count, comment: "Number of open WebSocket connections"
  end
end

ActiveSupport::Notifications.subscribe "perform_action.action_cable" do |event|
  name = "#{event.payload[:channel_class]}##{event.payload[:action]}"
  duration = event.duration / 1000.0 # convert ms to seconds
  
  Yabeda
    .action_cable
    .action_execution_duration
    .measure({ action: action }, duration) # I've used a tag here
end

ActiveSupport::Notifications.subscribe "client_latency.action_cable" do |_name, _start, _finish, _id, payload|
  client_server_latency = payload[:value]
  Yabeda
    .action_cable
    .client_server_latency
    .measure({}, client_server_latency)
end

You might have noticed that I've used a tag when reporting the action execution time. You can think of tags as groups of measurements. Instead of collecting the action execution time of all actions in ActionCable, using tags I'll collect the action execution time of each individual action. That way I can easily pin point which action has the highest duration, but I can still get the average execution time by finding the average of all the tagged measurements.

Without tags I'd have to define dozens of histograms - one for each action. But with tags I can define one histogram and let Yabeda group my measurements for me.

I have to update the metrics collector too

module ActionCable
  module MetricsCollector
    MUTEX = Mutex.new
    STREAM_NAME = "internal.action_cable.metrics"
    COLLECTION_TRESHOLD = 60.seconds

    cattr_accessor :last_collected_at

    class << self
      def measure
        ActionCable.server.broadcast(stream_name, measurement_payload)
      end

      def stream_name = STREAM_NAME

      def measurement_payload
        {
          sent_at: Time.now.to_f
        }
      end

      def collect(payload)
        run_unless_already_running do
          next if measurement_already_collected?

          self.last_collected_at = Time.now
          collect_measurement(payload)
        end
      end

      def measurement_already_collected? = last_collected_at&.after?(collection_trashold.ago)

      def collection_trashold = COLLECTION_TRESHOLD

      private

      def run_unless_already_running(&block)
        lock_acquired = MUTEX.try_lock
        return unless lock_acquired

        block.call
      ensure
        MUTEX.unlock if lock_acquired
      end

      def collect_measurement(payload)
        pub_sub_latency = [0, Time.now.to_f - payload.fetch("sent_at", -Float::INFINITY)].max
        Yabeda.action_cable.pub_sub_latency.measure({}, pub_sub_latency)

		connection_count = ActionCable.server.connections.length
		Yabeda.action_cable.connection_count.set({}, connection_count)
      end
    end
  end
end

Now that my metrics are in Yabeda, I have to somehow expose them to Prometheus. For that I've added the yabeda-prometheus-mmap gem and mounted its route

# config/routes.rb

Rails.application.routes.draw do
  mount Yabeda::Prometheus::Exporter, at: "/metrics"
  # ...
end

There is also yabeda-prometheus, if you are running a single worker server these two are effectively the same. But if you are running Puma with multiple workers, or SolidQueue with multiple workers then the mmap version of the gem will be much faster.

Additionally, when running multiple workers, you probably also want to add "aggregation: :sum" to all gauges. Without that option, if a worker fails and gets restarted, you'll get very wonky metrics because the gauges will get counted twice with the old worker's gauge being stuck at whatever value it was before the worker crashed.

Now, with the route mounted I can connect to my ActionCable server and then open /metrics in a separate browser window to see the output

...
# HELP action_cable_action_execution_duration_seconds Multiprocess metric
# TYPE action_cable_action_execution_duration_seconds histogram
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="+Inf"} 2189
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="0.005"} 101
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="0.01"} 559
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="0.025"} 2044
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="0.05"} 2174
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="0.1"} 2181
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="0.25"} 2189
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="0.5"} 2189
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="1"} 2189
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="10"} 2189
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="2.5"} 2189
action_cable_action_execution_duration_seconds_bucket{environment="development",action="ChatChannel#post_message",le="5"} 2189
action_cable_action_execution_duration_seconds_count{environment="development",action="ChatChannel#post_message"} 2189
action_cable_action_execution_duration_seconds_sum{environment="development",action="ChatChannel#post_message"} 34.04095268249512
...

That's my action execution duration. It works!

My coworker had already configured Prometheus to scrape that endpoint every minute or so and the metrics appeared in Prometheus within a few minutes.

Analyzing the metrics

Now that I had all the metrics somewhere where I could query them I had to make them easy to analyze. I've decided to go with Grafana as we already had an instance to monitor our infrastructure.

Grafana allowed me to take this set of metrics and turn them into graphs that are easy to parse. First, I exposed the number of connections. The graph now clearly shows how many clients were connected at each minute.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The number of connections to ActionCable plotted over time

Next, I plotted the action execution latency. This is a bit more involved just because I wanted to plot the average, median, 95th percentile and 99th percentile.

The average is important for scaling as it's used in Little's law to determine the number of workers necessary to handle the load. The median is a better representation of what most clients experience as it represents the latency of 50% of all clients. The 95th and 99th percentile represent the worst cases - the latency experienced by 95% and 99% of all clients.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The action execution latency of all actions plotted over time

In addition to plotting the overall latency across all actions I've also plotted the latency per action as a heat-map. Here each rectangle represents 1 minute, red rectangles represent high latency and blue represents low latency. If I see a bright red action here I know that it's causing a problem.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A heatmap that breaks down the action execution latency by action over time

I've plotted the client-server latency and PubSub latency the same way as the overall action execution duration.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Client-Server latency plotted over time

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

PubSub latency plotted over time

With all these metrics in Grafana I could now focus on finding and fixing the root cause. As the incident had passed and I didn't have any metrics from it so I'd have to wait for another one.

Step six, fixing the root cause.

The app was now stable so I tried a few things that I guessed could be the cause while I waited to see if this would occur again.

Then, after two weeks, the incident happened again!

I quickly opened Grafana and saw that the PubSub latency was extremely high.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

PubSub latency during the incident

(the 95th and 99th percentile are capped at 10s because that's the largest bucket I've configured)

This means that Redis is the culprit. For some reason It takes ages to deliver messages. As a quick fix, I opened the AWS console and scaled up our Redis instance. Within a few minutes the incident cleared up.

I spent the next week investigating ElastiCache Redis and confirmed that it was the root cause of both incidents. I'll go into more detail on this in a subsequent post, but the gist of it is that AWS is a piece of crap.

Looking back

I would never have found and fixed this without the monitoring that I've set up.

In the following month I've exposed more metrics like the broadcast and transmission latency, and the number of confirmed and rejected subscriptions.

ActiveSupport::Notifications.subscribe "broadcast.action_cable" do |_name, start, finish, _id, _payload|
  Yabeda.action_cable.broadcast_duration.measure({}, finish.to_f - start.to_f)
end

ActiveSupport::Notifications.subscribe "transmit.action_cable" do |_name, start, finish, _id, _payload|
  Yabeda.action_cable.transmission_duration.measure({}, finish.to_f - start.to_f)
end

ActiveSupport::Notifications.subscribe "transmit_subscription_confirmation.action_cable" do |_name, start, finish, _id, _payload|
  Yabeda.action_cable.transmission_duration.measure({}, finish.to_f - start.to_f)
  Yabeda.action_cable.confirmed_subscription_count.increment({}, by: 1)
end

ActiveSupport::Notifications.subscribe "transmit_subscription_rejection.action_cable" do |_name, start, finish, _id, _payload|
  Yabeda.action_cable.transmission_duration.measure({}, finish.to_f - start.to_f)
  Yabeda.action_cable.rejected_subscription_count.increment({}, by: 1)
end

I don't really need those metrics now, but through this incident I've learned that it's better to have a metric before you need it so I exposed everything I could expose with ActiveSupport Notifications.

Additionally, I've added a few more gems - like yabeda-puma-plugin, yabeda-rails and yabeda-activejob - that expose various metrics to help me pinpoint problems in the future.

Software is made by people

2024-06-03T05:05:00Z

The recent Hotwire pop-up drama has shown me that some have forgotten that software is made by people - people with different backgrounds, different values, different skills, and different flaws.

The diversity in opinions and aproaches is what makes modern day software development great. That’s what drives innovation.

There are many tribes of people that like to build software in a certain way, that like to use a certain language, that approach problems a certain way, that prefere certan conventions over others. It’s easy to find a tribe and join it, or to start your own.

When I started programming professionally I’ve aleardy found my tribes and thought that people in other tribes - especially those with diametrically opposite beliefs - were just doing software wrong.

But I was wrong back then.

Today I understand that all tribes in our community formed not because we interpreted something differently, but because we have different values about the same thing - software.

We are like visitors at an art gallery looking at an abstract sculpture - each of us sees something different even though we are looking at the same thing.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Optical illusion that sometimes looks like a rabbit and sometimes like a duck (well, bird)

Where one person sees a rabbit another will see a duck. There is no one right answer. If it’s a rabbit, or a duck, or both, or neither is dependent on you.

HTML-over-the-wire and SPAs, in this context, are two ways of doing the same thing - rendering content in a reactive way. There is no one right answer as to what the best way to do it is. Which one you prefere is up to you.

Why does Rails put the type column first in an index for a polymorphic association?

2024-04-01T12:00:00Z

Last week, I had a discussion with a coworker about how Rails indexes columns used in polymorphic associations. He thought that the order of columns in the index should be flipped - instead of indexing by type and ID, it should index by ID and type - as that way the most restrictive column is first, and therefore the index is more efficient. While I argued that the way that Rails indexes polymorphic associations is very pragmatic while also being efficient.

First off, what are we talking about?

We have an Access Log model in our app. It holds a log of who granted access to which device and when. But the thing granting access can be a Person, another Device or a Schedule.

In Rails, an association that can point to many different types of models is called a "polymorphic associations".

class AccessLog < ApplicationRecord
  belongs_to :device
  belongs_to :grantor, polymorphic: true
end

# Since grantor is polymorphic we can assign any type of model to it
log = AccessLog.create!(device: Device.all.sample, grantor: Person.all.sample)
log.grantor.class # => Person

log.update!(grantor: Device.all.sample)
log.grantor.class # => Device

log.update!(grantor: Schedule.all.sample)
log.grantor.class # => Schedule

# Since the device association isn't polimorphic we can't assign anything
# but a device to it
log.update!(device: Person.all.sample) # => ActiveRecord::AssociationTypeMismatch: Device expected, got Person

A polymorphic and a regular association are implemented differently in the database.

A regular association is just a column that holds an ID that points to a record in the other model's table; usually it also has a foreign key constraint, which ensures that the assigned ID actually exists in the other table.

While a polymorphic association uses two columns, one to hold the type of model it's for and another to hold the ID of the record it's for.

create_table "access_logs", force: :cascade do |t|
  # Regular association - holds the ID of the record from the devices table
  t.bigint "device_id", null: false
  
  # Polymorphic association
  t.string "grantor_type", null: false # holds the model name
  t.bigint "grantor_id", null: false # holds the ID of the record
  
  t.index ["device_id"], name: "index_access_logs_on_device_id"
  t.index ["grantor_type", "grantor_id"], name: "index_access_logs_on_entry"
end

# ensures that every ID in device_id coresponds to an id in the devices table
add_foreign_key "access_logs", "devices"

When you generate an association, Rails will automatically create an index for it. In the case of a polymorphic association, it will create a composite index - that's an index that indexes two or more columns simultaneously - in which it will put the type column first and the ID column last.

Why does it matter which column is first in the index?

This has to do with how databases build indices. By default, MySQL, MariaDB and PostgreSQL build b-tree indices.

An index is similar to a regular book index, but instead of having chapter names and page numbers, it has values from the indexed column and internal record IDs.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Comparison between a book and a naive database index

The problem with this kind of index is that it will grow as the number of distinct values in the indexed column grows.

E.g. if you create an index on a column that holds a person's email address, this kind of index would have one entry for each email, and the database would have to scan through each email until it finds the right one. To speed things up, we can do what dictionaries usually do and create a multi-level index.

In a dictionary, you'd usually have an index that points you to another index. E.g. the main index would tell you that the index for all words starting with R is on page 6. Then the R sub-index would tell you that words starting with RU are on pages 342 to 361. Then you'd go to page 342 and search for the word "Ruby" until you reached page 361.

= 768px) 520px, 50vw"> = 768px) 520px, 50vw">

A multi-level index from a dictionary

= 768px) 520px, 50vw"> = 768px) 520px, 50vw">

A naive multi-level index in a database

B-trees are similar to multi-level indices in dictionaries, though they have extra rules and nuances that aren't important now.

What's important is that a b-tree index can have a sub-index, which in term can have a sub-index, and so on. And the more items each sub-index has, the slower it is to find something.

These indices work for single values such as words and emails, but how would you make an index of two words, or an email and a number?

To create a composite index, most databases just concatenate the values together and build a regular b-tree index. This is a gross simplification of what actually happens, but in essence, that's what they do. Because of that, the order of the columns becomes important.

If you tell the database to build an index over grantor_type first and then grantor_id it would take the two and concatenate them together like so "Person|443". But if you tell it to build an index over grantor_id first and then grantor_type it would concatenate them like so "443|Person".

The order of the columns in the index changes the structure of the index.

When the ID column is first, the index points to fewer rows, so the database has to scan through fewer rows, and therefore it can find a match quicker. That's why it's a general recommendation to put the most specific column first in a composite index - it narrows down the search a lot.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Naive composite database index with the ID column first and type column second

When the type column is first, there are a lot more rows to scan through in the end.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Naive composite database index with the type column first and ID column second

Why would anyone create an index and put the type column first then?

You don't have to follow a multi-level index all the way to the last index entry to find something. E.g. if you are searching for all words that start with R, you can go from the main index to find all sub-indices of words starting with R. Take the start page of the first index and the last page of the last index. Now you know that all words between these two pages start with R, and you didn't have to scan through anything.

In a real app, you'd sometimes want to filter all records that are associated with another record of a certain type. E.g. you might want to fetch all Access Logs that were granted by a Schedule, or all People associated with a Property through an Access Log.

class Property < ApplicationRecord
  has_many :access_logs
  
  has_many :schedule_activations,
    -> { where(grantor_type: "Schedule") },
    class_name: "AccessLog"
    
  has_many :active_people,
    through: :access_logs,
    source: :grantor,
    source_type: "Person"
end

# Calling
Propert.find(42).schedule_activations.count
# Executes the following SQL statement
# ```sql
# SELECT COUNT(*)
# FROM access_logs
# WHERE property_id = 42 
#   AND grantor_type = 'Schedule'
# ```

# Calling
Propert.find(42).active_people.count
# Executes the following SQL statement
# ```sql
# SELECT COUNT(tenants.*)
# FROM tenants INNER JOIN access_logs
#   ON tenants.id = access_logs.grantor_id
# WHERE access_logs.property_id = 42
#   AND access_logs.grantor_type = 'Person'

In that case, the composite index over grantor_id first and then grantor_type is basically useless. The database would have to traverse the whole index to aggregate a range that it then has to scan through.

While the index over grantor_type first and then grantor_id can quickly give you the range that you have to scan through by looking at the first and last entry of a sub-index.

We can test this out.

I created a test MariaDB database, with an access_logs table that I've filled with a million rows. Then I created a composite index over grantor_type and grantor_id that I've named test_index.

When I ask MariaDB to explain how it would go about counting all rows where the grantor_type is Person, it says that it would use test_index to generate the count.

MariaDB [test]> CREATE INDEX test_index ON access_logs (grantor_type, grantor_id);
Query OK, 0 rows affected (1.209 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [test]> EXPLAIN  SELECT COUNT(1) FROM access_logs WHERE grantor_type = 'Person';                                                 
+------+-------------+---------------+------+---------------+------------+---------+-------+--------+--------------------------+
| id   | select_type | table         | type | possible_keys | key        | key_len | ref   | rows   | Extra                    |
+------+-------------+---------------+------+---------------+------------+---------+-------+--------+--------------------------+
|    1 | SIMPLE      | access_logs   | ref  | test_index    | test_index | 183     | const | 498290 | Using where; Using index |
+------+-------------+---------------+------+---------------+------------+---------+-------+--------+--------------------------+
1 row in set (0.002 sec)

MariaDB [test]> SELECT COUNT(1) FROM access_logs WHERE grantor_type = 'Person';
+----------+
| COUNT(1) |
+----------+
|   333400 |
+----------+
1 row in set (0.042 sec)

If I ask the same question, but change the order of columns in the index, then MariaDB says that it can't use any index to generate the count but it still tries to use test_index, and it takes about twice as long to count the rows.

MariaDB [test]> DROP INDEX test_index ON access_logs;
Query OK, 0 rows affected (0.045 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [test]> CREATE INDEX test_index ON access_logs (grantor_id, grantor_type);
Query OK, 0 rows affected (1.146 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [test]> EXPLAIN  SELECT COUNT(1) FROM access_logs WHERE grantor_type = 'Person';
+------+-------------+---------------+-------+---------------+------------+---------+------+--------+--------------------------+
| id   | select_type | table         | type  | possible_keys | key        | key_len | ref  | rows   | Extra                    |
+------+-------------+---------------+-------+---------------+------------+---------+------+--------+--------------------------+
|    1 | SIMPLE      | access_logs   | index | NULL          | test_index | 192     | NULL | 996580 | Using where; Using index |
+------+-------------+---------------+-------+---------------+------------+---------+------+--------+--------------------------+
1 row in set (0.000 sec)

MariaDB [test]> SELECT COUNT(1) FROM access_logs WHERE grantor_type = 'Person';
+----------+
| COUNT(1) |
+----------+
|   333400 |
+----------+
1 row in set (0.095 sec)

These results will vary from database to database, but the principle still stands.

For example, PostgreSQL can sometimes utilize an index even when it isn't optimal. In the case of an index for a polymorphic association, changing the order of the columns has much less impact than in MariaDB. In both cases, PostgreSQL uses the index and produces a result in about the same time.

postgres=# CREATE INDEX test_index ON access_logs (grantor_type, grantor_id);
CREATE INDEX

postgres=# EXPLAIN ANALYZE SELECT COUNT(1) FROM access_logs WHERE grantor_type = 'Person';                                                                      
QUERY PLAN                                                                       -----------------------------------------------------------------------------------------------------------------------------------------------------------------------                                                                            Finalize Aggregate  (cost=6457.79..6457.80 rows=1 width=8) (actual time=12.408..13.780 rows=1 loops=1)                                                ->  Gather  (cost=6457.57..6457.78 rows=2 width=8) (actual time=12.403..13.777 rows=3 loops=1)
    Workers Planned: 2
	Workers Launched: 2
	->  Partial Aggregate  (cost=5457.57..5457.58 rows=1 width=8) (actual time=10.314..10.314 rows=1 loops=3)
	  ->  Parallel Index Only Scan using test_index on access_logs  (cost=0.42..5107.64 rows=139972 width=0) (actual time=0.027..6.566 rows=111133 loops=3)
	    Index Cond: (grantor_type = 'Person'::text)                                      Heap Fetches: 0
Planning Time: 0.145 ms
Execution Time: 13.801 ms
(10 rows)

postgres=# DROP INDEX test_index;
DROP INDEX

postgres=# CREATE INDEX test_index ON door_releases (grantor_id, grantor_type);
CREATE INDEX

postgres=# EXPLAIN ANALYZE SELECT COUNT(1) FROM door_releases WHERE grantor_type = 'Person';
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Finalize Aggregate  (cost=13778.29..13778.30 rows=1 width=8) (actual time=14.570..15.922 rows=1 loops=1)
   ->  Gather  (cost=13778.08..13778.29 rows=2 width=8) (actual time=14.566..15.919 rows=3 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         ->  Partial Aggregate  (cost=12778.08..12778.09 rows=1 width=8) (actual time=12.210..12.210 rows=1 loops=3)
               ->  Parallel Index Only Scan using test_index on door_releases  (cost=0.42..12428.15 rows=139972 width=0) (actual time=0.037..8.808 rows=111133 loops=3)
                     Index Cond: (grantor_type = 'Person'::text)
                     Heap Fetches: 0
 Planning Time: 0.079 ms
 Execution Time: 15.944 ms
(10 rows)

Simply put.

The type column is first in an index for a polymorphic association because then the same index can be used to speed up association look-ups (querying by both type and ID columns) and to filter by the type of the associated record (querying by just the type column).

These are the two most common use cases that you'll encounter in a Rails app. And also the use-cases for which Rails has built-in interfaces for (e.g. the source_type option).

This is more efficient in terms of speed (index creation and updating) and space (the size of the index on disk) than having two or more separate indices, which would be required in some databases if the columns were flipped.

Running Campfire behind traefik

2024-02-06T07:10:00Z

I rent a bare metal server from Scaleway where I host all my apps. For me, that's much more economical than running multiple small VPS boxes. 50€ a month buys me 2TB of disk space, 64GB of RAM and an 4 core / 8 thread CPU on which I can host two or three dozen apps. Compare that to AWS or DigitalOcean where for the same money I get about one quarter of those resources.

All my apps are deployed as Docker containers, most of them using Kamal, in front of which I put traefik as a reverse proxy to direct traffic and provide HTTPS via Let's Encrypt.

Last weekend I wanted to setup a Campfire instance for Ruby Zagreb - my local Ruby user-group - on that machine.

So I ran the official installer and got an error.

#!/usr/bin/bash
sudo once setup 1111-1111-1111-1111 

 ██████╗ ███╗   ██╗ ██████╗███████╗
██╔═══██╗████╗  ██║██╔════╝██╔════╝
██║   ██║██╔██╗ ██║██║     █████╗  
██║   ██║██║╚██╗██║██║     ██╔══╝  
╚██████╔╝██║ ╚████║╚██████╗███████╗
 ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝
                            
Failed to start the software
                            
                                                                                                                                                                                                                                              
The software couldn't be started. Please try again.

I expected the installer to fail since it can't know my setup, and it didn't ask me anything besides the domain of the instance.

Sadly, the email you get with your purchase doesn't explain what the installer does, and I can't check what it's doing since the installer is a binary blob. So I had to snoop around for a bit.

The first thing I found was the installer's error log which said that the container couldn't bind to port 443.

cat .once-error.log
2024/01/27 19:33:11 Error response from daemon: driver failed programming external connectivity on endpoint campfire (e32f1a334f8ac25b9f3df0c283adb3eb3e8a554074f0827d17f8b566d03940a5): Bind for 0.0.0.0:443 failed: port is already allocated

Since traefik is bound to port 80 and 443 this makes perfect sense.

The error looks like it came from Docker so I checked my images, and sure enough there was a new "registry.once.com/campfire:latest" image and a stopped container which I promptly inspected.

Within the container runs a custom proxy called thruster that terminates SSL and handles the sendfile protocol. That's what's bound to port 443 and 80 of the container.

Inside the container is also a Redis instance, which implies that the installer sets at least "sysctl vm.overcommit_memory=1" else Redis wouldn't start.

I also saw that there were ENV variables injected into the container so I looked where those came from and found a config file in "/root/.config/once/config.json".

sudo cat /root/.config/once/config.json | jq
{
  "token": "1111-1111-1111-1111",
  "product": "campfire",
  "product_name": "Campfire",
  "email_address": "phony.email@example.com",
  "ssl_domain": "chat.rubyzg.org",
  "validation_token": "PHONY_VALIDATION_KEY",
  "secret_key_base": "PHONY_SECRET_KEY_BASE",
  "vapid_private_key": "PHONY_PRIVATE_KEY",
  "vapid_public_key": "PHONY_PUBLIC_KEY",
  "storage_location": "/var/once/campfire",
  "cron_hour": 2,
  "once_binary_etag": ""
}

Seems pretty straight forward. This is roughly what the installer does

sudo sysctl vm.overcommit_memory=1

docker run \
	-d \
	--name campfire \
	--restart unless-stopped \
	--env-file campfire/.env \
	-p 443
	-p 80
	-v campfire/storage:/rails/storage \
	registry.once.com/campfire:latest \
	bin/boot;;

Where "campfire/.env" contains the values from "/root/.config/once/config.json" plus a few extras.

# campfire/.env 
SECRET_KEY_BASE=PHONY_SECRET_KEY_BASE
VAPID_PRIVATE_KEY=PHONY_PRIVATE_KEY
VAPID_PUBLIC_KEY=PHONY_PUBLIC_KEY
SSL_DOMAIN=chat.rubyzg.org
DISABLE_SSL=YES

To run Campfire without the installer, behind traefik, all I had to do is

sudo sysctl vm.overcommit_memory=1

docker run \
	-d \
	--name campfire \
	--network apps \
	--restart unless-stopped \
	--env-file campfire/.env \
	--label-file campfire/labels \
	-v campfire/Procfile:/rails/Procfile \
	-v campfire/production.rb:/rails/config/environments/production.rb \
	-v campfire/storage:/rails/storage \
	registry.once.com/campfire:latest \
	bin/boot;;

Put this into "campfire/labels"

# Enable traefik for the container 
traefik.enable=true 

# Convert HTTP traffic to HTTPS 
traefik.http.routers.campfire.entrypoints=websecure
traefik.http.routers.campfire.tls.certresolver=letsencrypt

# Register the container to a domain - CHANGE THIS TO YOUR DOMAIN 
traefik.http.routers.campfire.rule=Host(`chat.rubyzg.org`) 

# Return a service not available screen if Campfire isn't running 
traefik.http.middlewares.campfire.retry.attempts=10 
traefik.http.middlewares.campfire.retry.initialinterval=1s 
traefik.http.services.campfire.loadbalancer.healthcheck.path=/up 
traefik.http.services.campfire.loadbalancer.server.port=80 
traefik.http.services.campfire.loadbalancer.server.scheme=http

Finally, I had to disable thruster and bind Puma to port 80 by updating the "web" entry in the Procfile (you get the source files in the purchase email)

# campfire/Procfile 
web: bin/rails server -b 0.0.0.0 -p 80 
# ...

This will boot the server, it will allow you to setup an admin account, create rooms, add a logo, but you won't be able to post messages because Action Cable can't connect to the server.

Traefik has a bug where it doesn't forward x-forwarded-for headers for WebSocket handshakes.

To fix that I had to update "production.rb" with the exact URL that Action Cable would get requests from

# production.rb
# ...
config.action_cable.url = "wss://#{ENV.fetch("SSL_DOMAIN")}/cable"
config.action_cable.allowed_request_origins = ["https://#{ENV.fetch("SSL_DOMAIN")}"]
# ...

That's it.

It's isn't as straight forward as I'd like it to be, and it would be nice if Kamal and Campfire could play along nicely out of the box. I'd bet there is a large overlap between those communities, and both are 37signals projects.

Deconstructing Action Cable

2024-02-05T13:00:00Z

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Deconstructing Action Cable

I knew what Action Cable was about and roughly how it worked since it was released. But lately I've developed a much deeper understanding of it because I had to extend it and explain its internals to others.

The following is how I explained Action Cable to myself. I took a top-down approach, starting at what I could observe Action Cable doing in my browser and following the code until I found the implementation, which I then deconstructed to understand why it does what it does.

This was a along journey down a deep rabbit hole, but it was worth it. At the end I now understand how and why Action Cable does what it does, and appreciate how simple and elegant it actually is.

From HTTP to WebSocket

Action Cable is a framework that allows your server to push messages to the browser, or any other client.

Usually your server can only respond to requests that a client, like your browser, makes to it. But Action Cable opens a WebSocket between the client and the server through which the server can push messages to the client at any time.

The client can also push messages to the server through that WebSocket. But that's in my opinion less interesting as we could do that without Action Cable.

How does the client connect to Action Cable's WebSocket?

When anyone or anything wants to connect to your Rails app via Action Cable they first have to make a HTTP request to "/cable" and after a bit of back and forth out pops a WebSocket.

But how does a HTTP request become a Websocket?

Back in 1997 HTTP version 1.1 was released. It's very similar to HTTP 1.0 with a few additions, the one that's important for WebSockets is the addition of status 101 aka. Switching protocols and the "Upgrade" header. The intent of status 101 was to enable browsers to gracefully transition from HTTP 1.x to an eventual HTTP 2 or some other protocol that would be incompatible with HTTP 1.

And that's exactly what a browser does when it connects to a WebSocket. It first makes a request with the "Upgrade" header set to "websocket" which tells the server "hey, I want to switch to using WebSockets for this request". If the server supports WebSockets it responds with status 101 indicating that the upgrade was accepted and that, from now on, it will communicate using the WebSocket protocol on that request's connection.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A WebSocket upgrade request

But why do we need the HTTP request in the first place? Can't we just connect directly via the WebSocket protocol since we already know that the server supports it?

The genius of WebSockets lies in the initial HTTP request. Because of that initial HTTP request we have access to all the browser's cookies that are sent over, so we can use them to figure out who wants to open a WebSocket, if they are allowed to do that, and we can remember who the socket is for.

So Action Cable converts HTTP requests to WebSockets?

Yes! It's a WebSocket server, but it also does so much more.

Connections

So someone's connected to our WebSocket, but how do we know who we are talking to and how can we send messages to them?

In Action Cable, each open WebSocket has a corresponding Connection object. The connection object is responsible for keeping track of who the WebSocket is for and figuring out if the client is allowed to open a WebSocket.

When a client makes the initial HTTP request to "/cable", Action Cable will take the request's headers, cookies, URL and create a Connection object from it.

If it exists, the Connection object will be an instance of "ApplicationCable::Connection". If that class doesn't exist it will be an instance of "ActionCable::Connection::Base". You can change the class of the connection by setting "config.action_cable.connection_class = ->{ WhateverClassIWant }".

When a connection is established the "#connect" method on our "ApplicationCable::Connection" object will get called. In that method you'd usually want to figure out who opened the connection, and reject it if they aren't allowed to open it.

module ApplicationCable
  class Connection < ActionCable::Connection::Base
    attr_accessor :current_person
    
    def connect
	  Rails.logger.debug "Someone wants to connect via WebSocket!"
	  set_current_person
	  
	  unless current_person
	    Rails.logger.debug "I don't know who this is. Closing the WebSocket."
	    reject_unauthorized_connection
	  end
	  
	  Rails.logger.debug "It's, #{current_person.name}"
    end

    private

    def set_current_person
      person = 
	    # We can access the param of the HTTP request
	    Person.find_by(token: request.params[:token]) ||
	    # We can access the session of the HTTP request
        Person.find_by(token: request.session[:person_id])
        
      unless person
        # We can access the cookies of the HTTP request
        session = Session.find_by(id: cookies.encrypted[:session_id])
        person = session&.person
      end
	  
	  self.current_person = person
    end
  end
end

There is also a "#disconnect" method that gets called when a WebSocket is closed. It's useful if you want to set something up in "#connect" and then tear it down when the WebSocket closes. E.g. if you'd want to count the number of open connections that a person has, you could do something like this.

module ApplicationCable
  class Connection < ActionCable::Connection::Base
    attr_accessor :current_person
    
    def connect
	  set_current_person
	  reject_unauthorized_connection unless current_person

	  current_person.increment!(:connection_count)
    end

    def disconnect
      current_person&.decrement!(:connection_count)
    end

    private

    # ...
  end
end

Now you know who the WebSocket connection is for, but Rails doesn't. To tell Rails which attr_accessor(s) we'll use for identification we have to change our "attr_accessor" to "identified_by", which is a class method that Action Cable provides.

module ApplicationCable
  class Connection < ActionCable::Connection::Base
    #attr_accessor :current_person # CHANGE THIS
    identified_by :current_person # TO THIS
    
    def connect
	  set_current_person
	  reject_unauthorized_connection unless current_person
    end

    private

    # ...
  end
end

Ok, now we've got a connection and we know who it's for, how do we send a message? Do we have to send some kind of JSON or XML or whatnot?

The Wild WebSocket West

We are used to HTTP in our Rails apps. With it we can transfer text, images, audio, video and more from our server to the browser. But WebSockets are the wild west of the Web.

As the "socket" part implies, a WebSocket is similar to a Unix, TCP or UDP socket in that you can transfer whatever you like over it and in any fashion you'd like. It's up to you to implement a protocol between your server and the browser on top of your WebSocket.

Luckily we don't have to implement anything as Action Cable is not only a WebSocket server but also a protocol.

When you send a message to a client - or vice versa - the the object you are sending will first be turned into a JSON-compatible object like a String, a Number, a Hash, or an Array. Then it will be wrapped inside a Hash under a key called "message" together with some additional things I'll explain in a bit. Then that Hash will be dumped as JSON and sent as an UTF-8 encoded String, byte for byte through the WebSocket. On the other end the message is assembled into a String again and parsed into an object.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Raw message sent via ActionCable to the browser

Why do we need the wrapper Hash?

The wrapper Hash allows us to have different types of messages. The protocol has three types of messages that a client can send to the server:

subscribe
unsubscribe
message

And six kinds of messages that the server can send to the client:

welcome
disconnect
ping
confirm_subscription
reject_subscription
message

The "welcome" message is the simplest of them all. It's just a Hash with a key "type" that holds the value "welcome". All messages sent from the server to a client have a "type" field. This is always the first message that a server sends to any client that connects to it. It's used to indicate to the client that the WebSocket is functional.

{
  "type": "welcome"
}

The "ping" message is used to keep a heartbeat. It's sent to all connected clients every 3 seconds so that the connection doesn't become stale and therefore terminated by some load balance in the middle. And it enables clients to detect when they have disconnected from the server. If the client doesn't receive a heartbeat message within 6 seconds - that's 2 heartbeats - it will try to reconnect. In addition to the "type" field, a ping message will also have a "message" field which holds the Unix timestamp of the moment the message was sent. This timestamp isn't used in the official client implementation.

{
  "type": "ping",
  "message": 1705848059
}

The "disconnect" message instructs a client to disconnect from the server. In addition to the "type" field it has a "reconnect" field which holds a Boolean. If the value of this field is true, the client should immediately try to reconnect after it disconnects. This is useful in case an error occurs and the connection has to be restarted. And in addition to the "reconnect" field there is a "reason" field which holds a String that gives you a brief explanation why your client is being disconnected.

There are four possible disconnect reasons:

unauthorized - sent when "reject_unauthorized_connection" is called
invalid_request - sent when the initial HTTP request was malformed
server_restart - sent when the server is about to restart
remote - send when the client is kicked for whatever reason

{
  "type": "disconnect",
  "reconnect": true,
  "reason": "server_restart"
}

To explain the "message", "subscribe", "confirm_subscription", "reject_subscription" & "unsubscribe" messages I first have to explain Channels.

Channels

What are Channels?

Channels are Action Cable's controllers.

In a regular HTTP request you'd specify which method you want to call on which Controller by setting the request's path and method. E.g. if you'd want to call the "create" method of the "ArticlesController" you'd make a POST request to "/articles" and pass any parameters you'd like the new Article object to have.

class ArticlesController
  # POST /articles
  def create
    if Article.create(params.require(:article).permit(:title, :content))
      redirect_to action: :show, status: :see_other
    else
      render :new, status: :unprocesssable_entity
    end
  end
end

In Action Cable you specify which method you want to call on which Channel. To do that, you first have to subscribe to to a Channel and then you have to send a message to it.

class ChatChannel < ApplicationCable::Channel
  def post_message(data)
	  Chat::Message.create(
	    content: data[:content], 
	    poster: current_person
	  )
  end
end

(Note that we get access to "current_person" because we used "identified_by" in our Connection object)

Now if you want to invoke "ChatChannel#post_message" you first have to subscribe to the "ChatChannel" with a "subscribe" message.

The "subscribe" message has a "command" field instead of a "type" field. All messages sent from the client to the server have a "command" field. In addition to that it also has an "identifier" field which holds a JSON encoded Hash. The identifier Hash has at least a "channel" field which is the class name of the channel. But it can also have other, user defined, values which will become available in the Channel object through the "params" method.

{
  "command": "subscribe",
  "identifier": "{ 
    \"channel\": \"ChatChannel\", 
    \"room_name\": \"Ruby Zagreb\" 
  }"
}

These params can be used however you'd like. E.g.

class ChatChannel < ApplicationCable::Channel
  def post_message(data)
	  current_person
	    .chat_rooms
	    .find_by(name: params[:room_name]) # PARAMS FROM THE SUBSCRIBE MESSAGE
	    &.messages
	    &.create(content: data[:content])
  end
end

When someone subscribes to a Channel the "subscribed" method on the channel will be called. In it you can do various things, one of which is to authorize the subscription. Let's say that you want to reject subscriptions for chat rooms that a person isn't a member of, you could do something like this

class ChatChannel < ApplicationCable::Channel
  def subscribed
    @chat_room = current_person
	  .chat_rooms
	  .find_by(name: params[:room_name])
	  
	reject unless @chat_room
  end
  
  def post_message(data)
	@chat_room.messages.create(content: data[:content])
  end
end

If the "subscribed" method doesn't call "reject" the server will respond with a "confirm_subscription" message.

The "confirm_subscription" message has a "type", and an "identifier" field. The identifier holds the same value that was sent in the original "subscribe" message.

{
  "type": "confirm_subscription",
  "identifier": "{ 
    \"channel\": \"ChatChannel\", 
    \"room_name\": \"Ruby Zagreb\" 
  }"
}

If "reject" was called then the server will respond with a "reject_subscription" message.

The "reject_subscription" message has the same format as the "confirm_subscription" message.

{
  "type": "reject_subscription",
  "identifier": "{ 
    \"channel\": \"ChatChannel\", 
    \"room_name\": \"Ruby Zagreb\" 
  }"
}

Now that the client is subscribed they can invoke an action on the Channel using the regular message type.

The "message" type messages has two variants - one for messages sent from the server to the client, and another for messages sent from the client to the server.

If the client is sending the message, then the message will contain a "command" field with the value "message", an "identifier" and a "data" field. Again, the identifier holds the same value that was sent in the original "subscribe" message. While data can be anything depending on what the client has sent.

{
  "command": "message",
  "identifier": "{ 
    \"channel\": \"ChatChannel\", 
    \"room_name\": \"Ruby Zagreb\" 
  }",
  "data": "Hello, Zagreb!"
}

When the channel receives a message, it will check if it implements a "receive" method and if it does it will invoke it and pass the message's data to it.

Though, if the client sent over a Hash, then the Channel will check if the data Hash contains the key "action" with a String value. If it does, and the Channel implements a method with the same name, then it will be invoked to process the message's data.

{
  "command": "message",
  "identifier": "{ 
    \"channel\": \"ChatChannel\", 
    \"room_name\": \"Ruby Zagreb\" 
  }",
  "data": {
    "action": "post_message",
    "content": "Hello, Zagreb!"
  }
}

I'll explain the server-to-client message format in a moment.

Now we know how to send messages from the client to Action Cable. But how do we send messages from Action Cable to the client?

This is in my opinion the more interesting part. To send something we first have to create a stream to which we can publish messages to.

A stream is a PubSub channel. I guess it's called a stream in Action Cable because, semantically, this PubSub channel acts like a stream of messages for the client.

What is a PubSub channel?

PubSub (short for Publish-subscribe) is a common programming pattern for sending messages between objects. An object can subscribe to messages from, or publish messages to, a channel (sometimes called a topic). When one object publishes a message to a channel, all objects subscribed to that channel receive it. The message can be anything - a String, a Hash, a Number, or another Object.

In Action Cable, our Channel object is the subscriber and our application (controllers, models, jobs, ...) is the publisher.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Illustration of how PubSub works

To create a stream we can use either "stream_from" which creates a PubSub channel using a String identifier that we pass to it (e.g. "chat:1337"). Or “stream_for” which accepts an object (e.g. "Chat.find(1337)"). "stream_for" internally generates a String identifier for the given object and calls "stream_from" with it.

class ChatChannel < ApplicationCable::Channel
  def subscribed
    @chat_room = current_person
	  .chat_rooms
	  .find_by(name: params[:room_name])
	  
	reject unless @chat_room

    # creates a PubSub channel to which we can publish messages to from anywhere
    stream_for @chat_room
    # the above is basicaly the same as
    # stream_from "chat_room:#{@chat_room.id}"
  end
end

Now that we have a stream we can publish messages to it from anywhere using "broadcast_to" which accepts two arguments - the object we are broadcasting to, and the messages to broadcast.

ChatChannel.broacast_to(
  ChatRoom.find(params[:id]), # The same record that we gave to stream_for
  "Hello, Zagreb!" # The message I want to send
)

If you used "stream_from" to create your stream, then you have to use "ActionCable.server.broadcast" instead of "broadcast_to". It also expects two arguments - the stream we are broadcasting to, and the message we are broadcasting.

ActionCable.server.broadcast("chat_room:123", "Hello, Zagreb!")

You can also send a message directly from the Channel object using the "transmit" method.

class ChatChannel < ApplicationCable::Channel
  def subscribed
    @chat_room = current_person
	  .chat_rooms
	  .find_by(name: params[:room_name])
	  
	reject unless @chat_room

    transmit "Welcome back, #{current_person.first_name}!"
  end
end

Ok, but what happens when we broadcast or transmit a message?

Well, it's wrapped in a hash containing two keys - "identifier" and "message". The identifier holds the same value that was sent in the original "subscribe" message. While the message holds whatever you are sending to the client.

{
  "identifier": "{ 
    \"channel\": \"ChatChannel\", 
    \"room_name\": \"Ruby Zagreb\" 
  }",
  "message": "Hello, Zagreb!"
}

Then that Hash is turned into JSON and sent via the WebSocket.

When the client gets that message, it can figure out for which of its subscriptions it's for based on the "identifier", and then it can process the "message" however it likes.

In the browser

Now we have sent a message from the server, but how can we respond to it in a browser?

Action Cable ships with an official JavaScript client that enables you to connect to a server via Action Cable, subscribe to any channel, and receive messages. Just like its server counterpart, it hides the protocol from you and allows you to focus only on the messages.

To connect to a server you have to create a consumer, which is a wrapper around a WebSocket connection to our server.

You can create a consumer that will connect to "/cable" without any extra params just by calling "createConsumer".

import { createConsumer } from "@rails/actioncable"

const consumer = createConsumer()

If you want to pass extra params to the server's Connection object, e.g. for authentication, you'll have to pass the exact URL with params and all to "createConsumer".

import { createConsumer } from "@rails/actioncable"

// Fetches the user's auth token from a meta tag in the DOM
const token = document.querySelector("meta[name=auth_token]")?.content

// Takes the current URL,
// changes it's path to "/ws",
// and adds "?token=#{token}" as the query params.
// The result looks something like "https://example.com/ws?token=123"
const webSocketURL = new URL(window.location.href)
webSocketURL.pathname = "/ws" // usually it's "/cable"
webSocketURL.search = `?token=${token}`

const consumer = createConsumer(webSocketURL.toString())

Now that you have a consumer, you can subscribe with it to any channel you'd like. To subscribe, you have to create a subscription with an identifier.

If you remember from before, an identifier has to have a "channel" field, but can also have any additional fields you want and you'll have access to these fields as "params" in your Channel object.

import { createConsumer } from "@rails/actioncable"

const consumer = createConsumer()

// Subscribe to the ChatChannel 
// and pass { room_name: "Ruby Zagreb" } as params
consumer.subscriptions.create(
  { channel: "ChatChannel", room_name: "Ruby Zagreb" }
)

How do I process an incoming message with this?

Well, when your subscription receives a message it will try to call a "received" method of the subscription object to process the message.

You can create a received method on the subscription yourself, like so

import { createConsumer } from "@rails/actioncable"

const consumer = createConsumer()

// Subscribe to the ChatChannel 
// and pass { room_name: "Ruby Zagreb" } as params
const subscription = consumer.subscriptions.create(
  { channel: "ChatChannel", room_name: "Ruby Zagreb" }
)

subscription.received = function(message) {
  document
    .querySelector("#messages")
    ?.insertAdjacentHTML("beforeend", `<div>${message}</div>`)
}

Though this approach has a problem - if you get a message immediately as you subscribe, but before you create your received method, you'll miss that message.

Action Cable allows you to extend the Subscription object as you create it to avoid this problem. You can pass a second argument while creating a subscription. That argument has to be an object with which the newly created Subscription object will be extended. Extending an object in JavaScript is similar to including a mixin in a class in Ruby.

import { createConsumer } from "@rails/actioncable"

const consumer = createConsumer()

// Subscribe to the ChatChannel 
// and pass { room_name: "Ruby Zagreb" } as params
const subscription = consumer.subscriptions.create(
  { channel: "ChatChannel", room_name: "Ruby Zagreb" },
  {
    received(message) {
	  this.appendMessage(message)
    }
    
    appendMessage(message) {
	  this.messageContainer()?.insertAdjacentHTML(
		"beforeend", 
		`<div>${message}</div>`
	  )
    }
    
    messageContainer() {
	  document.querySelector("#messages")
    }
  }
)

But there are other events that you can process in your Subscription object besides receiving a message.

You can process initialization events, connect and disconnect events, as well as subscription rejections. Each event requires you to define a method to process it. Initialization requires an "initialized" method, rejection a "rejected" method, connect and disconnect a "connected" and "disconnected" method.

import { createConsumer } from "@rails/actioncable"

const consumer = createConsumer()

const subscription = consumer.subscriptions.create(
  { channel: "ChatChannel", room_name: "Ruby Zagreb" },
  {
    // Called right after the Subscription object is created
    initialized() {
      console.log(`A subscription to ${this.identifier} was created`)
    }

    // Called if the subscription was rejected by the server
	rejected() {
	  console.log(`The server rejected the subscription to ${this.identifier}`)
	}

    // Called when the subscription gets confirmed by the server
    connected(data) {
	  // The data object has a single property `reconnected`
	  // which indicates if this was a resubscribe after a disconnect
      console.log(`The server confirmed the subscription to ${this.identifier}! Reconnected: ${data.reconnected}`)
    }

    // Called when the WebSocket closes
	disconnected(data) {
	  // The data object has a single propert `willAttemptReconnect`
	  // which indicates if a reconnect attempt will be made or not
	  console.log(`WebSocket to ${this.consumer.url} closed! Will attempt reconnect: ${data.willAttemptReconnect}`)
	}

    // Called when a message is sent from the server
    received(message) {
	  console.log(`Received message: ${message}`)
    }
  }
)

There are also two actions that you can perform from your subscription - send messages to the server, and unsubscribe.

To unsubscribe just call "unsubscribe".

import { createConsumer } from "@rails/actioncable"

const consumer = createConsumer()

const subscription = consumer.subscriptions.create(
  { channel: "ChatChannel", room_name: "Ruby Zagreb" },
  {
    received(message) {
      console.log(`Received message: ${message}`)
      
      if (message.toLowerCase() === "avada kedavra") this.unsubscribe()
    }
  }
)

And to send a message call the "send" method with the message you want to send.

import { createConsumer } from "@rails/actioncable"

const consumer = createConsumer()

const subscription = consumer.subscriptions.create(
  { channel: "ChatChannel", room_name: "Ruby Zagreb" }
)

subscription.send({ action: "post_message", content: "Hello, Zagreb!" })

Thousands of connections, one server

Now you know how to connect to Action Cable and send messages back and forth. But if you have ever tuned a Rails application you probably know that there is a thread pool that determines how many simulations requests your server can process.

Does that mean that there is a maximum number of connections that Action Cable can handle?

Well, no. As long as your server has memory available it will be able to accept and serve more Action Cable connections. Granted, it will process them slower and slower, but it will work.

How does that work?

Puma, currently the most popular Ruby/Rack application server, is a multi-threaded server. This means that when you make a request to it, your request will be assigned to one thread that will process it and generate a response.

But, you don't want to have too many threads. Each thread running on your server gets a very short slice of time to run its code on the processor. The more threads you have, the more any one thread has to wait to get to run its code on the processor.

So the smarter thing to do is to have a preset number of threads - a thread pool - that can process requests. If a request comes in, and all threads in the pool are busy, the request goes into a queue. It will wait there until a thread becomes available. That's why you want to take as little time as possible to generate a response for a request. The more a request takes to process, the higher the chance that some requests will end up in the queue which means that your app feels slow.

But Action Cable turns requests into WebSockets, which can stay open for days, how don't we run out of threads in the pool?

That's the interesting part. Action Cable processes only the initial HTTP request in Puma's thread pool. After you get a 101 response from the server the request is "hijacked" and put into an event loop. So Puma's thread pool determines the maximum number of WebSocket connections that can be opened at once.

Hijack? Event loop? What?

Rails doesn't integrate directly with an application server (like Puma). Instead it implements Rack's protocol through which it gets requests from the application server and returns responses to it. This enables applications and frameworks like Rails to work with any application server like Puma, WebBrick or Falcon.

Rack is great for request-response interactions like HTTP, but a WebSocket isn't request-response like. It's a bi-directional stream of data.

To support WebSockets and similar stream-like protocols, Rack offers a way to read and write bytes directly to and from a connection initiated by a request - it's called hijacking the socket.

When you hijack a Rack request you get it's underlying socket. You can read and write bytes to the sockets freely. But this means that it's your responsibility to implement whatever protocol you want over that socket.

Action Cable hijacks the socket and passes it to Faye's WebSocket Driver library which implements the WebSocket protocol and it puts that socket into an event loop.

An event loop is a pattern for responding to events. You start an infinite loop in which you wait for some event, when it occurs you process it and the loop starts again.

In Action Cable's case it's waiting for bytes to read from the socket it hijacked. When they are available, it reads them and passes them on to the WebSocket driver.

That's easy enough for one socket. But to wait for any of an infinite number of sockets to have some bytes ready to be read would require an infinite number of threads - one for each socket.

To avoid spawning a thread for each WebSocket, Action Cable uses nio4r which notifies it when a socket has some bytes ready to be read, without spawning any threads. It does so using different functions of the server's operating system's kernel - such as select, epoll and kqueue.

Once it's notified that a socket is ready, it reads its bytes, parses them into a message, and then passes the message to the Connection object, which then puts in in a thread pool to be processed - called the worker pool - to process that message.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Illustration of all the pools that a request goes through

Another thread pool?

Yes. This worker pool allows Action Cable to process multiple incoming messages simultaneously.

You can tweak the size of this worker pool by setting "config.action_cable.worker_pool_size" to the number of threads you'd like to have in that pool.

The more threads in this pool the higher the number of incoming messages that you can process concurrently. More threads also means that your latency will go up (as one threads will have to wait for longer to get access to the processor), your database connection count will go up (as more threads can access the database simultaneously), and your memory consumption will go up as more messages and objects are in memory at the same time.

But there is one more thread pool in Action Cable - the event loop thread pool.

This one is used to dispatch events like sending heart beat messages, subscribing to and unsubscribing from PubSub channels created by stream_from, attaching and detaching hijacked sockets, and triggering periodic timers (to which I'll get to in just a bit).

This thread pool has a fixed size as it's intended to be an event dispatch of sorts where you just schedule a task which the pool only trigger, the execution of the task is mostly done in the worker pool.

Things I wish I knew right away

There are a few things that I learned about Action Cable that I wish I knew earlier.

You can trigger actions and send messages periodically, like every few seconds. This is extremely useful if you have some state that you want to synchronize periodically, or some house keeping you want to do.

In your Channel object you can call a "periodically" class method, give it a method name or a block, and an interval. It will then run that method/block for you with whatever interval you specified.

class StockTickerChannel < ApplicationCable::Channel
  # this will send a message to the client every 2 seconds 
  # as long as they are subscribed
  periodically every: 2.seconds do
    transmit value: @stock.value, timestamp: Time.now.to_i
  end
  
  def subscribed
    @stock = Stock.find_by(symbol: params[:symbol])
	  
	reject unless @stock
  end
end

Some error trackers won't catch errors from your Channel objects unless you explicitly send them. You can do that in your Connection using "rescue_from", just like you would in a Controller.

module ApplicationCable
  class Connection < ActionCable::Connection::Base
    rescue_from Exception do |error|
      MyErrorTracker.capture_exception(error)
    end

    # ...
  end
end

There is a callback on the Connection object for when an action gets invoked on your Channel object. You can register such a callback using "before_command", "after_command" and "around_command", which is extremely useful if you use Current attributes.

module ApplicationCable
  class Connection < ActionCable::Connection::Base
    around_command do
	  Current.set(person: current_person) { yield }
    end

    # ...
  end
end

There are callbacks on the Channel object for when someone subscribes or unsubscribes. There is "before_subscribe", "after_subscribe", "around_subscribe", and "before_unsubscribe", "after_unsubscribe", "around_unsubscribe". These are useful for implementing common behavior through inheritance or mixins without having to call "super" form the "subscribed" or "unsubscribed" method.

module AppearanceTrackable
  extend ActiveSupport::Concern

  included do
	after_subscribe unless: :subscription_rejected? do
	  Current.person.came_online!
	end
	
	after_unsubscribe do
	  Current.person.went_offline!
    end
  end
end

class ChatChannel < ApplicationCable::Channel
  include AppearanceTrackable
  
  def subscribed
    @chat_room = current_person
	  .chat_rooms
	  .find_by(name: params[:room_name])
	  
	reject unless @chat_room

    stream_for @chat_room
  end
end

You probably want to bump Puma's worker timeout if you are debugging your Connection object. Puma will kill any worker that doesn't generate a response within 60 seconds.

This can be annoying if you are trying to debug something with "debugger", "binding.irb" or "binding.pry".

But you can raise that timeout using the "worker_timeout" method in "puma.rb".

# config/puma.rb
require File.expand_path("../config/environment", File.dirname(__FILE__))

# ...

# Kill a worker thread if it didn't generate a responsw in 8 hours
worker_timeout 8 * 3600 if ENV.fetch("RAILS_ENV", "development") == "development"

You have to tweak both Puma's thread and Action Cable's worker counts. Puma's thread count controls how many new WebSocket connections can be created simultaneously. Action Cable's worker pool size determines how many WebSocket messages you can process simultaneously. Both pools will create database connections!

# config/puma.rb
max_threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
min_threads_count = ENV.fetch("RAILS_MIN_THREADS") { max_threads_count }
threads min_threads_count, max_threads_count

# config/application.rb
config.action_cable.worker_pool_size = ENV.fetch("RAILS_MAX_THREADS") { 5 }

In development, sometimes the server can behave a bit wonky. I'm not a 100% sure what's going on but it seems like code reloading can cause rogue threads with stale code. If that happens just run "touch tmp/restart.txt" to quickly restart the server and reset everything.

When a client looses it's Internet connection it will take Action Cable up to 20 min to notice that, close the connection, and trigger callbacks. I wrote about that in my previous article. This can cause you headaches if you want to allow only a certain number of connections per client.

You can use the official JS client outside of the browser, like in Node or Bun. The official client doesn't use the browser's WebSocket object directly. It exports an "adapters" object which has a property called "WebSocket" which holds the object that will be used to establish WebSocket connections. So you can drop-in your own WebSocket object and use it.

You can remotely disconnect any client. There is a "ActionCable::RemoteConnections" class that acts like an Active Record relation. You can query it using its "where" method to get a connection. The where method searches for a connection by its "identified_by" fields. If it finds such a connection it gives you a proxy for that connection on which you can only call "disconnect" which then disconnects the client.

class Person < Applicationrecord
  def ban!
    update!(banned: true)
    
    ActionCable::RemoteConnections
      .where(current_person: self)
      &.disconnect(reconnect: false)
  end
end

There are multiple PubSub adapters available besides Redis. You can use Postgres instead of Redis as a backend (it has some limitations). And you can provide your own if you need.

# config/cable.yml
production:
  adapter: postgres

Final thought

Action Cable is an amazing piece of software. It can seem complex, but that’s because it does a lot. When you look at each piece individually the complexity goes away.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Block overview of everything discussed

Tracking online presence with ActionCable

2023-11-14T05:17:00Z

This month I worked on tracking when a device connected and disconnected to a WebSocket backed with ActionCable. At first, this seemed like a simple problem to solve, but it turned out to be much more complicated.

I started with a simple solution, which was to mark a Device as being online in the #subsribed method, and marking it as offline in the #unsubscribed method. These methods are called when a client subscribes or unsubscribes to a channel. This is also the solution from the ActionCable guides.

class EventsChannel < ApplicationCable::Channel
  def subscribed
    Current.device.came_online
    stream_from Current.device
  end

  def unsubscribed
    Current.device.went_offline
  end
end

#came_online and #went_offline internally just update an online boolean column to either true or false, and also create a log record that the device came online or went offline.

class Device < ApplicationRecord
  has_many :online_status_changes, dependent: :destroy

  scope :online, -> { where(online: true) }
  scope :offline, -> { where(online: false) }
  
  def came_online
    update!(online: true)
    online_status_changes.create!(status: :online)
  end

  def went_offline
	update!(online: false)
	online_status_changes.create!(status: :offline) 
  end
end

And that works pretty well. If I connect as a device I can see a green orb appear next to to the device's name, and if I disconnect the orb turns red.

Demonstration of the device going online and offline within a simple presence tracking app

I soon discovered a problem with that solution. If I open the page on my phone, connect as a device, and then turn airplane mode on, the app will still think that the device is online even through the phone isn't connected to WiFi anymore.

Demonstration of the device staying online when the device is unplugged from the Internet

What is going on?

Well, most operating systems keep a connection "alive" and buffer messages for it if the other end suddenly disconnects (without closing the TCP connection). This is so that small hiccups in the network don't cause you problems, errors or warning messages. This connection state is known as half-open.

Depending on the OS, a connection can be considered held half-open in this way anywhere from a few minutes to nearly half an hour, or until the buffer overflows which can be dozens to hundreds of megabytes of messages.

Since I want to know when a device disconnects as soon as possible this buffer and the delay it introduces is a problem.

This behavior surprised me a bit since I know that ActionCable keeps a heartbeat which is supposed to detect sudden disconnects and prevent half-open connections from lingering.

Why didn't the heartbeat save me this trouble? I had to dive deep into ActionCable to figure out.

A heartbeat is a simple message, sent back and forth between the client and server every few seconds. It generates traffic, so the connection doesn't become stale, and if one side doesn't receive the heartbeat in time it knows that it has lost connection to the other side. The heartbeat messages are commonly called "ping" and "pong". The WebSocket protocol itself has a special ping & pong message, but most WebSocket libraries implement their own application-level ping & pong for various reasons I won't get into.

The ping message can be sent from the server to the client - this is known as a server-initiated ping - or from the client to the server - which is known as a client-initiated ping. Usually, whoever gets the ping responds back with a pong message. If one side doesn't receive a ping or pong back within a given time frame they assume that the other side has disconnected.

Turns out, in ActionCable, the heartbeat is server-initiated and one-directional. This means that the server periodically sends a ping to all clients, but it doesn't expect a pong back.

That's a bit curious, but it makes sense. The point is to help the client discover that it has lost connection so that it can reconnect. The downside of such a heartbeat is that the server can't know that the client has disconnected abruptly, so it can't close the connection until either the OS buffer overflows or times out.

In my case, just adding a pong response to the client, and the ability for the server to close any connection that didn't receive a pong within two heartbeats would allow me to detect a disconnect within 9 sec which is a great improvement over the 30 min from before.

So I monkey patched that in, and it worked pretty well.

Demonstration of the device going offline when unplugged when PONG messages are added

In my opinion this is a generally useful feature so I went to open up a PR to ActionCable and checked for any preexisting PRs or issues related to ping and pong messages. There were two closed issues (#24908 & #29307) and another open one.

While slightly different all of them seem to focus on the same underlying issue of handling sudden disconnects. But the proposed solution for them isn't a pong response, instead it's a switch to client-initiated heartbeats.

I was a bit surprised by the discussion leaning towards client-initiated heartbeats. Pong messages are a much simpler and less invasive solution which solves the same issue. So I read through all the discussions and did some research to understand what else client-initiated heartbeats bring to the table.

The issues highlight the following as advantages of client initiated heartbeats:

Ability to change the heartbeat on the client-side, this would allow some clients to do heartbeats more frequently to detect if the connection broke more quickly
The client could measure latency - the time it takes for a message to do a round trip - to the server

There were two other features mentioned:

Dropped message detection
Reporting of the last-received message

These are technically doable by adding a Lamport timestamp to each message, but they raise a new question "What do we do when we detect a dropped message?" which is application-level stuff and a major expansion of the framework, so I won't delve into these features.

I did a proof of concept implementation of client-initiated heartbeats, before I found out socket.io actually changed from client-initiated to server-initiated heartbeats.

Reading through the GitHub issue where the switch was decided, I learned that Chrome has started throttling setTimeout and setInterval for tabs that aren't in the foreground in 2021 to lower power consumption (extend battery life). When your app is in a tab that gets throttled the fastest that the heartbeat timer could run is once a minute. When that happens the client would send out heartbeats so slow that the server would think that it disconnected and it would close the connection, that would cause the client to reconnect, only to get disconnected again before the next heartbeat. Basically, the client would reconnect loop.

Because of this, I abandoned client-initiated heartbeats and returned to the original pong message idea.

There were a few things to iron out in the pong monkey patch. The most important one was forwards and backwards compatibility. The original implementation waited for the client to respond with a pong to switched the connection to expect pong messages. This allows clients that don't support pong responses to still function, while clients that know about pong responses get improved presence detection.

This is forwards-compatible (client that don't send pong messages can connect to servers that expect them), and somewhat backwards-compatible (clients that do send pong messages can connect to servers that don't expect them) if you don't mind the error log messages that this will cause (it won't crash the WebSocket).

But there is also a race condition in this upgrade process, when a client connects and then immediately disconnects before they respond to the first heartbeat. When that happens the server is stuck thinking that the client doesn't support pongs even though they do, and will wait for the OS to close the connection and improved presence detection is lost. That’s exactly what I don’t want.

The more elegant and robust solution would be for the client to signal to the server that it supports pongs when it’s opening the WebSocket. And for the server to respond back that pong messages have to be used, so that the client knows it should send them. That way I can mark the connection as expecting pongs right away, and the client can connect to servers that don't support pong messages.

Luckly, WebSockets provide a mechanism for exactly that - it's called sub-protocols. When a client makes a HTTP request to open a WebSocket connection it can send a list of sub-protocols to the server in a Sec-WebSocket-Protocol header. The sub-protocols have to be ordered by preference (the one you'd like the most first). The server chooses the first protocol from the list that it supports, and adds a Sec-WebSocket-Protocol header to it's response with just the chosen protocol. With that both the server and the client know exactly how to communicate over the WebSocket.

ActionCable already uses Sec-WebSocket-Protocol both on the server and in the official client. The name of its protocol is actioncable-v1-json. Since the pong message is a change to the protocol, I created a new protocol revision - actioncable-v1.1-json - which expects a pong response to the server's heartbeat ping, and updated the monkey patches.

The only thing left to do now was to open a PR to Rails.

The first step towards that was to expose the latency metric to the application somehow. Up until now I only measured the latency and logged it in the connection object itself. It would be nice if I could somehow hook into that metric so that I can process it and store it on the device model somehow. For this I chose ActiveSupport Notifications. I've added a new connection.latency notification, through which I can monitor, and respond to, the latency of any ActionCable connection.

The second step was to improve the naming of methods. As I developed the monkey patches I changed terminology a few times so the names are all over the place. I settled on calling such connections half-open instead of dead, stale or expired. That seems to be the most apt description.

The third step was to treat the connection as open if any message was received within the PONG timeout. After all, you can't receive messages if the connection is half-open, and if the server is processing a lot of incoming messages it might take too long to process a PONG message from some client and incorrectly think that the connection is half-open. There is also an open PR to add the same behavior to the official JS client.

The fourth step was to fork Rails, create a new branch from main, apply all the changes from the monkey patches to it, and add tests.

The tests were a bit odd compared to other core Rails gems. In all other gems you only have to run bin/test to run the whole test suite, but in ActionCable you also have to run yarn pretest, bin/test and yarn test. This isn't well documented, I stumbled upon this by accident when I intentionally wrote a failing test.

And the final step was to double check that I did everything in the contributing to Rails guide, and opening a PR.

With that out of the way, there is one more edge case that I've found. A device can suddenly lose connection, notice that, regain Internet service, and reconnect to the server, before the server closes the leftover half-open connection. When the server finally closes the half-open connection the device will be marked as offline in the database, even through it isn't.

To solve that all I had to do is change the online boolean column to a connection_count integer column. Every time the device connects that column is incremented, and every time it disconnects it's decremented. If the count is zero, the device is offline, if it's positive the device is online. The only thing that I had to do in addition to that is to lock the row before updating the count so that a race condition doesn't change the count incorrectly.

class Device < ApplicationRecord
  has_many :online_status_changes, dependent: :destroy

  scope :online, -> { where(connection_count: (1...)) }
  scope :offline, -> { where(connection_count: 0) }
  
  def came_online
    with_lock do
     old_count = reload.connection_count
     update!(connection_count: old_count + 1)
    end
    
    online_status_changes.create!(status: :online) if old_count.zero?
  end

  def went_offline
	with_lock do
     old_count = reload.connection_count
     update!(connection_count: old_count - 1)
    end
    
	online_status_changes.create!(status: :offline) if connection_count.zero?
  end
end

And that's it.

Rails World

2023-10-10T09:45:00Z

It was Wednesday morning and I got up even earlier than I usually do because I had to be on a plain to Amsterdam in a few hours. I haven't been to Amsterdam since 2019 when I briefly meandered through it's streets on my way to EuRuKo in Rotterdam. This time I was going there for Rails World which was special to me since I've never been to Rails-specific conference before.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Rails World banner

Two brief flights and a train ride and I was at Amsterdam Centraal train station. It was a cloudy and windy day. I made my way to Beurs Van Berlage - a beautiful, old, ornate, red brick building with a tall clock tower, right in the heart of the city - to register for the conference.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Beurs van Berlage

Then I made my way to the Zoku hotel where I would be staying until Sunday. The hotel looked modern from the Booking.com listing, but the pictures don't do it justice. To check-in I had to take an elevator all the way up to the 6th floor, which turned out to be a rooftop terrace with small ponds, a glassed-in walkway to the lobby, and a garden on each side of the walkway. The view was amazing, the clouds had cleared a bit, and the sun was just starting to set over the city.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The sunset over Amsterdam as seen from the rooftop lobby of the Zoku hotel

The room was very stylish. It had everything a regular apartment would have - a stove top, a kitchen sink, a dishwasher, a dining table, a TV, a speaker, a couch, an office desk, shelves full of books, a bed, plus a bathroom - but it was arranged in a very compact way which made it fun to just explore the room. For example, the bed is above the shelves and to get into it you have to pull out a set of stairs out of the shelves to be able to climb up. The office desk is in a cupboard under the bed. In fact everything under the bed is storage space with various doors to it that act like closets. And there were a lot of quirky design touches like the office desk chair being made of concrete, the light switch in the bathroom being a plunger on a red string hanging from the ceiling, or the athletic rings hanging from the living room ceiling.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

My room in the Zoku Hotel Amsterdam

The next morning, at 8:30 am, I made my way back to Beurs Van Berlage. There weren't many people out an about, which was strange to me. In Zagreb, there would be a flood of people in the streets at this time of day.

The venue was even more beautiful from the inside than from the outside. The red brickwork from the outside continued on the inside. There were four rooms, each of which was an atrium lit by sunlight coming through the colored glass ceiling held up by iron beams. Each room was three stories tall, and three of them had brick walkways on the first and second floor held up by arches and pillars.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The ceiling of the Beurs van Berlage

The opening keynote was about to start, so I made my way to the Track 1 room to find a chair. The room had moody lighting and a stage as wide as the whole room.

Amanda Perino, the executive director of the Rails Foundation, walked up on stage and give a captivating introduction. She explained what the role of the Rails Foundation is. Then she showed everywhere where people came from, the person that traveled the longest distance, and the person that traveled the shortest distance. People from all over the world came to Rails World. Finally she introduced the MCs, which in term introduced DHH.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The stage on Track 1 of Rails World

His keynote was a banger. It started off by introducing all the new features coming to Rails - solid_cache, solid_queue, Strada, Kamal, Propshaft, and Turbo 8 - and then switched to the concept of the “Renaissance Developer”, an analog to the Renaissance man or Polymath. A developer knowledgeable in many subjects - the backend, the frontend, deployments, and mobile apps - that can act as a one-person team. And Rails is exactly what enables such a developer. With money being tight as of late, these are the people we need to make great software.

The idea of a Renaissance Developer deeply resonated with me. I always like to broaden my knowledge, learn new thing, learn how things work, how people solved some problems and what were their reasons behind making certain decisions, and then dive deep into topics that interest me most. But I was always met with advice like “specialize in one thing, because that’s financially smarter” or “don’t be a jack of all trades and a master of none”. This never stopped me from perusing knowledge, but it wore me down to hear that I was going in “the wrong direction” every other day. Hearing DHH, and later people in the hallways, talk about the return of the Renaissance Man - the Renaissance Developer - was a breath of fresh air, inspiration, and motivation. I felt like I wasn't the crazy one after all.

I went to the talk about Strada by Jay Ohms next. He gave an introduction into using Strada to build iOS and Android apps. Explained how it interacts with Turbo native and Rails/HTML, the design decisions behind it and trade-offs. Like, it being designed to easily deploy new versions of the app without going through AppStore reviews, and having backwards and forwards compatibility in mind. It seems to be a really solid way to build mobile apps. It doesn’t yet fully cover offline, but it seems that there is a plan how to expand into offline territory with local storage and IndexedDB. All in all a great talk.

I learned how to hide UI elements using CSS and move them to native controls instead. You can do this because Strada updates a data attribute on the HTML element with a list of components it supports, so you can check if a component is enabled or not using square brackets and =~ selectors. Basically you implement a native method that adds a navbar button, for example, with which elements it should have and what it should send back when a button is pressed, from there you click the corresponding hidden UI element with JS and that’s it. It’s simple and that’s what I love about it. Everything we know and use in the web daily is translated over to mobile with a thin wrapper to control native UI elements.

Next, I went to Jorge Manrubia’s talk about Making a difference with Turbo. His talk was great. He explained all the new features in Turbo 8 with a focus on Turbo Morph. Basically, they are building a calendar for Hey (if you watched closely during DHH’s keynote you could notice a calendar button in the top center of his Imbox) and rendering it with turbo streams would have been difficult so they looked for simpler solutions and developed Turbo Morph in the process.

Generally, the easiest thing to do is to refresh a page when something updates - that's the old-school HTTP request-response way. But a full refresh is a jarring experience - the page flickers, you lose your scroll position, opened menus close again, CSS animations reset.

The next easiest solution are turbo frames, which allow you to update the page in sections as you interact with that section. This is similar to a full page refresh though without most of the jarring side effects, but you have to keep track of when you want to break out of the frame and that each page you can navigate to from the frame has a frame of the same name in it.

Finally, the hardest solution are turbo streams which are basically raw HTML manipulation events from the server. You have to know in which context you are sending them so you manipulate the HTML correctly. But they are the least jarring and feel like a native app.

Now with Turbo Morph you have the same native look and feel as turbo streams with the ease of use of a full page refresh. Morph basically does a full page refresh, takes the new HTML and morphs the current HTML into the new one. This preserves CSS animations, scroll state, open menus (with a flag), and doesn’t flicker the screen. All of this is done using a library called idiomorph. They were inspired by Phoenix’s LiveView does the same thing just using a different library.

I’m hyped for the release of Turbo 8. Morph will make my life so much easier. I can basically delete most turbo stream responses at work and simplify the code a lot.

Then came the lunch break. The food was amazing! The catering service deserves a lot of praise. There were many different options and all of them were delicious.

After lunch, I went to Marco Roth’s talk about the future of Rails as a full-stack framework powered by Hotwire. His talk was also great. He focused more on the ecosystem around Hotwired, and how he came to Hotwired. He added a lot of nice user-friendly errors to Stimulus, like when you have a typo on a controller name or action it will tell you that instead of doing nothing. He also made an LSP for stimulus which looks amazing. It can suggest you completions specific to your app, like the exact controller, action, class, value or target name. He is also starting a weekly newsletter about Hotwired. And he started hotwired.io to aggregate and share all the cool thing in the ecosystem and community.

Then I went to Alivija Rojas’ talk about an offline experience with a Rails powered PWA. It was a very insightful talk about how to build a progressive web app with Rails, that caches certain pages locally and stores any form submissions made with no Internet to be synced later when Internet service is available. This can be done with a service worker with which you cache the index and new pages. Then you add a Stimulus controller on the form that checks if Internet service is available before submission and if it isn’t then you store the form data in Indexed DB. On the index page you check what you have in your local storage and render that. A neat idea.

Next, I went to Xavier Noria’s talk about Zeitwerk’s internals. I liked his presentation style a lot and enjoyed the talk. I did read through Zeitwerk some time ago so it was nice to hear the explanation of each concept in the codebase and how some tricky situations are resolved when loading in classes and modules.

For the last regular talk of the day I went to Breno Gazzola’s talk about Propshaft. The goal of propshaft is to simplify asset serving a lot. It intentionally doesn't to anything besides generating digests and serving pre-processed files. This is in response to Sprockets, which do everything from integrating with Gulp to compiling coffeescript.

Finally, the closing keynote by Eileen Uchitelle. It was the same keynote she gave at RailsConf this year. A very inspiring talk, that motivated me to contribute to the framework more when I could. This talk also reminded me that I had an open PR to Rails that I forgot about.

That was a wrap for day one so I walked back to my hotel room. Now there was a sea of people in the streets, seems like Amsterdam comes alive in the afternoon.

There are a few things I’ve heard that I didn’t go to a talk to or that didn’t have a talk about them. First, solid_queue is essentially, or at least is inspired by, good_job but will be database agnostic. And it's coming by the end of the year. Second, solid_cache is a database backed cache. The speed of retrieving data is worse than e.g. caching in Redis, but we are talking about 1.2ms instead of 0.8ms, nobody is going to notice that difference. The benefit of using the database as a cache is that your cache can live for much longer because SSDs are larger and cheaper for storage than RAM. Having longer lived caches means that you have to compute results less often which can dramatically decrease response times.

The next day came around and I made my way back to the conference for an AMA with the Rails Core Team (10 out of 12 members were present). The AMA was lead by the creator of OhMyZSH, Robby Russle. It was both inspiring and funny. Inspiring because the core team had nothing but words of encouragement towards the public, they want to help people contribute to Rails and know that some things can be daunring to beguinners which they could and are improving. And funny because of the answers, haggling with the host - especially between Aaron Patterson and Robby - and the bloopers. There was a wonderful moment when Robby asked all the contributors to rails to stand up, and about 10% of the audience stood up. Then he asked that anybody that has ever commented on a PR or similar to stand up, and another 20% of the audience stood up. Around 30% of the audience contributed to Rails in some way, which is a lot.

Next were two talks that were both very interesting and which I wanted to attend but they were happening at the same time - that was kind of the theme of the second day.

I went to the Rails and Ruby Garbage Collector talk by Peter Zhu first. I met the guy at EuRuKo last year and he seemed excited about this topic then so I figured this would be a great talk, and it was. He started off by introducing how GC works in Ruby. I wasn’t there for the rest of the talk but I heard that he created a gem called autotuner to help you tune your Rails app’s GC.

Midway through Peter Zhu’s talk I switched to the Powerful Rails features you might not know talk by Chris Oliver. This was a fun talk to listen to, Chris is an excellent presenter and speaker. He showed many interesting tips and tricks that you can use in Rails. So many that they’d warrant a post of their own to cover.

I can’t wait for the videos to come so that I can watch both talks in full.

I didn’t go to any of the next two talks that were next because I met a few people I knew from other conferences and some that I met yesterday so we chatted for a bit. This was also perfect timing since lunch was up next and we could get a head start on the food before queues form.

After lunch, I went to Ryan Singer’s talk about Applying Shape Up in the Real World. He sells a course that seems to be a much more through version of this talk. Shape Up formalized most intuitions I already had about project management and taught me very helpful tools and processes for project management and development that are so simple and obvious that they are just beautiful, so I was looking forward to this talk. The talk was amazing, in my opinion it could be a keynote.

He started off with introducing all the problems of modern project management and development and drew analogs from that.

Like, “working on tickets is like putting your project plan through a paper shredder”. I can’t state enough how strongly I agree with this analogy. I’m amazed that people don’t notice that most problems during a project occur because nobody sees the big picture and decisions are made basically in the dark.

Next he had an analogy about cooperation which deeply resonated with me. “Imagine renovating a room and spending moths laying out the new interior and particularly picking a lamp that will hang from the wall above the couch. Only to show the plan to the electrician and for him to tell you that there is no wire in that wall and that he’ll have to rip up the wall if you want to put the lamp there”. I’ve been so many times in a spot where somebody just threw something extremely hard to implement over the wall for me to solve. This was always made worse by working on tickets since often times I didn't notice a problem until I was in the middle of the project. Just asking the team how hard something is to implement is a huge improvement in cooperation between project management and development.

There were so many nuggets of wisdom in this talk, it’s well worth the watch when the video comes out.

Next, I went to Adam Wathan’s talk about Tailwind. It was an introduction to Tailwind with some tips and tricks along the side, all explained and shown through live coding examples. Respect for pulling that off, it was an excellent talk.

Then I went to Irina Nazarova’s talk about making profit on OSS. This was a very interesting talk about which strategies you have if you want to make money on OSS and how you can bootstrap yourself. Among many other things, I learned that you can ask for sponsorship when a company asks for a feature on your project.

For the last regular talk on Friday I went to Don’t call it a Comeback by Jason Charnes. It was a very inspiring talk about the comeback of Rails with a reminder that we have to work to keep it that way and not take it for granted. We have to help people learn Rails, make the guides better, improve documentation, make videos, record podcasts and more to keep it relevant.

The closing keynote was Future of Developer Acceleration with Rails by Aaron Patterson. The talk started out with a lot of jokes poking fun at the opening keynote.

Then he turned to his aggravated onion, a personification of layers of things he’s mad about. It was fun to hear him rant for a bit. He mostly focused on how rewriting things to other languages is way more complicated than improving the algorithm something uses and it doesn’t actually solve the fundamental issue - a slow sort is slow in Rust and in Ruby. That people think that interop between Rust and Ruby is safe when in fact it comes with new problems, like memory leaks. And concluded that we all should just enable YJIT since it gives a great performance boost for no extra cost.

After that he turned to profiling. With profiling you can figure out which parts of your app are slow and should be optimized. He went over how a sampling profiler works and how to read a flame-graph and a call tree. Then he introduced Vernier, a sampling profile made by John Hawthorn and himself. Vernier is easy to integrate into Rails and comes with a nice UI to help you figure out what’s going on in your app.

Finally, he turned to language servers. He created a language server for Rails that shows you model attributes, routes and other useful information right in your editor. This is now part of ruby-lsp-rails which is in term an addon for for ruby-lsp. His goal is for Rails to have one standard, built-in, LSP so that developers have the best experience possible out-of-the-box.

Except for the aggravated onion, these are things that he covered on his live streams in the past few months.

And that was it. Just the 20th anniversary celebration was left. The lights in the rooms were dimmed, and there were celebratory cupcakes given out. I was so tired that I decided to go to the hotel room after 15min, but I heard that the party was a blast.

I met a lot of new people, and a few old friends.

I got to chat with Chris Oliver who’s podcast - RemoteRuby - is my go to Ruby podcast. It’s a blast to listen to him, Jason and Andrew. I even got a hand crafted, artisan, limited edition podcast sticker.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Remote Ruby podcast sticker

I also met DHH and Rafael Franca. Though the only thing I mustered to say to them was “Thank you both, Rails really changed my life”. I wanted to ask and say so much more but I was too tired to think at that point. And decided to stop there and go rest.

Meeting these people really humanized them. Until then they almost seemed super human, but talking to them, meeting them in person, and just seeing them in the crowd really showed me that they are just regular people which is inspiring to me.

I think Rails World was a huge success. The talks were so amazing that it was hard to pick which one to go to. Everything was well organized. The people were extremely nice. The volunteers/employees were super helpful. It wasn’t exorbitantly expensive (except for the hotels, but that's not their fault) and the location was great. Overall, just amazing. Amanda Perino did an outstanding job. I’m glad that I could attend it, and be there for Rails’ 20th anniversary - 20 years and still going strong!

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Rails' 20th anniversary pin

The next morning was a busy day. It was my last day before going back so I wanted to tour the city and enjoy the museums. The Rijksmuseum is my favorite museum, I’ve been to it three times already, but I wasn’t visiting it this time. This time I bought tickets for the Van Gogh museum since I’ve never been there before.

The museum was huge. There are three floors full of his and his contemporaries' works. The first floor goes over his early life and how he got into painting. He used to be an art dealer, and at the age of 27 decided to become a painter.

The second floor covers his life as a painter, his struggles, and how he searched for his style. I learned that he was influenced by Japanese art, which he and his brother collected and admired. He traveled around but found happiness in the countryside and the simple life of peasants. On this floor are the sunflowers, one of his most famous works.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

My girlfriend in front of the sunflowers by Van Gogh

On the last floor are the works immediately before his death, and the work of authors that he inspired. His painting career was relatively short lived with him committing suicide 10 years after becoming a painter. On this floor there were a number of his most famous paintings like the almond tree blossoms and the wheat fields.

I was surprised to see that most paintings were covered by glass. This really sucks because Vincent put on very heavy brush strokes which gave his pictures a texture, but through the glass this is barely noticeable due to the glare. I mean, it makes sense to protect these paintings when people throw tomato soup on priceless works of art, but it still sucks.

Something that I never noticed in prints of his work is how good he was at capturing depth. His paintings draw you in, there is a 3D effect going on that just isn’t visible in a photograph of the same painting. And the colors on the paintings seem much more vivid in person.

There was a crossover exhibition going on too. Since Vincent was inspired by Japanese art, the Pokemon illustrators made a few Pokemon cards and paintings in his style. Needless to say, the gift shop was raided for anything and all Pokemon.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Pikachu with a hat, drawn in the style of Van Gogh

Next up was the the De Hortis, Amsterdam’s botanical garden. It’s one of the best botanical gardens I’ve been to. The garden has 7 glass houses, an outdoor park, and bee hives. There is a tropical, sub-tropical, and desert glass house. The desert plans look like they came from an alien planet. While the tropical glass house had gigantic plants and was full of small brown singing frogs.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Gian lily pads from the tropical glass house at the De Hortis

There was a very beautiful glass house with all sorts of plants.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A glass house at the De Hortis

And a glass house that served as a nursery for butterflies with dozens of them flying about pollinating plants.

By now it was already late in the afternoon so I went back to the hotel to pack up.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Walkway from the elevator to the lobby of the Zoku hotel

The next morning I made my way to Schiphol airport and then back to Zagreb.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Terminal B36 at Schiphol Airport

Hold your own Poison Ivy

2023-08-28T05:00:00Z

After two years, this week I finally caved and changed my current employment title on LinkedIn from Software Engineer to Software Architect.

This isn’t a position where I'm am supposed to do much programming - the thing I love most, and the thing I'm good at - but rather make high-level technical decision and manage people to achieve them. My problem with that, and any position like that, is best captured in the Croatian proverb "It's easy to hold poison Ivy in somebody else's hands" (that's the SFW Americanized version, here's the NSFW original).

That proverb describes a decision made by someone who isn't involved in the execution of the decision, and doesn't suffer its consequences.

I don't want to be the kind of person who just lobs some solution over the wall and lets other people deal with the fallout.

I’ve had "managers" like that in the past, and it was hell working with them. Being told to do something that you know will cause problems, then bringing that up for discussion, explaining all the implications and new problems we will face, only to be shutdown with some hand-wavy explanation, and then having to deal with the consequences of that decision months or even years later is soul crushing. And if they can’t admit to their mistakes it can become burnout inducing.

And I know that engineering isn't really about solving problems as much as it’s about displacing them. We are constantly trading one problem for one or more other problems. But we try to do so conscientiously, weight the pros and the cons, and pick a solution that produces problems that we can live with.

The problem with Software Architects, or any purely managerial position, is that you aren't eating your own dog food, and therefore you don't live with the consequences of your decisions. So you produce misfit solutions.

The problem of displacement is compounded by the existence of designers—special people whose job it is to solve problems, in advance, for other people. Designers, like landlords, seldom if ever experience the consequences of their actions. In consequence, designers continually produce misfits. A misfit is a solution that produces a mismatch with the human beings who have to live with the solution. Some mismatches are downright dangerous.

— Donald C. Gause, Gerald M. Weinberg in Are Your Lights On?: How to Figure Out What the Problem Really Is

After two years, I’m confident that I’ve managed to avoid becoming that kind of “manager” by holding my own poison Ivy, and by helping others make decisions for themselves.

When I'm presented with a problem, I try to implement a proof of concept solution my self. Not a demo, held up with glue and duct tape; but a proper minimal implementation. Then I present the solution to my coworkers and gather feedback. That way I can feel the consequences of my solutions.

When someone comes to me with a problem, we go over all the possible solutions that we can come up with, then we discuss the pros and cons, and that's it. I let person decide for themselves which solution they will go with. All I ask of them is to tell me which one they chose. The point is to shine some light on problems that the person maybe couldn't see on their own, and point out some bigger-picture things that they might not know about. That way they can make an informed decision and choose the solution with which they can live with - so there aren't any misfit solutions.

This isn't a silver bullet by any means. Bad decisions still get made, I fuck up, my coworkers fuck up. But we learn, fix stuff, build trust, and improve.

Rent in Zagreb is hell

2023-07-31T12:00:00Z

Last month my landlord called me and told me that I had to move out because he was going to sell the apartment I live in.

In a way, this came as a blessing. I wanted to move out of that place by the end of the year either way. But at the time I had no clue what hellscape finding an apartment in Zagreb had become since I last moved. This is a cautionary tale of what to look out for and what to avoid when renting in Zagreb.

I immediately started looking for a new place on Njuskalo (a local classified ads site).

The first thing I noticed was that rent has gone up by about 25%. I read about that in the news, so that didn’t surprise me too much. But what the news articles didn’t say is that for 3 bedroom apartments the price went up by 100% or more, which sucks because I always go for a 3 bedroom apartment so that I can have an office. Having the ability to clearly separate my work and my life environment is important to me.

After some searching I found a few reasonably priced apartments and arranged to go see them.

Chapter 1: Paradigms & Marketing

Seems like in the years since my last move most landlords took a philosophy course and decided to tackle the age old question of “What is the ground floor of a building anyway?”.

The first apartment I went to see was supposed to be on the ground floor of a small two story building. But upon entering the building - from the street - the real estate agent took me down a stairwell to a floor with a single door, opened it, and we stepped into the apartment.

“Very weird ground floor” I told him, “It’s almost 3 meters underground”. The real estate agent responded without hesitation with a happy but somewhat hostile sounding tone “Depends on what you consider the ground floor to be. This apartment is on the ground, and it has windows to the outside which are at waist height, it even has an exit to the garden. By all means this is the ground floor.”

I wasn’t in the mood to discuss paradigms like Physics professors defending the foundation of their lives work at some conference, so I sad my goodbyes and left.

Little did I know that this would just be the first in a series of apartments that were on the ground floor while at the same time at least one stairwell below the entrance and 1+ meters underground.

There were two camps of people; one believes that the ground floor is what's at the main entrance and everything above or below that is either + or - 1 floor, while the other camp believes that the ground floor is everything that touches the ground.

I am a firm believer in the ground floor is at the main entrance, but I was thrown into some bizarre world out of Kafka's fever dreams and just didn't know it yet.

How could a fundamental fact like this be subject to relative interpretation? I tried to see the the other camp's perspective - to understand their paradigm - but no matter how I sliced it this was just complete madness.

If everything that touches the ground is the ground floor then some buildings would have two ground floors or more - one on the street level and another in the basement - which made no sense at all. How would you explain to someone where you live?

How would someone know by looking from the street how many stories the building had, or even if they entered it? There would have to be a label on the building that warned visitors that there were more stories below ground just for them to be able to find the right floor without a map.

I already imagined a conversation like this: “See George, if you are one of the street-entrance-is-the-ground-floor folk, then I live on the 2nd floor; but if you are one of the-lowest-apartment-on-the-ground-is-the-ground-floor folk then I must tell you that there are two stories below street level in this building so I live on the 4th floor; but, on the other hand, if you are the there-are-multiple-ground-floors kind of guy then I', on the 2nd floor counting from the street entrance”.

Of course this wasn’t madness, this was just marketing.

Dugouts, and basement apartments aren’t hip in Zagreb anymore. Nobody in their right mind would pay 800 Euro or more to live in a damp basement, have to listen to 3 families living above their head, and battle mold all day long. When for 850 euro you can live on the 1st or 2nd floor of the same building, or buy your own apartment (while interest rates were low).

That’s a problem for real estate agencies as they are paid by commission, and for landlords as they want to get the most out of their property. But Zagreb has a real estate problem - price per square meter is comparable to Berlin, yet over half the buildings are 70 to 130 years old, and most of them are in a bad shape from being neglected for years due to co-owners not being able to agree on anything.

(seriously though, in my last building the co-owner's counsel couldn't agree on the color of the front gate for 2 months and then they just gave up and let the gate rust)

So realtors and landlords shifted the paradigm and the basement suddenly became the ground floor. In a way this is just a white lie. If you really like the apartment and the price is right for you, all they did was bend reality a bit to get you to see the apartment. But in the process they are gas-lighting everybody and inflating housing prices.

Chapter 2: Lies, lies, lies!

No matter which apartment I went to see all of them were built after 1963. This was odd for Zagreb - a city known for it’s extremely old buildings.

For apartments in Zagreb, and Croatia in general, knowing that a building was built after 1963 is very important due to earthquake code.

Before 1963 the code didn’t cover earthquakes of magnitude 6 or 7. And with a recent 5.4 magnitude earthquake in 2020 that severely damaged a lot of buildings, people don’t want to pay as much for a building that could collapse on their head at any moment.

So landlords started lying, and over night all buildings got 50 years younger. A case so curious that Benjamin Button would be surprised.

Some didn’t outright lie and just said that they don’t know when the building was built. This is very odd since they invested several hundred thousand euros into that apartment, one would expect them to do due diligence.

But some outright lie. One apartment claimed to be build in 1965, but on Google Street View it was clearly visible that the facade had a stamp with the name of the architect that designed the building and the build year - which was 1898.

In my opinion this should be a criminal offense, as this lie could get someone killed when they thought that they were perfectly safe (and that they were paying for that safety).

This was the worst lie I encountered, but it wasn’t the only one.

People lied about the building’s construction. For example, wooden beam floors and ceilings become concrete floors and ceilings. One is paper thin and through it you can hear your upstairs neighbor like they are your roommate, while the other blocks all sound but direct impacts on the floor above.

Wooden windows become PVC windows. Two single pane windows become double pane windows. And stuff like that.

The funniest lie I saw was a "Modern *rustic* apartment in the center" which was actually an apartment in which somebody's grandmother lived until she passed away (recently) and they decided to rent it out with her furniture. This is usually OK, but the furniture was from the 60's, they charged for it like the place was freshly renovated, and to explain that they just slapped on "rustic" in the name of the ad like it would magically explain that the cabinets were made before nuclear reactors were invented and are starting to fall apart.

Chapter 3: *Upkeep not included

For a number of years there has been a growing number of landlords that don’t take on the upkeep but instead expect the tenant to pay it. They claim that upkeep is a utility like water or electricity so they exclude it from the rent.

Upkeep can be 30 to 150 euro or more, and it can change at any time. So an 850 euro apartment plus upkeep can be as expensive, if not more, as a 1000 euro apartment with upkeep included.

But the real problem is that upkeep isn’t like a utility. I, as a tenant, can control which utilities I use and how much I use them. And I consume the utility that I pay for.

But I can’t consume the upkeep and I can’t influence how the upkeep is spent. Only the owner of the apartment can do that.

This is like taxation without representation. The upkeep can be 30 euro today, and my landlord lobbies for a new facade for the building which then raises the upkeep to 150 euro and I'm stuck with the bill for it. I don’t get to keep the facade when I move out, and I don’t get to vote on if the building gets a new facade or not. So why should I pay for the upkeep?

This trend is just another way for landlords to reduce their risk and offload it to the tenant. It ensures that the landlord gets a constant sum of money every month, regardless of what the other co-owners of the building decide to do or if something in the building breaks. It also enables them to improve their property on the tenant's expense without having to renegotiate rent.

Chapter 4: Parking in my apartment

A few apartments seemed quite small for their advertised size. For example one 60 square meter apartment seemed more like a 40 square meter apartment to me.

So I asked the real estate agent what was going on, and he explained that the two out door parking spaces (8 square meters each) count towards the total area of the apartment.

Turns out that this is completely normal, and has been like this since forever. But to me this is utterly insane.

A parking space isn’t the same as a room in an apartment or in a garage. And I can’t believe that this requires explaining.

For one you can’t leave stuff unattended in a parking space. Just imagine leaving your laptop in a parking space and then complaining to the police that it was stolen while you weren't there. Or imagine one of your neighbors grilling on their parking space, or building a shed, or something like that...

But it seems I’m the crazy one here. Just keep this in mind when comparing two apartments by their listed size.

Chapter 5: Contracts and Xenophobia

I was down to a few apartments and started negotiating lease contracts when I learned that another major thing has changed in the years since my last move - contracts.

Before, it was normal to have a lease contract that basically said that you are leasing the place, that you have to give one month’s rent in advance as insurance, that you will return the place in the state you found it, and that by the 15th of each month you have to pay rent and utilities to the landlord.

Today, you have to sign a contract which gives the landlord the right to seize money from your bank account (or anything you own in case there isn't enough) in case you miss payment or owe them anything. This is in addition to a one month’s rent as insurance.

When I consulted a lawyer they warned me that the “in case you miss payment” is arbitrary and that the landlord can actually seize any amount whenever they want as long as the contract is active and they have semi-valid evidence to do so.

A government body does the seize in the name of the contract, they would give me 7 days to produce evidence that the seize isn’t valid after which they would just deduct money from my account and send me a processing fee for it. The problem here is that producing evidence is fairly easy for both sides. Just think about it, how would you prove that you are missing money? Or that it wasn’t paid in cash? Or deducted for a repair or other? So most cases end up in favor of the landlord and the tenant has to sue the landlord to reclaim the money, which can take years.

This if of course completely insane. Who in their right mind would allow a random person they don’t know unrestricted access to everything they own for the privileged to live in an apartment that they happen own?

When I asked why this was needed I got just one reply from all landlords “We had bad experiences with foreigners“.

So… xenophobia and lies.

Obviously I’m not a foreigner yet I’m still required to sign such a contract. But even if I was, how would this be any insurance? The Croatian government has only jurisdiction in Croatia - if a Canadian signs the contract, moves in, skip a months rent, then moves back to Canada, you’d still have to sue them and drag me to court to get your money.

So this is again just another new way of shifting risk from the landlord to the tenant.

Things can happen, people lose jobs, people get sick, people die, but the landlord doesn’t care about any of this; they just want a sure way to get their rent and this contract ensures it.

Another interesting addition to some contracts was a non-termination clause. In other words the contract can’t be terminated by the tenant, if they decide to move away before the lease expires they have to pay the lease in full for the duration of the contract. And, of course, leases are non transferable meaning that you can’t find them another tenant to replace you, and neither can you sub-lease your apartment.

This is slavery, and should be illegal. I can’t know going into a lease what will happen 9 months down the line. And I can’t know if the landlord lied to me before I move into the apartment, or what the neighbors are like. Yet I am expected to lock in for at least a year of rent, give unlimited access to everything I own, based on a 15min tour of the apartment… insanity…

This isn't normal, and this was never normal. Avoid such contracts at all cost. If you really like the place offer up two month's rent as insurance and see if they go for it.

Chapter 6: Living in a forest

My dream is to one day live in a house in a forest, or at least very close to one.

After a month of searching for an apartment I found one that’s in a forest, in a new building, made out of concrete, has a balcony, has an acceptable price, has a bog standard lease, has upkeep included, and is in a quiet neighborhood!

But it only has two bedrooms, and it’s 40min by foot to the closes store (a very European problem since until now I didn't have a reason to own a car).

I decided to go for it, and I’m so glad I did.

Now I enjoy my morning tea looking at the forest and the fog that emanates. The cold night air carries scents of leaves and lulls me to sleep. It's much easier to recharge and take a break than it was in a city apartment.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

View from the balcony at night

Corporate newspeak

2023-04-26T06:01:00Z

If thought corrupts language, language can also corrupt thought

- George Orwell

One thing that annoyed me anywhere I worked was that people limit themselves to a narrow subset of English. A subset that's full of buzzwords and phrases that sound impressive but lack meaning. It's a subset designed to limit your ability to say anything that could be misinterpreted, and anything that expresses any emotion beyond a notion of happiness or dissatisfaction. This language couldn't mimic George Orwell's Newspeak more even if it tried, and therefore I call it "Corporate newspeak".

What annoys me is that speaking with someone in corporate newspeak feels like we are speaking in memes. We aren't saying what we think, we aren't being honest with one another, instead we are communicating in vague concepts and ideas. We are obscuring our though instead of clarifying them.

Take "We don't have the bandwidth" as an example. What does that sentence mean? It could be "The network switches in our server farm can't handle any more network traffic", or it could be "There isn't anyone available to work on this project in the next month, we would have to pause another project", or "Your project isn't something I think we should invest time in right now". There is no way to know without asking for clarification. But that will probably yield an answer with more newspeak and more buzzwords. Miscommunication is the goal of newspeak. If no one knows exactly what was said then there is a lot of room to avoid any kind of conflict.

Another thing that annoys me with newspeak is that it makes everything sound insincere, hollow, and boring because the vocabulary is made up of cliches. Take "We have an exciting announcement. From next quarter our Widget 3000 will come in two new colors - blue and green" as an example. Is that really an exciting announcement? I mean, it could be. But the excitement is undermined by every other announcement that was made that day that was also "exciting". If everything is exciting then nothing is exciting - it rings hollow.

Some find newspeak to be sincere, but I don't and maybe that's a culture difference. I find only language that I would use every-day sincere. And I don't know a single person that uses newspeak to talk to a friend, or a cashier, or a random person in the street. Maybe that's because of where I grew up? In Croatian the only difference between how you would talk to your friend and a business client is that you'd us the formal "You" instead of the regular "you" and you'd use "Greetings" instead of "Hi" when addressing them.

Communicating in memes slowly becomes thinking in memes. The cliches drown out any authenticity. Thoughts become blurry and hard to explain to others. That's a very high price for conflict avoidance.

From Markdown to ActionText

2023-04-05T08:00:00Z

I started this iteration of my blog because I grew dissatisfied with Medium. Like anyone migrating from any one platform to another I requested an export of my blog posts from them and got back a ZIP file containing HTML files.

The first thing that stood out was a lack of any media files - there were no images or gifs in the export. Opening the HTML files revealed that they contained the exact same HTML as was served by Medium.com. The source of the images and gifs pointed to Medium's servers. And the HTML itself had a lot of formatting and hidden elements specific to Medium and SEO.

I spent a whole day downloading images and cleaning up the files into something that could be imported somewhere else. I decided to convert my blog posts into Markdown - the pendulum had swung too far in the opposite direction. That way I could easily import them anywhere later. I fantasied about writing blog posts directly in my text editor with no distraction like headings, italics, bold, fonts - just my thoughts and text. And it was fun to develop a markdown rendering pipeline for the blog!

After a few months I gave up on the fantasy of writing in a text editor. While markdown is great, having a few simple options for editing text and being able to see the result right away is much more pleasing than knowing that # will eventually turn into a h1 element.

I started writing and editing articles in Notion, exporting them to markdown, adjusting them for my blog, and then uploading them here. This was unnecessarily complicated, but it gave me the ability to write wherever I went (I like to go for a walk, think, then sit down on a bench when I figure something out and write it down).

Now I had a different problem - editing an article after it was published. With this process I had to first update the version in Notion, export it, adjust it again, and upload it again. This was annoying when I started publishing more often and therefore catching less typos in editing.

This convoluted process eventual led me to stop publishing articles here. Then a few weeks ago I had the urge to write and publish something. But the thought of having to go through this publishing process made me put it off. This week I decided to correct this pendulum swing and migrate to ActionText to make things as simple as possible.

ActionText is a framework within Rails that enabled rich text editing and presentation. Adding it took no time at all, but migrating data to it took a whole day. There are many opinionated decisions in the framework with which I butted heads in the process.

The first "problem" I ran into were attachments. ActionText uses Rails' own ActiveStorage framework for uploading, processing and serving attachments, while the blog used Shrine (because I like Shrine). But the way I used Shrine was very reminiscent of how ActiveStorage works, so instead of reinventing the wheel further I decided to migrate over.

This wasn't an issue and was as simple as iterating through all Article::Attachment records, taking their attachment, calling to_io on it and then passing that to ActiveStorage::Blob.create_and_upload!

blob = ActiveStorage::Blob.create_and_upload!(
  io: attachment.attachment.to_io,
  filename: attachment.attachment_data&.dig("metadata", "filename"),
  content_type: attachment.attachment_data&.dig("metadata", "mime_type"),
  identify: false
)

The second "problem" I ran into was paragraph support. ActionText doesn't use paragraphs (p elements) for text, instead it puts all text into a single div element and breaks it into "paragraphs" using line breaks (br elements). This was a problem because I wanted to use Tailwind's Typography plugin and its prose class - which expects paragraphs as p elements. After fighting with the framework to make it work with p elements I ran across a GitHub issue where the reasoning behind using a div was explained which made me realized that it was far easier to write a bit of custom CSS than to make paragraphs work with ActionText in all browsers.

Without the explanation from the GH issue the decision to use divs seemed completely arbitrary and I wanted it to work with Tailwind. It would have been nice to have this explanation in the Readme.

The third problem was converting my previously rendered markdown into ActionText's HTML format (using a div and br elements) and embedding images into it. Though this turned out to be easier than I initially thought. A simple script and some Nokogiri magic and everything work like a charm.

Nokogiri.HTML(article.rendered_markdown).tap do |doc|
  # Remove the article's title
  doc.css("h1").first&.remove
  
  # Convert all paragraphs into a DIV with BRs
  content = doc.css("p").first
  content.name = "div"
  node = content
  while (node = node.next_sibling)
    content.inner_html += "<br>#{node.inner_html}"
    node.remove
  end

  # Convert images to attachments
  doc.css("img").each do |node|
    node.name = "action-text-attachment"
  end
  
  article.update!(content: doc.to_s)
end

The fourth problem was the lack of support for tables. But I decided to use Judo here and just embed the tables as content attachments, then take screenshots of them and embed them as images. This isn't ideal, but it works and it's what I used to do on Medium. I intend to add table support at some point, but currently only two articles use tables and spending a few days on this wasn't worth it.

The fifth "problem" was code block highlighting. At first I though this would be a significant undertaking that would require me to hook into ActionText, but in the end I used the Judo approach again and simply created a helper that changes the HTML before rendering.

module RichTextHelper
  def transform_rich_text(rich_text, transforms: nil)
    if transforms.blank?
      transforms = methods
        .select { |name| name.to_s.ends_with?("_rich_text_transform") }
        .map { |name| name.to_s.gsub(/_rich_text_transform$/, "") }
    end

    document = Nokogiri.HTML(rich_text.to_s)

    transforms.reduce(document) do |doc, transform|
      public_send("#{transform}_rich_text_transform", doc)
    end.to_s
  end

  # Makes code blocks look prettier / more readable
  def highlight_code_blocks_rich_text_transform(document)
    document.css("pre").each do |code_block|
      code = code_block.text
      formatter = Rouge::Formatters::HTML.new
      lexer = Rouge::Lexer.guess({ source: code })
      code_block.inner_html = formatter.format(lexer.lex(code))
      code_block["class"] = [code_block["class"], "highlight"].select(&:present?).join(" ")
    end

    document
  end
end

# then in some view
<%= transform_rich_text(@article.content) %>

Though my initial idea of hooking into ActionText led me to read it's source code, because of which I'm now much more familiar with the framework. So this detour wasn't a waste time.

The final "problem" were link previews. That's the feature I love the most in this iteration of the blog and wanted to have it in action text. At first I through that I'd have to extend the helper from before but then it dawned on me that I could inject everything needed for link previews through Stimulus. I liked that approach in this case because the previews are a "frontend" feature and they don't work if someone disabled JS or blocks it. But I'm still unsure it that was a good idea and I might move the code to setup previews into the helper.

All in all, I love using ActionText and I'm glad I fixed my over-correction. Having an editor baked into the blog makes it so much easier to edit (at home and on the go). And since the HTML format is well documented and simple I can still convert it to some other format if need be.

ActionText was easy to integrate, and with hindsight I can say that it was also easy to migrate to it.

P.S. while reading through ActionText's source and issues I ran across an interesting quote form the author of markdown:

I have no idea why there are now apps that use Markdown as their back end storage format but only show styled text without the Markdown source code visible. Hey World, for example, gets this right: they just do simple WYSIWYG editing where bold is bold, italic is italic, and links look like links and the linked URL is edited in a popup. If you want WYSIWYG, do WYSIWYG. If you want Markdown, show the Markdown. Trust me, it’s meant to be shown.

- John Gruber

Vanilla Rails view components with partials

2023-01-24T14:00:00Z

Many projects I work on have some kind of view component that is repeated multiple times in the same view, or is present in multiple different views. These view components can be anything that has a specific styling, JavaScript specific attributes (like Stimulus controllers), rendering logic, and HTML element structure that always has to be the same for the component to render properly - like cards, containers, content boxes, modals, lists, tables, and so on.

There are three ways that I have seen people work with view components in Rails - by copy-pasting them around, by using one of the view component gems, and partials.

Copy-pasting, even when using CSS conventions like BEM, has the downside of being laborious to update should the component ever change. And I find the gems to be unnecessary since vanilla Rails already has all the features to render and manage view components through partials.

This is an example of three different view components combined together to render a table-like list and a map within a content box with a title using just partials.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Code and browser view of multiple vanilla Rails view components composed together to show location details and a map side-by-side within a titled content box.

The component method might seem like there is a lot of magic under the hood, but there really isn’t. I added this method through a view helper and it does just two things - it calls render with "components/#{component_name}" so that I don’t have to write the same incantation all over the place, and I’ll explain the the second purpose later.

For now, this is what the helper looks like

module ComponentHelper
  def component(path, options = {}, &block)
    full_path = Pathname.new("components") / path
    render(full_path.to_s, options)
  end
end

So component("map", longitude: 15.9765701, latitude: 45.8130054) is the same thing as render("components/map", longitude: 15.9765701, latitude: 45.8130054).

Partials allow you to render a snippet of a view wherever you like. For example, instead of having to repeat all the HTML elements and Ruby code that renders two tables:

<!-- app/views/matches/show.html.erb -->
<h2>Winners</h2>
<table>
  <thead>
    <tr>
      <th>Rank</th>
      <th>Player</th>
      <th>Score</th>
    </tr>
  </thead>
  <tbody>
    <%= @match.winners.each do |player| %>
      <tr>
        <td><%= player.rank %></td>
        <td><%= player.name %></td>
        <td><%= player.score %></td>
      </tr>
    <% end %>
  </tbody>
</table>

<h2>Losers</h2>
<table>
  <thead>
    <tr>
      <th>Rank</th>
      <th>Player</th>
      <th>Score</th>
    </tr>
  </thead>
  <tbody>
    <%= @match.losers.each do |player| %>
      <tr>
        <td><%= player.rank %></td>
        <td><%= player.name %></td>
        <td><%= player.score %></td>
      </tr>
    <% end %>
  </tbody>
</table>

With partials you can just render the partial once for each table and pass it a local with the content it should render:

<!-- app/views/matches/show.html.erb -->
<h2>Winners</h2>
<!-- we are passing `@match.winners` as the local `players` to the partial -->
<%= render "players_table", players: @match.winners %>

<h2>Losers</h2>
<%= render "players_table", players: @match.losers %>

<!-- app/views/matches/_players_table.html.erb -->
<table>
  <thead>
    <tr>
      <th>Rank</th>
      <th>Player</th>
      <th>Score</th>
    </tr>
  </thead>
  <tbody>
    <!-- the `players` local shows up in this partial as the `players` variable -->
    <%= render collection: players, partial: "matches/player" %>
    <!-- the above ^ is a shorthand for:
    ```
      <% players.each do |player| %>
        <%= render "matches/player", player: player %>
      <% end %>
    ```
    -->
  </tbody>
</table>

<!-- app/views/matches/_player.html.erb -->
<tr>
  <td><%= player.rank %></td>
  <td><%= player.name %></td>
  <td><%= player.score %></td>
</tr>

Using partials like this works for components that have predetermined content (like tables, maps and lists), but it’s a pain for components that can have anything inside them (like containers, content boxes and modals).

You could extract the content of each component into its own partial and pass the the name of the partial you want to render.

<!-- app/views/teams/show.html.erb -->
<%= render "content_box", content_partial: "weekly_metrics", content_options: { team: @team } %>
<%= render "content_box", content_partial: "monthly_metrics", content_options: { team: @team } %>

<!-- app/views/teams/_content_box.html.erb -->
<section>
  <%= render content_partial, content_options %>
</section>

<!-- app/views/teams/_weekly_metrics.html.erb -->
<p>Matches won: <%= team.matches.where(created_at: (1.week.ago...)).won.count %></p>
<p>Matches lost: <%= team.matches.where(created_at: (1.week.ago...)).lost.count %></p>
<p>Upcoming matches: <%= team.matches.where(created_at: (Time.current...)).count %></p>

<!-- app/views/teams/_monthly_metrics.html.erb -->
<p>Matches won: <%= team.matches.where(created_at: (1.month.ago...)).won.count %></p>
<p>Matches lost: <%= team.matches.where(created_at: (1.month.ago...)).lost.count %></p>

This can be avoided by passing a block to the partial and yielding to it.

<!-- app/views/teams/show.html.erb -->
<%= render "content_box" do %>
  <p>Matches won: <%= @team.matches.where(created_at: (1.week.ago...)).won.count %></p>
  <p>Matches lost: <%= @team.matches.where(created_at: (1.week.ago...)).lost.count %></p>
  <p>Upcoming matches: <%= @team.matches.where(created_at: (Time.current...)).count %></p>
<% end %>

<%= render "content_box" do %>
  <p>Matches won: <%= @team.matches.where(created_at: (1.month.ago...)).won.count %></p>
  <p>Matches lost: <%= @team.matches.where(created_at: (1.month.ago...)).lost.count %></p>
<% end %>

<!-- app/views/teams/_content_box.html.erb -->
<section>
  <%= yield %>
  <!-- `yield` will be replaced by whatever is passed in the block -->
</section>

That’s how component("content_box", title: "Location") do works.

<!-- app/views/properties/show.html.erb -->
<% component("content_box", title: "Location") do %>
  <h1>Hello World!</h1>
<% end %>

<!-- app/views/components/_content_box.html.erb -->
<section class="rounded shadow bg-white">
  <!-- `local_assigns` is a hash that contains all the locals passed to a partial -->
  <!-- we can check if a local is present and access it's value through it -->
  <% if local_assigns.key?(:title) %>
    <h4 class="text-gray-900 text-lg">
      <%= local_assigns[:title] %>
      <!-- I could have used `title` instead of `local_assigns[:title]` -->
    </h4>
  <% end %>
  <div>
    <%= yield %>
  </div>
</section>

This brings me back to the second purpose of the component method - fixing localization.

If I would use full localization keys for everything then there wouldn’t be a problem with just using render

<!-- app/views/properties/show.html.erb -->
<% render("components/content_box", title: t("properties.show.location")) do %>
  <h1><%= t("properties.show.hello_world") %></h1>
<% end %>

<!-- config/locales/en.yml -->
en:
  properties:
    show:
      location: "Location"
      hello_world: "Hello World!"

<!-- rendered HTML -->
<section class="rounded shadow bg-white">
  <h4 class="text-gray-900 text-lg">Location</h4>
  <div>
    <h1>Hello World!</h1>
  </div>
</section>

But I’m lazy and like to use relative localization keys everywhere, which causes a problem.

<!-- app/views/properties/show.html.erb -->
<% render("components/content_box", title: t(".location")) do %>
  <h1><%= t(".hello_world") %></h1>
<% end %>

<!-- config/locales/en.yml -->
en:
  properties:
    show:
      location: "Location"
      hello_world: "Hello World!"

<!-- rendered HTML -->
<section class="rounded shadow bg-white">
  <h4 class="text-gray-900 text-lg">Location</h4>
  <div>
    <h1>
      <!-- This is what Rails render when a translation is missing -->
      <!-- Notice that it looked for the translation in `en.components.content_box.hello_world` -->
      <!-- instead of in `en.properties.show.hello_world` -->
      <span class="translation_missing" title="translation missing: en.components.content_box.hello_world">
        Hello World
      </span>
    </h1>
  </div>
</section>

With relative localization keys, Rails prefixed the key with the path of the component instead of the parent.

When a view yields the passed block is rendered in the context of that view, not in the context of the view where the block was created. Since the partial yields the relative localization key prefix is components.content_box.

To solve this I can either use full localization keys within components, which is annoying; or I can add the translation to the component’s localization, which means that I would have to think about possible key collisions when using relative localization keys within components.

Thankfully there is a way to capture a part of a view within the context of the current view and store it as a variable. With the capture method I can yield the block passed to the component in the context of the current view, store the result to a variable, and pass that variable in a block to render.

Here is the full ComponentHelper code with the relative locale keys fix.

module ComponentHelper
  def component(path, locals = {}, &block)
    full_path = Pathname.new("components") / path

    if block
      # Render the passed block within the current context and
      # store it in the `content` variable
      content = capture do
        block.call
      end

      # Call render but pass it a new block that yields just
      # the contents of the `content` variable
      render(full_path.to_s, locals) { content }
    else
      render(full_path.to_s, locals)
    end
  end
end

And that’s all there is to it - render, local_assigns, yield and capture.

COVID hangover

2023-01-21T16:00:00Z

With a lot of tech companies laying people off in the last month I got to see how insane the firing practices in the US are and how greed rules supreme in the tech industry. But this is only a harbinger of what’s to come.

Through friends and from what I’ve seen first hand, the standard procedure to fire someone in the US is to invite them to a meeting, tell them that they have been let go, and immediately suspend all their company accounts. Or email them to tell them they have been let go (effective immediately), or worse yet tell them they have been let go via a tweet.

There is zero humanity in laying people off this way. These people dedicated months or years of their lives building these companies and then one morning they get kicked out the door with a 60 day severance package - which is the legal minimum - and aren’t even given the decency to say good bye to their former coworkers.

These layoffs weren’t triggered by financial losses but by speculations of a looming recession. Companies are afraid that money won’t be cheap and profits will be low in the coming years and they are cutting their losses while they still can in an effort to keep investors and shareholders happy. In other words, they are cutting headcount to cut expenses in an effort to prop up profits. Which is extremely greedy after a year of record profits and shows no regard for the humans that made this growth possible in the first place.

I believe that these layoffs are not an indication of the tech bubble bursting, but are just the hangover after the COVID lock-downs when tech seemed like the only unaffected industry. This caused every company in the industry to over-hire and expand in the pursue of even higher profits. But expecting 20-30% returns year-after-year and infinite growth is unrealistic - that can only be a bubble. After a couple of good years a bad one is bound to happen. And if there is going to be a recession then profits should be low - don’t turn a bad day for you into a tragedy for others. Sit it out and don’t be greedy.

If the industry weathers this recession relatively unscathed people will see tech as an even better investment vehicle which will in turn lead to more investment, more hype, and the bubble will inflate even more. This seems to be the outcome that most big companies are banking on since even after the layoffs most of them still have more employees than they had two years ago, indicating that they want to use the extra manpower to expand.

I don’t know when or how this bubble will burst, but it will be one hell of a pop when it does.

Elden Ring

2022-12-04T20:00:00Z

This week I finished Elden Ring, and the moment I put the controller down I felt a void as if a good friend moved to another country and I wouldn’t see them for a long time. This was the best game I have played in years.

But why? On the surface it seems just like any other open-world ARPG, yet Elden Ring is bigger than the sum of its parts. This game is a master class in the omakase design approach and that’s what makes it so amazing.

The game makes two key decisions for you - the difficulty level and the story exposition.

There is no way to change the difficulty in this game. You find a boss difficult? Tough luck, you will have to get better. Everybody has to face the same challenging fights, enemies, quest and puzzles. This was the first Souls game by From Software I have played and this design decision felt hostile to me, sometimes I just want to relax with a game and enjoy the story.

But after I beat the first boss I felt a sense of accomplishment and then it clicked - the difficulty is what makes each win feel so rewarding. The combat is punishing, but it’s also fair. It’s like a dance to which you have to figure out the steps or get trampled (or get so overpowered that you can steamroll through it). There are a few bullshit attacks that are hard to dodge, but everything else is clearly telegraphed - if you make a mistake you know it was on you. The dance-like nature of the combat is also why getting attacked by more than one enemy at a time is infinitely more difficult - you have to keep track of two dance partners.

But the thing that makes the combat of Elden Ring so great is its open world. When I got stuck, bored or annoyed I could always just go somewhere else and have fun. There is no better example of this than the first mini-boss. I exited the tutorial level to find myself in a lush grassy field with a giant golden knight in the distance. As the naive spring chicken that I was, I walked up to the shiny knight only to see a large “YOU DIED” message a second later. Then I tried to fight the knight, and no matter what I did the knight would demolish me before I got him to half-health. At that moment I understood that the game was trying to teach me that if I’m not having fun that I should look elsewhere. Nothing was forcing me to fight that kings, I could just go around him and come back later.

There are a few traps in the game that take away your ability to travel and only then does it become obvious how crucial the open-world is to the combat experience of this game. I got eaten by the Abductor Virgin (Iron Maiden) in Raya Lucaria Academy and got teleported to a lava lake in Volcano Manor, under-geared and under-leveled, only to spend and hour in agony before I gave up and looked up how to cheese an escape.

The game does use its combat difficulty as a level-gate to steer you round the world. But because of this the world always feels challenging and keeps you on your feet. I was decimating archers and knight, fighting my way to a demigod boss only to emerge in another region of the world after the fight where a stray dog and a guy with a torch could mess my day up.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Strong Doge meme. The strong one tanks hits by demigods, the weaker one is bullied by two dogs and a guy with a torch

Exploration is not only important to get better gear and level up, but also for the story. The game doesn’t hold your hand when it comes to the main story, it expects that you will go out of your way to figure it out. This also seemed hostile at first.

There are a few expositions during key quest moments and demigod fights, but that’s not enough to figure out what is going on. The story through the cutscenes alone is a jumbled mess - you want to become Elden Lord, because that’s what one does, and there are demigods you have to kill to get shards of the Elden Ring, which shattered in the past, and only with those shards will you be able to become Elden Lord. What is this? Pokemon fan fiction? How many gym trainers do I have to beat to become a Pokemon master?

The details of the story are scattered in side-quests and item descriptions which can only be found by exploring. I was half-way through the game when I discovered that item descriptions contain the lore of the world. The other thing I discovered half-way through the game is that finishing side quests unlocks alternate endings. This made exploration more enjoyable and rewarding compared to other open-world ARPGs, I felt like Indiana Jones recovering parts of the Ark of the Covenant.

These two decisions - a single difficulty level for all and a vague story - complement each other in a way that makes the world feel like a sandbox where I can go explore and adventure. It doesn’t get old and it’s not a gimmick like in other ARPGs. It’s so good that this is the first game where, after beating it, I immediately started a second play-through to experience what I have missed.

This is only possible because of omakase. The team at From Software made a tailored experience, an experience that takes existing elements of ARPGs, sacrifices some quality-of-life features, but arranges what’s left in such a way that they become better than in all other ARPGs.

Cutting through the noise

2022-11-27T21:00:00Z

I use writing as a tool to organize my thoughts, but lately I have started to question if that’s the right order to do things.

Writing is thinking. Text is just someone’s thoughts on paper. It’s obvious that thinking has to come first, but that’s not always the case.

When I sit down to write something I often just have a notion about what I want to write, and only through the process of writing do I form a full thought. This process can take days to produce an article. But as I started writing more consistently, the time required to write an article went down to a few hours.

This week I heard that DHH writes articles in 15min and I was taken aback. I enjoy his articles, he has a way of explaining things in a clear and forward way - something I am trying to improve, and which I envy.

The only way I see that this is possible is by sitting down with a fully formed thought before writing a single word. He’s able to cut through the noise and distill a thought without the aid of writing.

Have I been putting the carriage before the horse all this time?

Well, yes and no. To write that way I have to trust my self more - something I see I have been avoiding by using writing as a crutch to verify my thoughts, double and triple check them. I’m just avoiding criticism, but that means that my written thoughts are a watered down vanilla version of the real thing. I have to become bolder in expressing my opinion. Then again, there is nothing wrong with using writing as a tool from time to time.

Email

2022-11-20T21:00:00Z

For me, instant messaging is overwhelming. Keeping up with Slack or Discord messages often feels like drinking from a fire hose of information. And some features of these apps bring out the worst in people. I wish these apps were more like email, because email got a lot of things right.

My biggest complaint with instant messaging is how distracting it is. Notifications pop up to grab your attention for every message, most of which aren’t intended for me.

This is a necessary byproduct of organizing communication into channels. On the surface this is a great idea to focus the discussion on a topic, but in practice it heavily hinges on the netiquette of the participants.

Like every group chat ever, a channel will spawn off-topic conversations that will ping everyone in the channel.

Eventually, somebody will start discussing sandwiches in a channel for handling outages. Or someone will start discussing a baseball game in the general channel. Both of these are real examples.

I feel like the only real option in such cases is to mute or leave the channel, but that’s not always an option. I can’t leave the outage channel, and I can’t mute it. So what now?

Topics (or sub-chats) are a great way to combat this. But again, they hinge on the netiquette of the participants. If nobody uses threads they are useless. If too many are created, they become hard to follow.

Then there is the instant part of instant messaging that entices some people to write down their stream of consciousness for others to interpret, instead of taking the time to organize their thoughts into a sentence.

You

Know

The

Type

People

Who

Send

Messages

This

Just

They

Can

Write

Something

Instantly

Then there is the other kind of instant in instant messaging - the instant response - to which some people feel entitlement.

I’ve had people ping me with “I saw you replied in the channel 15 min ago, but not to me”. Or people ping me with a question and 3 min later sending another one saying that my lack of response will be considered a no. And I had people ping me with “Are you there?” every 5 min.

I don’t know why instant messaging brings out the worst in people. But I know that a lot of people choose to always appear offline to avoid these problems.

Which brings me to email. It never had an online status to begin with, every conversation is a thread, there is no expectation of an instant response, and people usually write full sentence responses.

Some of these “features” were technological or practical limitations of the time, and some evolved out of the way people used email.

But email evolved into a great, distraction free, way to communicate by using these friction points to its advantage.

It has its problems, like not having a mechanism to leave a thread, or having some way to track branching threads. But apps like Hey have shown that it’s possible to resolve these problems. And showed that the federated design of email allows anyone to innovate and improve this 50-year-old protocol.

There are so many wonderful solutions on top of email out there. Apps like Mailbrew - which combines RSS, Reddit, Twitter even weather forecasts into a single daily email.

Anyone can send you an email. Which is a blessing and a curse thanks to spammers. But I’ve had wonderful conversations over the years that have cold-mailed me after stumbling upon one of my projects or articles. And, again, apps like Hey add ways to screen incoming emails before they end up in your inbox.

I recently tried Twist and found that by mimicking email it became a better instant messenger - if you can call it that. It only shows that there is still a lot of room left to improve instant messengers. At least the work-focused ones like Slack.

Email is still the best communication method for me.

Tech debt for non-techies

2022-11-13T12:00:00Z

Tech dept doesn’t occur by accident. You go into it intentionally - it is a loan like any other, but instead of loaning money you loan time.

When you want to “save time” by implementing a hack (something that goes against the grain of the software’s architecture) you are making an intentional bad decision - you don’t have the time to come up with a better architecture so you defer the problem to future you (or, with any luck, someone else). Essentially, you are borrowing time you don’t have now from the future.

Like with a regular loan, you have to pay that time back. But without a bank to remind you that your installment is due, it’s easy to forget about this loan. Forgetting the loan sometimes works out, and other times it doesn’t.

Tech debt won’t cause you problems if everybody on the team understands what it is and what its trade-offs are - how time is traded for functionality, and how to manage this kind of loan. Technical team members are painfully aware of tech debt because they are the ones that have to manage it and pay it back if necessary, but non-technical team members are usually oblivious to it.

And I don’t blame them, it’s a concept that can be hard to fully grasp. Sure, most people will think they understand it from reading the loan analogy from before. But they won’t be able to see it even when it’s happening in front of their eyes.

But I found a way to illustrate the problems and processes that cause tech debt through Factorio.

Montage that demonstrates the basics of Factorio - gathering resources, fighting aliens, building factories and finally launching the rocket

Factorio is a game in which you play as an engineer stranded on an alien planet. Surrounded by nothing but trees, rocks, minerals, and a hoard of aggressive aliens. Your goal is to make shelter, fight or co-exist with the aliens, and get off the planet by building and launching a rocket.

The problem is obvious - how do you make a rocket and rocket fuel from sticks and stones? The answer is simple - step by step.

In Factorio, more complex devices can be made out of simpler ones or out of raw materials like rock, iron, copper, coal, oil and uranium.

In the beginning you can only gather and build things. So you do exactly that. You gather rocks to make a furnace, then you gather Iron ore to smelt it into plates.

Gathering of 5 stone to craft a furnace, and then smelting an iron plate in it by feeding it iron ore and coal

Iron plates enable you to build drills and conveyor belts. Now you don’t have to dig and walk so much anymore.

Soon you can build robotic arms that can pick up things from the conveyor belts and feed them to other machines. Now you can automate whole processes like smelting iron and copper.

A basic automated coal and iron mining setup

And that is the basic loop of the game - gather, build, automate. But it gets complicated. Over time you will build a rocket building factory. There will be machines making iron, turning it into electronics, building other machines that are parts of other machines that are part of your rocket, and all of them will be connected with conveyor belts and robotic arms.

Running through one part of a bigger factory

Building such a factory is no easy task if you can’t manage tech debt. And you will run into tech debt pretty quickly.

Your first encounter with tech debt will be through conveyor belts. In the beginning you will place parts of your factory wherever they are easiest to fit. Probably close to the resources required to build the parts

You will build conveyors close to where you smelt iron. That’s only logical since conveyor belts are expensive and you just spent a lot of resources on an assembler to make the gears and conveyors.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A furnace and two assemblers connected with conveyors, building new conveyors

This kind of thinking and behavior is known as “greedy behavior” in software. Greedy behavior is usually short-sighted, highly rewarding short-term, but it can be much worse than other approaches long-term.

And that is exactly what will happen here. If you now want to create an assembler that builds robotic arms, you will run into an issue - the gears that you need to make them are in a box right between the two assemblers and are fully surrounded by conveyor belts. There is no space left to add a robotic arm to extract them from the box and put them on another conveyor.

Now you have two choices - tear everything down and rebuild it better, or squeeze in a few conveyors.

Rebuilding is the better choice now that you know that you want to automate the assembly of robotic arms. But some people will choose to squeeze in conveyors because of the sunk cost fallacy - they can’t let go of the time and effort they have already invested - or because they don’t have the time to rebuild the factory - in which case they are entering tech debt by intentionally making a bad decision to save some time now.

Over time, squeezing in more and more conveyor belts will result in conveyor belt spaghetti. The factory will get so densely covered with conveyor belts that it will become difficult to keep track of which resources are going where. And even worse, it will become hard to add new conveyors and grow the factory - you will have to duplicate parts of your factory to be able to cart the resources to another part of the factory.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

What it looks like when you squeeze in conveyors and duplicate parts of the factory. There are two assemblers making gears, and two furnaces making iron plates. And everything is covered in conveyor belts.

Short-term, squeezing in conveyors will solve your problem. Long-term, it will cause you bigger problems. But there is a third solution - squeeze in the conveyor belts today, and reorganize the factory tomorrow. That is, enter tech debt today and pay it tomorrow.

To long term players this is obvious. They would build the factory with expansion in mind from the start. But to new players this isn’t obvious at all.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The same factory as before, but built with expansion in mind. Production of raw resources and parts is centralized and then the resources are carted around the factory on conveyor belts. There is plenty room to add more machines or expand existing ones.

That’s why developers complain and moan about hacks and tech debt, while managers usually struggle to understand the problem. Developers know that, eventually, the code they write today will become the foundation of a new feature tomorrow and they want to prepare for it. While managers are concerned with delivering something today.

That is also why tech debt rarely gets addressed in some organizations. When a manager - the person deciding what the developer or designer will build - has to choose between working on a feature or maintenance they will always choose the feature. It is a feature today, they understand that. They don’t understand that maintenance will be a feature tomorrow.

In their eyes one is fact, the other is a hypothetical. While in the builder’s eyes both are fact.

Fight perfection

2022-11-04T12:00:00Z

Seeking perfection is a fool’s errand.

Perfection is a fading spark that can only be achieved for oneself - never when working with others - and as soon as it’s gone the search for the new perfect begins.

By constantly searching for perfection you end up living in the fantastic future - in the “what could be” - instead of living in the here and now - in reality. It’s an escape from facing what you should do now.

Perfection is the siren song that will steer you into a vortex.

So fight it, seek reality, seek better, seek good enough.

In my childhood I was encouraged to seek perfection - to get straight A grades, to write the best assignments, to know every subject matter inside out at any moment - and that thought me that one can only achieve perfection for oneself.

When I wrote an assignment that I though was perfect - one that I was happy with in every regard - most often I got a B for it. Why? Because it wasn’t perfect for my teacher. Even when it was grammatically perfect, my teacher thought that I could improve flow, or phrasing, or something else. This theme continued in high school, college and at work. Every time I was satisfied - though something was perfect - at least one other person didn’t. There was always something to add, adjust or remove to transform my perfect into their perfect. And sometimes I came to adopt their perfect as my perfect, but sometimes I didn’t.

This taught me that perfection is in the eye of the beholder. Every person will have a different opinion about what is perfect. Sometimes people can agree that something is perfect, other times they can’t. Even when a collective perfect is achieved it doesn’t last because every new person that comes along will have a new notion about what could be improved.

Searching for perfection among many people can be an impossible task. So don’t do it. Fight this notion. Don’t seek a consensus about what is perfect, do what is good enough for everybody, do that which is better than what you have now.

Later on, when I started working, I took part in meetings where we discussed that the perfect solution to a project would be.

This usually resulted in some sort of a shared notion of a perfect solution that was presented to the client, but no clear way how to get from where we are now to that perfect solution.

Since we didn’t want to inconvenience or disappoint the client we clang to that notion of perfect, this lead to many inconveniences and disappointments for us and the client.

Because we had such a rigid goal we never reconsidered if what we were doing made sense. So every now and then we ran into an insurmountable problem or built out nonsensical features that would eventually make sense after something else was built. We were going straight to perfect, regardless of what it took.

By thinking about the future we forgot about the road that we have to follow to get there. So don’t focus on the future where everything is perfect, focus on reality, focus on the here and now.

We could have presented the perfect solution as an eventual goal that may or may not come true, and commit to build out a project that brings us closer to it, and then commit to another project, and another, and another.

That way there isn’t a commitment to a grand goal, so both we and the client are free to change course at any time to avoid problems as they arise. We might not end up at perfect that way, but we would certainly get close to it. And maybe the definition of perfect would change along the way, to one which we could reach.

The worst kinds of projects at work were ones that never left the drawing board. On these kinds of projects the client was so obsessed about figuring out the perfect solution that they ran out of money before the product was even close to being done.

They spent weeks in meetings seeking the perfect solution, and always came up with something better than what they had before - they found a new perfect and moved the goal post there. That is the nature of being human - we always find a way to make things better than they are now, we can always dream bigger.

But you won’t ever make anything better if all you ever do is think about how you can make something better. Don’t obsess about doing something perfectly, just do it better, or good enough.

Seeking perfection can give you a vision of the future, but remember that it’s only a vision. If you follow it blindly it can lead you down a twisty and thorny road to nowhere. Use it as a compass to make things better than they are, or else fight its siren song tooth and nail for it can ruin everything it touches.

P.S. This topic has been on my mind for a long time. I tried to write it up a couple of times before but always fell short of the goal I set for my self - I didn’t notice that I was chasing perfection. And that’s something that I have struggled with as an adult. It’s only a few years back that I noticed how corrosive this chase for perfection can be. And it’s only a few weeks back that I learned that Tolkien shared the same notion. He said that Sauron started out as a perfectionist which desired order, and in his effort to make things perfect he became the embodiment of evil.

What is a professional tool anyway?

2022-10-29T12:00:00Z

One weekend in 2017 I was using my Macbook Pro when all of a sudden the screen turned black and it started spewing white smoke like it had just elected the new Pope.

Luckily, it didn’t catch fire. Later that day I opened up the laptop to see what happened.

To my surprise it wasn’t the battery, but it was the hard drive - the only replaceable component in a 2013 Macbook Pro.

I ordered a replacement, and a week later the laptop was working again. I even upgraded it’s storage from 512GB to 1TB.

But this problem got me thinking - was the MacBook Pro really a professional tool?

I always think of a DSLR camera when I think of a professional tool.

Its parts are mostly replaceable, and the parts that aren’t have redundancies.

You can switch out lenses, flashes, microphones. You can look through the view finder or the LCD to see what you are shooting. All settings can be changed using the buttons on the device or through the on-screen menu. The lenses, flashes and microphones are interoperable with newer and older camera bodies.

All of these functions give the tool a long service life and make it reliable and dependable. Therefore a higher price is justified and still a good investment for a professional user.

Yet there still exist cheaper point-and-shoots which offer similar image quality but without the modularity for less money.

As a developer, my most used tool is a computer. And for a long time my go to computer was a laptop.

In the laptop space modularity and repairability have been dead and buried long ago.

The ability to add more memory or space to your machine, to repair parts that break, has been sacrificed in the name of “progress” and “design”.

The question is the progress of what? And what design?

The hardware was never smaller and more reliable than today. Machines that were thin, slick, and modular yesteryear are just thin and slick today. The original MacBook Air was a few millimetres thicker than today’s and had a replaceable battery.

The only progress I see here is the progress of companies maximizing profits.

The simple truth is that it’s cheaper to make hardware this way. And it forces people to buy the more expensive model up front out of fear that they might outgrow the cheaper one.

It used to be normal to buy a machine with moderate specs and upgrade it later if you wanted more memory or space. Hell, I upgraded my old 2007 iMac with more memory and an SSD back in 2009 - these machines used to be user-upgradeable.

Today it’s becoming harder and harder to find a computer that allows you to fix or upgrade anything.

Some companies even go out of their way to make repairs and upgrades difficult. Probably to up-sell you a more expensive model or to sell you a new device when the first one fails due to a banal problem.

I wouldn’t call these devices professional.

Sure, they have a lot of features, and they have cutting edge specs. But just like with the point-and-shoot camera, you get what you get and once something breaks the whole machine becomes a paperweight.

In the last few years a few companies that offer upgradeable and repairable laptops did appear, and I wish them the best of luck. We could use more companies like these, and more devices that aren’t designed to fail, that are repairable and upgradable.

A week in Helsinki

2022-10-23T12:00:00Z

Last Wednesday my girlfriend and I woke up at 4 AM and headed to the Zagreb Airport to catch a flight to Helsinki.

I haven’t been in Helsinki since Junction 2016, but with EuRuKo 2022 (the European Ruby Konference - this is not a typo) being there this was the prefect opportunity to visit again.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Above the clouds somewhere between Zagreb and Munich

One hour after boarding our flight in Zagreb we were in Munich where we had a 40min layover. I though we would have to run through the airport to catch the next flight, but to my surprise there was a person with a “Helsinki” sign waiting for us just as we stepped off the airplane.

They took us with a van from one part of the airport to a small border crossing to check our passports. This confused me at first, but it occurred to me later that it was due to the Schengen area rules. Croatia isn’t a member yet so we have to go through passport control.

After the passport check the man took out a key and started unlocking a door that until that point I didn’t even notice existed. This was the first in a series of doors that he opened. Each lead to a hallway that connected to a staircase or elevator and then to another locked door that eventually spit us out right next to our gate. It felt like we were going through a pocket dimension or a worm hole.

A trip that would take about 40 to 50 min of running through the airport was cut short to just 30 min of inter-dimensional airport travel.

The next flight wasn’t as exciting as the previous one. We landed in Helsinki some two hours later and went to the baggage clam to find that our luggage went on trip of it’s own, but neither we nor Lufthansa knew where to. Thanks Lufthansa.

A taxi ride later we were in our hotel. With nothing to unpack we met up with a few friends and went for a walk towards the city center.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Statues at the Helsinki railway station

The walk turned into lunch at the Zetro - a tractor (farming machinery) themed restaurant. Inside which everything is dimly lit, there are tractor tables, and all kinds of farming equipment. When you sit at a table the waiters light a few candles without which it would be hard to read the menu.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A tractor turned into a dining table in a room dimly lit by red light

Everything on the menu has a cheeky name and a funny description. I ordered the “Momma’s boy meet balls”, while my girlfriend ordered a steak that is allegedly cooked on the bonnet of a Zetro tractor. Both meals were amazing as was the service.

The following day was the first day of EuRuKo.

The organizers really outdid themselves. The venue was the Paasitorni, a beautiful and ornate workers’ house from the early 1900s. Registration was quick. There weren’t queues for food, water or hot beverages - what more can you ask for?

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Inside of the ornate room where the conference was held

All talks are already available for everyone to watch.

My favorites are How music works, using Ruby by Thijs Cadier, Implementing Object Shapes in CRuby by Jemma Issroff, The Technical and Organizational Infrastructure of the Ruby Community by Adarsh Pandit and Looking Into Peephole Optimizations by Maple Ong.

But the magic on conferences is meeting people. I had many wonderful conversations with various people about how they run their applications (thanks Shopify team), and about the internals of the Ruby virtual machine. Exciting times are ahead for Ruby with version 3.2 and beyond.

After the conference we went out for dinner and found a newly opened Korean barbecue named Oppa. This was our first time in a Korean BBQ and it was an experience. We had so much fun, and the food was so delicious, that we came back the next evening with a whole group of friends.

A few pieces of meat on a Korean barbecue plate

On Saturday we took a ferry to Suomenlinna, an island some 10 min away from the city center.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A view of Helsinki from the harbor

We saw a few small islands with a single house along the way. It must be wonderful to live in such a beautiful, secluded, wooden house, only 10 min away from anything you might need or want to visit.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A beautiful, red painted, wooden house on a small island right outside the Helsinki harbor

As we stepped off the ferry we saw a sign on a pinkish-red building that said the island is an UNESCO World Heritage site. It houses the naval academy, a former ship yard, a former prison and a fortress that used to protect the city.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A pinkish-red tower of a building with a clock in it

The first thing that stood out on this gloomy autumn morning was the bright yellow naval academy. So we went to it first and then returned to the harbor.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Two bright-yellow buildings of the naval academy

The harbor and coastal regions of the island are littered with small wooden homes with a few brick and mortar buildings thrown in here and there. Many of the houses are private, some are hostels, museums, cafes and shops.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A brick and mortar building from the center of the island.

We visited the military history museum which is split across two buildings (keep the ticket when you enter the first building so that you can enter the next).

One building is full of military uniforms and equipment all the way from the early 1900s to WWII. While the second building is full of modern military equipment.

In the second building you can wear some of the historic uniforms and sit in the cockpit of a jet fighter simulator. Needles to say we had a fashion show.

And there is even a submarine!

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A submarine in a dry dock

After the museum we went for a walk around the island - which is full of hills, most of which have been hollowed out to make bunkers, caches and armories.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The inside of a hollowed out hill

To me the landscape was magical. I never imagined that an island could be covered with beech and oak trees on rolling green hills that end with granite rock cliffs in the dark blue sea. My mental image of an island or the seaside is Dalmatia - steep mountainous terrain with sharp white rocks covered in pine and olive forests ending in a teal-azure sea with pebbled beaches.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A cliff covered with in green moss, grass, and trees, looking over the sea

We went past the huge defensive walls, the canons looking over the harbor, through the fortress, all the way to the king’s gate and then back to the harbor building - where we sat down in a brewery to wait for the ferry back.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A green bench in front of a large defensive wall made of many gray, dark red and dark blue granite rocks topped off with a brick and mortar gable

Back in Helsinki we went to the cathedral and then went back to the hotel to pack our bags for the trip back home.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The Helsinki Cathedral - a tall, white palace with green domed roofs decorated with golden stars

Helsinki is a clean, beautiful, peaceful city, filled with beautiful buildings, wonderful parks, wide streets and cozy places. Given the opportunity I would move there in a heartbeat.

Everyone we met was kind, helpful and trusting.

I was surprised how quiet everybody is. Compared to my hometown everyone seems to be whispering all the time.

Coming from a country where people are quite distrusting and a bit paranoid, the trusting nature of the people in Helsinki felt odd. There was nobody to check our ferry tickets, no badge or person to check if we were conference attendees, no check to see if we were hotel guests when we went for breakfast. Croats often joke that we are the world’s best nation in circumventing rules and finding loopholes. In contrast, everyone in Helsinki seems to trusts others to be decent human beings.

I can’t wait to visit the home of the Moomins again.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A white notebook with a gold embossed Moomin

How an index made rendering slow

2022-10-10T12:00:00Z

I noticed that a view I was working on was rendering much slower than I would expect it to. The view showed a list of events, together with the person that generated the event and the device that the event belongs to. It took nearly half a second to render 25 events, while other similar pages render in a couple of milliseconds.

At first I thought that I had an N+1 - that each rendered event went to fetch its person and device from the database, generating 2 database queries per rendered row.

But in the controller the associations were clearly preloaded, and even marked for strict loading. Not only was an N+1 problem avoided, but the system would raise an error if one would occur.

def index
  @events = Device::EventLog.all
                            .preload(:user, :device)
                            .strict_loading
                            .page(params[:page])
                            .per(25)
end

So why did the view render so slow?

I went to look at the development server logs - which, in Rails, show a lot of useful information for debugging problems like these - and I saw this:

Started GET "/events" for 172.23.0.1 at 2022-10-10 08:57:51 +0000
Processing by EventsController#index as HTML
  Parameters: {}

...

Device::EventLog Load (2.8ms)  SELECT `device_event_logs`.* FROM `device_event_logs` LIMIT 25 OFFSET 0
↳ /app/views/events/index.html.erb:10:in `_app_views_events_index_html_erb___1845829941033837665_155040'

...

User Load (79.75ms)  SELECT `users`.* FROM `users` WHERE `users`.`deleted_at` IS NULL AND `users`.`id` IN (97212, 2526, ...)
↳ /app/views/events/index.html.erb:10:in `_app_views_events_index_html_erb___1845829941033837665_155040'

...

Device Load (79.95ms)  SELECT `devices`.* FROM `devices` WHERE `devices`.`deleted_at` IS NULL AND `devices`.`id` IN (8368, 137, ...)
↳ /app/views/events/index.html.erb:10:in `_app_views_events_index_html_erb___1845829941033837665_155040'

...

Completed 200 OK in 251ms (Views: 78.7ms | ActiveRecord: 162.5ms | Allocations: 226640)

It was clearly visible that the preload worked. ActiveRecord made three queries to the database - one to get the events, another to get the users, and a third to get the devices.

But where does the WHERE deleted_at IS NULL come from?

I looked at the User and Device models. Both invoke the acts_as_paranoid method which the paranoia gem adds to ActiveRecord::Record.

We use that gem at work to soft delete records in the database - the data is kept in the table, it just isn’t returned unless specifically requested.

Paranoia knows which records are deleted by checking a deleted_at column that has to be added to each model that can act as paranoid. When the record is deleted, deleted_at is set to the current timestamp. A record can be restored (un-deleted) by setting deleted_at back to nil.

Paranoia filters out deleted records by adding WHERE deleted_at IS NULL to all queries.

Why are the queries to fetch the users and devices so slow?

In situations like these I reach for the explain method on an ActiveRecord relation. It tells the database to explain how it will execute the query. This shows important information like the indices it plans to use, in which order it wants to filter rows, and more.

Here is the explain result for one of the slow queries:

User.where(id: [
  97212, 2526, 15027, 11982, 10846,
  17091, 19528, 10731, 7785, 23240,
  23238, 7244, 7221, 6993, 11959,
  16931, 38234, 7006, 8993, 6766,
  39565, 31681, 7088, 7786, 34029
]).explain
#   User Load (79.75ms)  SELECT `users`.* FROM `users` WHERE `users`.`deleted_at` IS NULL AND `users`.`id` IN (97212, 2526, 15027, 11982, 10846, 17091, 19528, 10731, 7785, 23240, 23238, 7244, 7221, 6993, 11959, 16931, 38234, 7006, 8993, 6766, 39565, 31681, 7088, 7786, 34029)
# => EXPLAIN for: SELECT `users`.* FROM `users` WHERE `users`.`deleted_at` IS NULL AND `users`.`id` IN (97212, 2526, 15027, 11982, 10846, 17091, 19528, 10731, 7785, 23240, 23238, 7244, 7221, 6993, 11959, 16931, 38234, 7006, 8993, 6766, 39565, 31681, 7088, 7786, 34029)
# +----+-------------+-------+------+-----------------------------------+---------------------------+---------+-------+------+-----------------------+
# | id | select_type | table | type | possible_keys                     | key                       | key_len | ref   | rows | Extra                 |
# +----+-------------+-------+------+-----------------------------------+---------------------------+---------+-------+------+-----------------------+
# |  1 | SIMPLE      | users | ref  | PRIMARY,index_users_on_deleted_at | index_users_on_deleted_at | 6       | const | 25   | Using index condition |
# +----+-------------+-------+------+-----------------------------------+---------------------------+---------+-------+------+-----------------------+
# 1 row in set (0.00 sec)

The result shows that the database sees two indices with which it can filter the table to generate the result - PRIMARY and index_users_on_deleted_at.

Out of these two it choose to use index_users_on_deleted_at, which is a bit odd since the query mostly filters by IDs.

Why did it choose that index instead of using the primary key? And why is that so slow?

I ran SHOW INDEX, an introspection tool in MariaDB that shows statistics about indices on a table. The query planner uses those same statistics to plan how to filter the requested rows. Understanding what the query planner sees can explain why MariaDB picked index_users_on_deleted_at instead of PRIMARY

These are the statistics for the users table:

+---------+------------+-----------------------------+--------------+--------------+-----------+-------------+----
| Table   | Non_unique | Key_name                    | Seq_in_index | Column_name  | Collation | Cardinality | ...
+---------+------------+-----------------------------+--------------+--------------+-----------+-------------+-----
| "users" | 0          | "PRIMARY"                   | 1            | "id"         | "A"       | 3202800     | ...
| "users" | 1          | "index_users_on_deleted_at" | 1            | "deleted_at" | "A"       | 4003        | ...
+---------+------------+-----------------------------+--------------+--------------+-----------+-------------+-----

The Cardinality column shows that the PRIMARY index has millions of values indexed, while index_users_on_deleted_at only has a few thousand.

But why is that index so slow?

The decision by the query planner to use index_users_on_deleted_at felt off. It seemed like a mistake so I dug deeper.

MariaDB offers query optimizer tracing, which is fancy term for a “detailed explanation why it did what it did”.

I opened a DB console and ran SET optimizer_trace='enabled=on'; to enable tracing. Then I ran the slow query and right after that I checked to see the optimizer trace with SELECT * FROM information_schema.optimizer_trace LIMIT 1;.

This returned a large JSON object, which showed every decision that the query planner made and explained why it chose the approach it did.

The trace showed that the PRIMARY index had a much lower execution cost (was much faster) than index_users_on_deleted_at, but it chose to use index_users_on_deleted_at for “rowid filtering”. I had no clue what “rowid filtering” was so I went to look it up in the documentation.

Row ID filtering is an optimization. The basic idea is this - if you have two indices on a table, an you are filtering by both; more often than not it’s quicker to filter by the more restrictive index first, and then scan the result to filter by the less restrictive one.

Or in other words, it’s better to pick the index that eliminates most values first so that you have fewer rows to work with, and then scan through the result and filter by any other filters. Makes sense.

MariaDB, unlike PostgreSQL, doesn’t support conditional indexing. This means that each index you add to a table indexes the whole table.

This is great when you have many rows with different values in the indexed column, but it can backfire when you have a table that has many rows with the same value.

In my case the deleted_at column has around 4k different values in about 3 million rows. But a single value - NULL - of those 4k is associated with nearly all 3 million rows. This means that the database will load and scan through 3 million records to find the ones matching the IN clause - which is extremely slow.

How do I make the query fast?

Luckily MariaDB, unlike PostgreSQL, provides an escape hatch for situations like this. Through the USE INDEX option, it allows you to specify the index it should use when filtering.

ActiveRecord doesn’t have a method for USE INDEX but it can be added by passing a string to the from method.

Telling the planner to use the PRIMARY index speeds up the query drastically. The execution time fell from 80 ms to 1 ms.

User.from('users USE INDEX(PRIMARY)')
    .where(id: [97212, 2526, 15027, 11982, 10846, 17091, 19528, 10731, 7785, 23240, 23238, 7244, 7221, 6993, 11959, 16931, 38234, 7006, 8993, 6766, 39565, 31681, 7088, 7786, 34029])
    .explain
#   User Load (0.7ms)  SELECT `users`.* FROM users USE INDEX(PRIMARY) WHERE `users`.`deleted_at` IS NULL AND `users`.`id` IN (97212, 2526, 15027, 11982, 10846, 17091, 19528, 10731, 7785, 23240, 23238, 7244, 7221, 6993, 11959, 16931, 38234, 7006, 8993, 6766, 39565, 31681, 7088, 7786, 34029)
# => EXPLAIN for: SELECT `users`.* FROM users USE INDEX(PRIMARY) WHERE `users`.`deleted_at` IS NULL AND `users`.`id` IN (97212, 2526, 15027, 11982, 10846, 17091, 19528, 10731, 7785, 23240, 23238, 7244, 7221, 6993, 11959, 16931, 38234, 7006, 8993, 6766, 39565, 31681, 7088, 7786, 34029)
# +----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
# | id | select_type | table | type  | possible_keys | key     | key_len | ref  | rows | Extra       |
# +----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
# |  1 | SIMPLE      | users | range | PRIMARY       | PRIMARY | 8       | NULL | 25   | Using where |
# +----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
# 1 row in set (0.00 sec)

But this doesn’t solve my problem. I can’t call the from method in a preload query - that query is built and executed by ActiveRecord internally.

Though, now that I understand the problem I can trick the query planner into doing what I want.

If I replace preload with eager_load I can force the query planner to use at least one PRIMARY index and thereby speed things up considerably.

Device::EventLog.all
                .eager_load(:user, :device)
                .strict_loading
                .explain
#   Device::EventLog Load (2.7ms)  ...
# +----+-------------+-------------------+--------+-------------------------------------+---------+---------+-----------------------------------------+------+-------------+
# | id | select_type | table             | type   | possible_keys                       | key     | key_len | ref                                     | rows | Extra       |
# +----+-------------+-------------------+--------+-------------------------------------+---------+---------+-----------------------------------------+------+-------------+
# |  1 | SIMPLE      | device_event_logs | ALL    | NULL                                | NULL    | NULL    | NULL                                    | 25   | Using where |
# |  1 | SIMPLE      | users             | eq_ref | PRIMARY,index_users_on_deleted_at   | PRIMARY | 8       | development.device_event_logs.user_id   | 25   | Using where |
# |  1 | SIMPLE      | devices           | eq_ref | PRIMARY,index_devices_on_deleted_at | PRIMARY | 8       | development.device_event_logs.device_id | 25   | Using where |
# +----+-------------+-------------------+--------+-------------------------------------+---------+---------+-----------------------------------------+------+-------------+

And it worked! Both PRIMARY indices were used when filtering, and the query returns in 3 ms.

Why did this work?

I had a hunch that a LEFT OUTER JOIN would force the optimizer to use at least one PRIMARY index on the joined tables. Because eager_load does the same thing as preload, in this case, but using a LEFT OUTER JOIN it was the perfect solution.

And I was right. The query optimizer trace for the eager_load query showed that the optimizer tries to apply the same optimization as before, but its cost is higher than using the PRIMARY index.

The Device::EventLog table doesn’t have any more restrictive index that is also used in the query so it can’t apply the same optimization. While the joined tables are implicitly filtered by so many IDs that the cost of using the PRIMARYindex goes way down compared to the optimization.

At least that’s what I think is going on.

Either way, sometimes replacing preload with eager_load can utilize indices better and therefore speed up query execution.

P.S. Because of this problem I took a deep dive into query planners and optimizers in both PostgreSQL and MariaDB. This is a fascinating topic, a science on its own, and a world with very different opinions about what the “correct” solution is. It made me appreciate the “no hints” approach that PostgreSQL takes, but at the same time showed me the value of escape hatches that MariaDB provides. It made me see that there really is no one-size-fits-all solution for SQL databases - as always in software, you stick with what works for you.

140 million rows later

2022-10-07T12:00:00Z

At work, as part of a project I’m working on, I wanted to add a new reference to a table. ”Simple enough” - I thought. Spoiler alert, it wasn’t.

Adding a reference from one table to another is straight forward in Rails.

You create a migration using rails generate and then write in it something like add_reference :device_event_logs, :door, foreign_key: true, null: true. And then you run rails migrate. Boom. Done!

This is where my problems began.

The table I was adding the reference to holds event logs for all devices. Each event represents some kind of user interaction with the device. I know that there are a decent number of devices in the field, and a decent number of users in the system.

Since adding a reference can lock the table (prevent reading or writing to it) until the migration is done I wanted to be sure that it wouldn’t cause any problems.

I opened the DB console, entered SELECT COUNT(1) FROM device_event_logs and hit enter. A few seconds passed and nothing happened - “there must be a lot of records in the table” I thought. Then 30 more seconds passed and still no result - this was a bad omen. After 45 sec the query returned a count of 140 million rows.

Yikes. This means that adding the reference would take quite a while and use up a decent chunk of storage space.

Next I went to check how often this table is written to - to determine how big of an impact a few minutes of downtime would have. So I checked the statistics tables in MariaDB and the number of requests to the controller that the devices call to log events. There were around 30 inserts per second.

Yikes again. This meant that 5 min of downtime would cause 9000 failed requests - potentially 9000 unhappy customers.

I wanted to find a way to add this reference quickly, without locking the table and without using too much storage space.

Digging through the Rails documentation for add_reference I didn’t find anything that could help me solve this, so I resorted to adding the reference manually, step by step.

In Rails, a reference like add_reference :device_event_logs, :door, foreign_key: true, null: true creates a column to store the ID of the row being referenced in the other table, creates and index on that column and adds a foreign key constraint from that column to the referenced table’s primary key.

# `add_reference :device_event_logs, :door, foreign_key: true, null: true` is equuivalent to
add_column :device_event_logs, :door_id, :bigint, null: true
add_index :device_event_types, :door_id
add_foreign_key :device_event_types, :doors

Now I read through MariaDB’s documentation to see how I could optimize each of these three steps.

Turns out that adding the column is already optimized out of the box so I could just add it through add_column :device_event_logs, :door_id, :integer, null: true. That would create the column in less than a second and wouldn’t lock the table.

When creating an index, the MariaDB docs say that one can specify an algorithm with which the index is generated. Some algorithms are locking, some aren’t, some create a copy of the whole table, some don’t. But if you don’t pass an algorithm MariaDB will try to use the fastest, least locking, and most space efficient one possible.

Sounds great, but as I have been burned before by documentation leading to incorrect assumptions I decided to verify this.

So I ran CREATE INDEX foobar ON device_event_logs (door_id) on our staging database. It took about 20 min to complete, increased the database’s size by nearly double and I couldn’t insert or select from the table while the index was being created.

Good thing I checked.

Then I tired specifying the algorithm explicitly with CREATE INDEX foobar ON device_event_logs (door_id) ALGORITHM inplace and it erred out with a message saying that INPLACE isn’t supported on this table. I also tried all other algorithms none of which were supported, except for the slow and locking one - COPY.

This threw me down a rabbit hole. I spent a day trying to figure out why the better algorithms weren’t supported until I finally found a paragraph in MariaDB’s documentation explaining that tables created prior to MariaDB version 10.something.something can’t use the newer algorithms. So I checked when the staging table was created, and sure enough, it was way before that version was released.

Since the staging environment was setup after the production environment I’d surely run into the same problem there. What now?

After some more digging I found out that this problem can be solved by running OPTIMIZE TABLE device_event_logs. Which will do all sorts of magic, fairly quickly and with a lock on the table, to get the table working with all features of the current MariaDB version. So I ran it, and a second later it was done.

Now I could add the index with add_index :device_event_logs, :door_id, algorithm: :inplace. The algorithm option isn’t necessary, but I wanted the migration to fail if it would lock the table - in case I forgot to optimize it.

When creating a foreign key constraint, the MariaDB docs say that, you can pass an algorithm (just like with an index), and a LOCK=NONE option. But add_foreign_key in rails accepts neither of these options, so I had to resort to writing SQL.

execute(
  <<~SQL
    ALTER TABLE device_event_logs
    ADD CONSTRAINT fk_rails_53a1f7c81d
      FOREIGN KEY IF NOT EXISTS (door_id)
      REFERENCES doors (id),
      ALGORITHM = INPLACE ,
      LOCK = NONE;
  SQL
)

But when I tried to run the migration it failed, stating that an index on the doors table’s id column was missing. What? How? It’s the primary key!

After some more digging I found out that the column types in a foreign key have to match, else you get that error.

I added an integer column to device_event_logs thinking it would automatically use bigint which is the default primary key type in Rails, but it didn’t. So I went back and changed the add column to add_column :device_event_logs, :door_id, :bigint, null: true.

Finally I could add the foreign key, but it was very slow. I had a hunch that it was due to MariaDB checking if the key actually exists in the other table.

To speed it up I disabled referential integrity by wrapping the SQL command in a disable_referential_integrity block after which the foreign key was added in less than a second.

The downside of disabling referential integrity is that a foreign key might point to something that doesn’t exist, but that wasn’t a problem in this migration since the column the foreign key was added to was completely empty.

My migration looked like this in the end

add_column :device_event_logs, :door_id, :bigint, null: true

add_index :device_event_logs, :door_id, algorithm: :inplace

# https://api.rubyonrails.org/classes/ActiveRecord/Migration.html#method-i-reversible
reversible do |dir|
  dir.up do
    disable_referential_integrity do
      execute(
        <<~SQL
          ALTER TABLE device_event_logs
          ADD CONSTRAINT fk_rails_53a1f7c81d
            FOREIGN KEY IF NOT EXISTS (door_id)
            REFERENCES doors (id),
            ALGORITHM = INPLACE,
            LOCK = NONE;
        SQL
      )
    end
  end
end

The production migration took 15 min, caused no locks or downtime, and increased the size of the database by a gigabyte.

All in all, a big win and learning experience.

“Having a monolith is a single point of failure”

2022-09-30T12:00:00Z

I recently took part in a discussion that brought to light the most unusual argument against having a monolith that I have ever heard - that a monolith is a single point of failure.

I want to make clear that I consider monoliths and microservices neither good nor bad, or universally better or worse at any particular job. To me these architectures are just tools that work better in some settings and worse in others. As I explained in “Misguided Mark misrepresents Micro-services“, to me they are just paradigms - different ways to see and explain the same thing.

The details of the discussion aren’t important for debunking the argument. The important thing is that the discussion was centered around should a new feature be added to a monolith or should it be a microservice.

We covered quite a few common arguments for microservices all of which fell apart when examined. And then the other side brought up that having a monolith is a single point of failure.

This argument shows a deep misunderstanding of what microservices are and how they work.

People usually think about monoliths in terms of classes and methods (or modules and functions), but they think about microservices in terms of services and APIs.

Services and APIs are the same thing as classes and methods.

They are two sides of the same coin - on one side everything is in one program, on the other each class is its own program.

In other words, they are two paradigms through which people build and think about applications. They are talking about the same thing just from different perspectives.

If you have a monolith in which a call to the process method on the class Payment internally calls the authorize method on class User, then you have two places where something can go wrong. Either Payment#process can error or User#authorize can error. In both cases the call to Payment#process has erred.

If you have microservices in which a request to the process_payment API on the Payment microservice internally makes a request to the authorize_token API on the User microservice, then you have two places where something can go wrong. Either the request to process_payment on the Payment service returns a 500 or the request to authorize_token on the User service returns a 500. In both cases the request to process_payment on the Payment microservice has erred.

The number of failure points is the same.

Though in reality, the microservices example has two more failure points because each request to a service can also fail due to various problems - like a misconfigured load balancer, or the machine running out of file descriptors, or a congested router in the data center, or any other reason a network request might fail…

So where does this single point of failure argument come from?

I believe that Netflix and a simple case of not thinking with your own head are the source of this odd argument.

As most programmers know, Netflix is a big proponent of microservices. They have been making headlines across various sites popular in the IT industry how microservices have enabled them to weather any outage. They even made a tool called Chaos Monkey that turns off services at random to test how resilient your system is.

So if it works for Netflix why wouldn’t it work for us? Right?

Let’s apply Chaos Monkey to our example with two services and see.

If all services are on, payment processing works. But if one or both are turned off, payment processing fails. It can process payments only 25% of the time. A monolith would perform much better, it can either be on or off meaning that it can process payments 50% of the time.

Animation that shows in which scenarios the system is up or down depending on which of the two services is on or off

What gives? Maybe if we had two of every service it would be more resilient?

Animation that shows in which scenarios the system is up or down depending on which of the four services is on or off

It would, but only 50% of the time. A monolith would perform much better, as it would fail to process the payment only when all instances are off which means it would process payments 75% of the time.

The point of Chaos Monkey is to test how resilient your architecture is. It works with services because Netflix is invested in microservices and they have built a tool to test the resiliency of their architecture.

But architecture is so much more than microservices or monoliths. It’s how flexible the system is, how easily it can be extended, how easily a team can maintain it, and how resilient it is to failure.

The fact that Netflix has built a resilient architecture with microservices doesn’t mean that you can’t achieve the same with other architectures.

Resilience to failure can be addressed both in microservices and in monoliths through the same patterns - circuit breakers, policies/strategies, and others.

The humble ActiveModel

2022-09-22T12:00:00Z

ActiveModel is one of my most used tools in Rails applications. I use it in service objects, form objects and objects that represent external entities.

Why? Because it provides a nice interface for validating inputs and results, it can have callbacks for pre and post-processing data, and it integrates well into various Rails conventions.

A simple login form is my go to example for explaining how nice ActiveModel is to work with.

I have seen people do login flows in controllers, in User models and in interactors. But no other approach is as straightforward as creating a Login model.

# /app/models/login.rb
class Login
  include ActiveModel::Model
  include ActiveModel::Validations::Callbacks

  attr_accessor :username, :password

  validates :username, presence: true
  validates :password, presence: true
  validate :validate_user_exists!
  validate :validate_password_for_user!

  before_validation do
    @user = nil
  end

  def validate_user_exists!
    return if user.present?

    errors.add(:username, "not found")
  end

  def validate_password_for_user!
    # `authenticate` comes from Rails
    # Reference:  https://api.rubyonrails.org/classes/ActiveModel/SecurePassword/ClassMethods.html#method-i-has_secure_password
    return if user&.authenticate(password)

    errors.add(:password, "is invalid")
  end

  def user
    @user ||= User.find_by(username: username)
  end
end

Which can then be used in a controller like any other model:

class LoginsController < ApplicationController
  def new
    @login = Login.new
  end

  def create
    permitted_params = params.require(:login).permit(:username, :password)
    @login = Login.new(permitted_param)

    if @login.valid?
      session[:current_user_id] = @login.user.id
      redirect_to root_path, notice: "Welcome back!"
    else
      render :new, status: :unprocessable_entity
    end
  end
end

You can use form_for without extra arguments or configuration, I18n works for the attributes and error messages just like it does for ActiveRecord models, and if the login fails the fields get repopulated.

It looks and feels like you are working with an ActiveRecord model, which most of us know and love.

When it comes to Service objects, the ActiveModel approach offers a way to communicate success and failure through validations. You can check the inputs with valid?, or you can check the result, and return a nice error message.

class Device::WarehouseReservator < ApplicationModel
  attr_accessor :device, :user

  validates :device, presence: true
  validates :user, presence: true

  after_initialize do
    self.user ||= device.creator
  end

  def reserve!
    return false if invalid?

    response = HTTP.post("http://device.inventory.com/api/v1/register", data: payload)

    if response.ok?
      errors.add(:device, “failed to register”)
      return false
    end

    self.warehouse_reference = device.create_warehouse_reference(reservation_id: JSON.parse(response.body).fetch(:id))
  end

  def payload
    { device_id: device.id, name: device.name, reservation_holder: user.name }
  end
end

reservator = Device::WarehouseReservator.new(user: User.all.sample, device: Device.all.sample.reserver!
reservator.valid? # => true
reservator.reserver!
Reservator.warehouse_reference.id # => 1

reservator = Device::WarehouseReservator.new(user: nil, device: nil)
reservator.valid? # => false
reservator.reserver! # => false
reservator.errors.full_messages # => ["User missing", "Device missing"]

This is a much nicer way to communicate what went wrong than raising and catching errors, returning symbols or custom Error objects.

validate and coerce inputs by type through ActiveModel::Attributes
handle password storage and checking through has_secure_password
add custom callbacks with ActiveModel::Callbacks (I usually implement an after_initialize_callback in ApplicationModel)
track changes to attributes using ActiveModel::Dirty

ActiveModel is quite a versatile tool for that alone, but there is much more that it can do like:

P.S. Many thanks to Hrvoje S. For reviwing this article.

Take a break

2022-08-27T12:00:00Z

At my first job I used to take a break every time I got stuck on a problem until one day my boss caught me looking out of the window.

He didn’t say anything the first time he saw me, but when he saw me in the same position some 15 min later he yelled “At least sit by your desk and have your laptop open so that I think you are working instead of slacking off”.

As I switched jobs, I learned that “at least make it look like you are working” is a prevalent philosophy in most offices. So I adapted and started scrolling through Reddit while on break to seem busy. But this didn’t work out as my boss had hoped for - I started taking more breaks because I couldn’t focus more often.

Thankfully, I didn’t have to work in an office for the last 4 years so I was able to return to my old routine. But why was this method so much more effective than scrolling Reddit for 15min?

Rachel and Stephen Kaplan developed a theory that might explain why - Attention Restoration Theory or ART for short. ART differentiates two kinds of attention humans can give - direct attention and effortless attention.

We use direct attention when we focus on something - like trying to solve a problem, searching for something, following rules… When giving direct attention we consciously block out most distractions.

Direct attention is a finite resource - if you use it too much you will run out of it and won’t be able to focus anymore - but it replenishes if it’s not used for a while. You can think of it as a muscle - if you use it too much it becomes sore and weak, but if you give it some rest it can work for much longer.

When you run out of direct attention you start suffering from attention fatigue - a state in which you are easily irritable, tired and stressed out.

In contrast, effortless attention is… well… effortless.

The Kaplans tell us that nature is full of fascinating things to experience and give our effortless attention to - like the shapes of clouds, the wind on your skin, rustling of leaves, the sun breaking through the treetops, the chirps of birds, the sound of water rushing in a creek... So by looking at or being in nature we can replenish our direct attention because we will mostly use effortless attention to process it.

The reason I took more breaks while scrolling Reddit was over use of direct attention. Reading through posts and commenting takes direct attention to do, therefore it doesn’t replenish during the break and I couldn’t focus as well when I started working again.

But there is another compounding problem - attention residue.

Attention residue is an effect that prevents humans from switching their attention between tasks effortlessly. When you switch tasks your mind will still pay attention to the previous task for a while and prevent you from fully focusing on the new task.

Whatever posts I read on Reddit during my break would linger in my mind and steal attention away from the problem I was trying to solve when I started working again. And when I took a break any problem I was working on before would still invade my thoughts and steal attention from what I was doing during my break.

The problem with scrolling through Reddit while on break was that my attention was depleted before the break, it couldn’t replenish during the break, and then it was split between Reddit and work after the break.

To be clear, Reddit isn’t the problem here - being unnecessarily busy is. I would have probably felt the same way if I read the news, a book, scrolled through Twitter, solved a Sudoku or played chess.

Today, when I feel like I’m having a hard time focusing, I go out for a 15min walk if the weather is nice. If the weather doesn’t play along I make myself a cup of tea, look out of the window and enjoy the birds, the clouds passing by, the falling snow, or the sound of wind and rain.

A proper 15 min break can keep you happy, focused and rested for hours.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The sun shining through the tree tops in a forest

P.S. Many thanks to Marko I. for reviewing drafts of this article.

My inspiration for this article was a conversation I had with a co-worker who didn’t want to take a break when he was stressed out, after which I started encouraging all my co-workers to take a break when they start to feel attention fatigue. This is a cautionary tale for them.

Keep it boring, don’t surprise me

2022-08-21T12:00:00Z

I’ve spent a lot of my life worried that people will think I don’t know enough. Sometimes, that worry has made me use big words when I didn’t need to.

–Randall Munroe in the foreword to the Thing Explainer

I used to be a stickler for organizing code by what it was. Models, decorators, form objects, view objects, value objects, service objects - all lived in their own separate directory. I felt that the code was well organized that way, that everybody knew what went where, and what the purpose of everything was just by how the code was organized.

My obsession with grouping things that way made me blind to what I was actually doing - writing surprising code.

When I think of the reasons why I did things that way only three things come to mind.

First, I wanted to show off how much I knew about patterns, algorithms and programming. I wanted to be the best programmer, and prove that to my peers by organizing code in a manner that showed off my knowledge.

Second, I didn’t know any better. The code was already organized into models, views and controllers so it was natural to me to continue this trend of having every kind of object separate.

Third, I didn’t want to have “fat” models or controllers because that has bitten me in the past.

Today I consider most things to be a model, with a few exceptions. Why? Because decorators, form objects, view objects, value objects, service objects, and others are just different ways to model data. Models aren’t just objects persisted in a database, a model is any object that represents a piece of data. Through them your app represents the world so that it can work with its data, similar to how you have mental models about the world around you (e.g. what is night and what is day).

Think about it. A model backed by a database - like User - just represents a row in that database that holds data about a person. It takes that row and wraps it into an object you can interact with - e.g. to get the person’s name or email.

Decorators, form, value, service and view objects do the same. They take one or more pieces of data and give you an object to interact with that data in different ways.

This might be a reductionist way of thinking because controllers and views do the same. But they are novel concepts that are used so often, and that are so important, that separating them out makes building the application easier.

For instance, I work on an IoT application. One of the most common actions we do is to send messages to a devices. Because it is so novel and so common, we have extracted “device messengers” out of models into their own thing. That way developers don’t have to search for them and we communicate that they are an important part of the app.

Treating most things as models, and making models heavier, has made the code I write much less surprising.

Having super skinny objects produces surprising code because you have to be very verbose. Something boring like user.purchase(line_items).charge(authorization_code: params[:authorization_code]) becomes a convoluted mess like PurchaseChargerService(purchase: PurchaseBuilderService.new(line_items: line_items).call, form: purchase_form, user: current_user).call.

Nobody expects things like PurchaseChargerService and PurchaseBuilderService to exist, yet alone that you have to call those in succession with a form object and the current user to make a purchase. While most people working on a web shop expect that a user can just purchase something, and that a purchase can be charged.

By writing skinny objects you are pushing the complexity onto the programmer instead of having it in code.

I’m not saying you should get rid of PurchaseChargerService or PurchaseBuilderService (ok, maybe the builder is pushing it a bit, I'd get rid of that one). I’m saying that you should make them as boring as possible.

For all you know user.purchase might call those same objects internally. You don’t have to know about that to make a purchase. It’s boring and it’s beautiful.

How to make things boring?

First, embrace the domain you are working with.

Naming something a service and putting it in its own directory solves no problems, but it can create some. Naming something after what it does and putting it next to the thing it works with tells a story that can help people understand what’s going on.

Instead of PurchaseChargerService that lives in a service objects directory, try naming it Purchase::Charger and putting it under the Purchase model (e.g. app/model/purchase/charger.rb). You will know just by looking at the file that it has something to do with purchases and charging them.

Second, write code that mimics how people think about the domain you are working in.

Expose methods for common actions in your domain. If a User can purchase line items, then add a purchase method to the User model that accepts line items.

Don’t push the complexity of figuring out how to make a purchase onto other people, solve it once in code, and explain it well so that others can build on top of your work. Don’t try to be too smart, most often people won’t care how the purchase is made so don’t try to expose it if it’s not needed.

Third, extract actions into models to tell a story and form new domains.

Don’t create new classes just to keep things DRY. If your model has many small or a single large method that deal with a single action, extract it into its own model.

If the User model has 5 methods related to purchasing or a single large purchase method, extract that into a Purchase model and delegate to it. If the Purchase model has 10 methods that deal with charging credit cards, extract those into a Purchase::Charger model and delegate to it.

Nobody will think less of you for writing boring code.

Nobody will fuss about a method, module or class that is understandable but is too pedestrian.

Nobody wants to work with code that requires a manual to figure out how to do the most basic actions.

Keep it simple, keep it boring, and don’t surprise other.

Update on 2022-08-27

After sharing this article with a few friends I got a wonderful question which I want to address.

Q: If you put everything in app/models isn’t that surprising too? People will expect an ActiveRecord-like object and they can get anything.

A: Yes that would be surprising, but I don’t see a situation where someone would write Purchase::Charger.new(purchase: Purchase.all.sample).save and hit Enter out of the blue.

If you see Purchase::Charger you’d at least be curious why the model name isn’t a noun - the name implies that it’s different.

Why would someone create and try to save an object they know nothing about? That implies that they are just passing random arguments to random methods expecting things to just work.

I don't see this as a realistic scenario, I expect people to know what they are doing and if they don't to read up until they do. In other words, I trust them to do good work.

But I agree that there are cases where it’s beneficial to have a similar interface for some objects. Therefore, I make most objects in app/models inherit from ActiveModel::Model. There are many reasons for this which I think I’ll cover in a separate article.

P.S. Many thanks to Hrvoje S. for reviewing drafts of this article.

Scrum is for time estimates, not projects

2022-07-15T12:00:00Z

From my experience Scrum is the prevalent project management framework in software development. Most of the teams I was part of used it to develop, deliver and maintain their projects. Despite its prevalence I always felt like Scrum was bogs me down - that it was more harmful than useful.

I knew what was bothering me the most - the daily standup. But even in teams that had a standup once a week I felt like Scrum didn’t make the project better. Yet we were burning two work-hours per person every week on its ritual.

One day while reading about Control Theory and PID controllers, I realized that Scrum was a time estimation tool.

To understand this, you first have to know what a PID controller is and how it works. In a nutshell, it’s a device that observes a dimension of a process - like the speed of a wheel - and tries to keep it at a desired value.

If the wheel goes too slow the controller speeds it up, if it goes too fast the controller slows it down. The magic of PID controllers is that they, once tuned, don’t over or undershoot the desired value. Their magic consists of 3 steps - P, I & D.

P, the proportional, corrects the difference between the desired and actual value. I, the integral, looks at past over or undershoots and corrects for them. D, the derivative, nudges the result closer to the ideal value.

That is exactly what Scrum does. Standups correct problems as they occur. Planning corrects over or undershoots in the committed amount of work. Retros nudge the process towards the ideal outcome.

If Scrum is a PID controller what dimension does it control?

The only one that has a fixed value - the duration of the sprint. All other values are in flux - the number of story points, the number of tasks in a sprint, the number of people in a team.

By controlling the duration of a sprint you can estimate the duration of a project. If you chop up a project into 30 tasks, and the team solves about 8 tasks every sprint, you can be somewhat certain that the project will take 4 to 6 sprints. If a sprint is 2 weeks long, then the project will take 12 weeks.

This chopping up of a project into tasks is why Scrum bogs me down - it kills the vision. Without vision you can’t know what you are building and therefore you can’t utilize your expertise to make it better.

Building a project task by task can produce something that doesn’t work as a whole. Each task makes sense on its own but all of them together don’t form a whole - they don’t accomplish the goal of the project.

Because tasks don’t carry a sense of how they fit together, any sense of wholeness is accidental. One feature can be split into multiple tasks which can produce a different result depending on which one is implemented first. New tasks can be added with no regard as to how they incorporate into the vision, which can produce nonsensical features. When a feature is broken into multiple tasks, changing one task without updating the others leads to a disjointed feature.

The worst possible case is when someone creates tasks that build on top of each other. Working on such tasks is like being led by a carrot on a stick through what someone else thinks is the best way to build the project. Instead of letting you, the expert at building software, decide what the best way to build it is. This is frustrating and demotivating, and you can’t prevent it because you can’t see the bigger picture.

Scrum isn’t a framework for people building the project, it’s a framework for people concerned with timelines and roadmaps.

Builders are concerned with how things are supposed to work, how they connect together and interact with one another. They are concerned with what they are building, with the vision.

Scrum has no mechanism to help you build your project. None of its rituals can help you shape a project. They can’t help you form a vision for what you are about to build, or help you figure out what is still undefined. Scrum assumes that this is already figured out when you start grooming and planning the project. The best you can ever do is say “I have no clue what we are building” and throw a sprint out.

Yet the burden of making Scrum work falls on the ones it doesn’t help - the builders - the ones concerned with building something great. It only helps them adhere to a timeline. But a bad project delivered on time is still a bad project.

P.S. Many thanks to Hrvoje S. for reviewing drafts of this article.

If you are looking for a framework that does help you shape your project and build something better, but has a looser concept of timelines, take a look at Shape Up. I recommend that book even if you aren't looking for a new framework, it describes excellent tools to help you identify if your project is well defined - like breadboards and fat marker sketches.

Misguided Mark misrepresents Micro-services: A story about paradigms

2022-07-11T12:00:00Z

Tom was brewing tea in preparation for a 5 o'clock meeting he knew little about. The agenda was so vague it might as well have been left out - it read “Mark introduces a new system for controlling 3rd party IoT water sprinklers”. That was standard practice for CorpCorp - the company Tom worked for - everything was a meeting and things were only ever presented through meetings with at least 10 attendees, even when an email would be more than enough.

But Tom wanted to attend this meeting. He was the head of the Web development department and 3rd-party integrations were his responsibility. His team was short-staffed to the point that they couldn’t start working on this integration until the next quarter.

When Mark, a developer from the in-house IoT sprinkler department, offered to develop this integration Tom immediately accepted - Mark used to work as a Web developer before he joined CorpCorp and Tom thought that maybe he wanted to switch departments.

5 o’clock rolls around, Tom joins the meeting from his home office and takes a big sip of tea.

On his screen is Mark, standing in front of a nice looking slide deck that in bold letters says “Controlling 3rd party water sprinklers: The future of CorpCorp”.

Mark greeted everybody and started to present. He explained that he had spent two weeks planning out this integration and came to the conclusion that it will be built using micro-services - 13 of them - even though most of CorpCorp’s Web services were in a monolithic Web app.

Mark explained that monoliths were a thing of the past - they didn’t scale, and always ended up too complex to maintain. Micro-services, on the other hand, scaled infinitely and were super easy to maintain because having everything separate improved encapsulation and separation of concerns.

Mark stopped, looked at all the confused faces, and asked if there were any questions. Immediately a few managers that didn’t listen to half of the presentation asked simple questions that Mark already explained during his presentation - this was also standard practice at CorpCorp.

Tom had a working theory that the managers get bored by the technical jibber-jabber and doze off, but when the meeting ends they want to prove that they actually listened and that they were a useful addition to the meeting, so they ask a question about the last thing that they remember. Further experiments and observations were required to prove this theory, but even knowing if that was the case wouldn’t solve the problem - that the meeting should have been an email.

Once the flurry of questions ended Tom started to talk. He thanked Mark for taking the time to research this integration, and confirmed that it would be a good solution to the problem, but that he shouldn’t move ahead with its implementation because the system would cause too many problems down the line. He then added that Mark couldn’t see how misguided he was about monoliths because he clinged too hard to his paradigm to think level-headed.

The managers woke up out of a sudden - they didn’t want to miss the drama.

Tom started to address each point of Mark’s presentation. He began by asking Mark ih he knew why CorpCorp developed a Monolith in the first place. Mark just shrugged and guessed that it was because they were at the peak of their popularity when the project was started. Tom refuted that and said that a monolith was conscious decision by the department because:

MONOLITHS ARE EASIER TO MAINTAIN FOR SMALLER TEAMS

Tom’s department counted 12 people, but those 12 people were enough to handle most of CorpCorp’s business. Only in recent months, after one employee retired and another quit, did Tom’s department start to struggle with meeting business demands.

Tom’s employees built a monolith and decided upon a convention for going about their most common tasks. This allowed them to be flexible - anyone could work on any part of the system because they knew how everything was supposed to be organized. And over time, as features were added or removed, and bugs fixed, everybody became familiar with most of the system. So anybody was able to add a feature or fix a bug anywhere in the system.

You could achieve the same by implementing a convention on top of micro-services, but there is nothing stopping you from implementing multiple conventions. The more people that work on your micro-services, the more opinions there are on how they should be implemented, the likelier the chance that convention will be broken or changed.

In fact that is a big selling point of micro-services - you can use any technology and any convention you want for each individual service. And that is a desirable trait for mammoth companies that suck up developers just so the competition won't get them - they can keep them happy by letting them do what they want in their own micro-cosmos. This sometimes works out and sometimes it doesn’t, but for these kinds of companies that is just part of doing business.

In general, it is better for individuals and small teams, to be intentional with what they do and how they go about it. Especially in software development - where the question isn’t “is it doable?”, because the answer is always “yes”, but “how much time will it take?”.

And a monolith is great at ensuring that - one vision, one codebase and everybody is aligned.

MONOLITHS ARE EASIER TO SCALE, BUT THEY DON’T SCALE AS WELL AS MICRO-SERVICES

Tom decided to use an analogy here so that the managers wouldn’t doze off again. A monolith is like a hamburger, it is a meal consisting of a bun, a patty and some salad. While micro-services are a patty, some bread and a salad as three separate meals.

With a monolith, if you want to double the throughput you double the number of running instances. This is the equivalent of ordering two hamburgers when you are very hungry. The good thing is that it’s simple - you just double everything and you are done, the bad part is that it can be wasteful.

In micro-services, if you want to double the throughput you first have to identify what the bottleneck is, and then you double that service and some of its upstream services. This is the equivalent of ordering two patties, a regular salad and a bun when you are very hungry - you only pay for what you really want.

But figuring out what the bottleneck is can be hard, both in monoliths and micro-services.

COMPLEXITY COMES FROM BUSINESS NOT CODE

The notion that monoliths are complex to maintain and micro-services aren’t is a myth. In a monolith and in micro-services complexity comes from the business domain - what the app does. If the business domain is complex or not thought out - if there are no clear rules, if the process is dependent on many factors, if the process has many interdependent steps, if the process is convoluted to humans - the code will reflect that.

The code you write mirrors how you think about a problem. If you don’t think clearly, or if the problem is inherently muddy, then you are out of luck - architecture has little to do with that.

MICRO-SERVICES IMPROVING ENCAPSULATION OR SEPARATION OF CONCERNS IS A MYTH

Tom had to resort to another analogy here to keep the managers listening. Encapsulation is the act of bundling together things that belong together. It is in a way similar to doing laundry - imagine you have two laundry baskets: one for colored clothes and another for white & black clothes.

Separating the clothes in their baskets is encapsulation - each basket encapsulates only its set of clothes. But where would you put dark blue underwear?

On the one hand it is blue - a color - so it should go into the color basket. But on the other hand it is dark blue, which is close to black - so it should go into the white & black basket. Depending on who you ask you will get a different answer.

But people who argue that micro-services improve encapsulation & separation of concerns argue that, somehow, the problem will solve itself if you put the laundry baskets really far apart.

If you are bad at separating concerns or encapsulating code, you will be equally as bad at it in a monolith and in micro-services.

Tom stopped for a moment to look at everyone’s faces, the managers were more alert than they have ever been while Mark was embarrassed. Tom felt bad, and decided to give a moral to this story to lighten the mood.

He said that this was a common mistake that young developers make - they get too attached to their paradigm and become unable to understand why someone would do things differently. They think that the existence of a new paradigm - a new school of thought - must invalidate the old one. Why else would then the new school exist? Why would micro-services exist if monoliths were good?

People form an understanding of the world through paradigms. There is no one perfect paradigm, no one ultimate school of thought, just like there is no one perfect theory of how the world works..

Look at Newton’s and Einstein's model of gravity. Einstein’s model is superior to Newton’s - it is better at explaining how objects will move when they are going slow and when they are going close to the speed of light.

This precision comes at a cost - it is much harder to compute Einstein’s gravity equations than Newton’s. But is this computational cost worth it? When the object is moving slowly, Newton’s equations are a fraction of a percent off compared to Einstein’s equations.

Many civil engineers say the trade off isn’t worth it. For their day to day work they use Newton’s equations because they can iterate bridge, road, and building designs quickly and find the best solution. But NASA engineers would beg to differ - a precise result is much more important to them.

Each paradigm is good at something and bad at something else. It is up to you to understand the trade off and pick what works best for you.

P.S. The style of writing in this essay was inspired by a book I recently read - Are your light on? by Donald C. Gause and Gerald M. Weinberg. Its a fantastic book about problem solving and thinking about problems.

Many thanks to Marko I. and Hrvoje S. for reading drafts of this article and providing feedback

How I stumbled upon Strada while forwarding an email

2022-07-01T12:00:00Z

I wanted to forward an email one evening, so I opened up the Hey app on my phone, found the email, tapped on the “More” button, and just before I hit “Forward” I noticed a “Share or print…” button at the bottom of the screen. I hit “Share” and to my surprise was greeted by a share sheet. It offered me to send a link to the email via SMS, Slack, Signal, and others.

If it was any other app this wouldn’t surprise me, but Hey is a hybrid app — its views are server-rendered HTML and some JavaScript wrapped in an app that is essentially a web browser. Showing a share sheet isn’t something you can do with HTML and JS, so how did the team at Hey do this?

Opening a share sheet in Hey

I assumed that the Hey mobile app used the same HTML and JS as the web version. So I opened the Hey web app on my computer and started poking around.

The first thing I noticed was that all the buttons that were visible in the mobile app were present in the web version. But they were hidden unless the frontend was running in a browser that used a special User-Agent header. So I changed my User-Agent and reloaded the page. The UI changed — the header row was gone, some buttons looked like they were disabled and new buttons showed up.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Left: Hey web app in Firefox with the default User-Agent. Right: Impersonating Hey iOS app by changing the User-Agent

I thought the layout changes were related to Turbo Native and the disabled buttons seemed like a CSS issue, so l focused on the “Share and print…” button.

Inspecting it showed that it was controlled by a bridge/share Stimulus controller. So I opened that controller and saw that it only had a component static variable and a #share method. The static variable was hard-coded to the value “share”. The #share method just passed the current URL to a #send method it inherited from bridge/base_component_controller.

// bridge/share.js

import BaseComponentController from "bridge/base_component_controller"

class ShareController extends BaseComponentController {
    static component = "share"

    share(event) {
        event.preventDefalt()
        this.send("share", { url: window.location })
    }
}

I followed the code and opened the bridge/base_component_controller. Its #send method took an event name, a data object, and a callback function. It built a message object with the event name, data, callback and the controller’s component. Then it forwarded everything to the window.webBridge.send method and stored the returned value as the ID of the message.

// bridge/base_component_controller.js
// Pseudocode for BaseComponentController's send method
send(event, data, callback) {
    let message = { component: this.constructor.component, event, data, callback }
  let messageId = window.webBridge.send(message)
}

But where did window.webBridge come from? I searched all the files and found it was initialized right after a class named “Strata” was loaded. From the name of the class I knew I was looking at Strada — the, yet unreleased, framework from Hotwired.

This got me excited! I started reading the code of the class to figure out what it does.

// strata.js
// Pseudocode for Strata's send method
send(originalMessage) {
    if (!this.supportsComponent(message.component)) return null
    let id = this.generateID()
    let message = {
        id: id,
        component: originalMessage.component,
        event: originalMessage.event,
        data: originalMessage.data || {}
    }
    this.adapter.receive(message)
    if (originalMessage.callback) this.callbacks[id] = originalMessage.callback
    return id
}

receive(message) {
    const callback = this.callbacks[message.id]
    if (callback) callback(message.data)
}

supportsComponent(component) {
    return this.adapter.supportsComponent(component)
}

On load, it initialized itself and was assigned to window.webBridge. Then it fired a web-bridge:ready on the document.

It had an adapter variable which was called in most of its methods.

Its #send method generated an ID for each message, then it appended the ID to the message, stored the callback and forwarded the rest to the adapter.

There was also a #receive method which seemed to take a message object and execute any callback method associated with its ID. That explained why the callbacks were stored when a message was sent.

Then there was the spportsComponent method that queried the adapter for components it could support. Probably to avoid sending messages that the adapter couldn't process.

Now I started looking for the adapter implementation but I couldn’t find it. This was where the code ended.

And then it hit me! This would usually run within a browser view controlled by the native app. The app can inject JavaScript into it. That’s why I couldn’t find the adapter implementation — it was in the native app.

But why would the app inject the adapter into the frontend? This didn’t make sense at first. Then I remembered that in browsers controlled by a native app there is a special window.external object (on iOS it’s window.webkit). The app can expose functions on that object that when called from JS invoke a handler function in the app.

So, the app probably registered an adapter by injecting JS into the browser. That would explain why web-bridge:ready fires when Strada loads — so that the app knows when it can register the adapter. And through that adapter, the frontend can send messages to the app.

// AppDelegate.swift
// Example of how you could register an adapter with Strada from iOS
let supportedComponents = ["share"]

let supportedComponentsJSON = String(
    data: JSONSerialization.data(
        withJSONObject: supportedComponents,
        options: []
    ),
    encoding: String.Encoding.utf8
)

let registerAdapterJS = """
window.registerStradaAdapter = () => {
  const supportedComponents = \(supportedComponentsJSON)

  window.webBridge.setAdapter({
        platform: "ios",
        supportsComponent: (name) => return supportedComponents.include(name),
    supportedComponents: supportedComponents,
        receive: (message) => window.webkit.messageHandlers.strada.receive(message))
    })
}

if (window.webBridge) {
  window.registerStradaAdapter()
}
else {
  document.addEventListener("web-bridge:ready", window.registerStradaAdapter)
}
"""

// Inject the script right after the document is loaded
webView.configuration.userContentController.addUserScript(
    WKUserScript(
      source: registerAdapterJS,
    injectionTime: WKUserScriptInjectionTime.AtDocumentEnd,
    forMainFrameOnly: true
  )
)

And the app could send messages back to the frontend by injecting them into the browser like JavaScript.

// AppDelegate.swift

// Example of how an iOS app could add a
// `window.webkit.messageHandlers.strada.receive`
// function and process calls to it.

// Somewhere in AppDelegate after you have created your
// browser view (webView) and have initialized Turbo Native

let strada = Strada()
let contentController = webView.configuration.userContentController
contentController.addScriptMessageHandler(strada, name: "strada")

// Strada.swift

// Example message handler that responds to calls to
// `window.webkit.messageHandlers.strada.receive`

class Strada: WKScriptMessageHandler {
    func userContentController(userContentController: WKUserContentController!,
                                                         didReceiveScriptMessage message: WKScriptMessage!) {
        // The `name` attribute holds the name of the function that was invoked
      if (message.name != "receive") {
        return
    }

    if (message.body["component"] == "share" &&
        message.body["event"] == "share") {
      // show share sheet
    }
    }
}

// Strada.swift

// Example of how you could call the adapter from iOS
let message: [String: Any] = [
  "id": "7",
  "data": ["foo": "bar"]
]
let messageJSON = String(
    data: JSONSerialization.data(
        withJSONObject: message,
        options: []
    ),
    encoding: String.Encoding.utf8
)

webView.evaluateJavaScript(
    "window.webBridge.receive(\(messageJSON))",
    completionHandler: nil
)

I realized that Strada was a bridge between the native app and the frontend.

And through that bridge one can patch in functions that the platform supports but the browser doesn’t — things like showing a share sheet.

The mystery of the share sheet was solved. But now my head was buzzing with possibilities.

Most native apps are functionally the same as, or a subset of, their web counterpart. Without Strada, a native app would have to re-implement some of the views and logic of its web counterpart. This means that a team would have to build the same app at least twice — once for the web and another time for iOS, Android, Mac, or Windows — just so they could use some platform-specific features. This is even more wasteful if we take into account that most web apps are already optimized to work on phones, tablets, and PCs.

But with Strada, any existing web app can become a native app.

Many thanks to Karla, Marko, Hrvoje & Vlado for reviewing drafts of this article.

The Judo way

2021-08-28T00:00:00Z

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A walkway at Plitvice Lakes, Croatia

As a kid I practiced Judo which is a martial art as well as a philosophy. And my Judo teacher gave us a philosophical story which I came to understand only recently and which helped me achieve more than I though was possible. So I wanted to share it.

The story was about a small tree in a strong wind. The strong wind could uproot big trees but couldn't do the same to small trees. How so?

The teacher explained that the small trees are flexible so they can bend and move with the wind, therefore the wind can never exercise it's full force on them. But the big trees aren't as flexible and their only way to take the force of the wind is to develop deeper roots, stronger trunks and branches. So you don't have to be a big strong tree to take on an opponent such as the strong wind if you can move with the wind.

This analogy completely flew over my 11 year old head at the time, but I remembered the story and came to understand it a few years ago.

The point of the story, and the philosophy of Judo, is to achieve maximum efficiency with minimum effort (similar to the Pareto principle or the 80/20 rule). The best way to do so is not to fight the incoming force but to use or steerer it in a direction that puts you in an advantageous position. The key to understanding the power of this philosophy is to understand that you can replace the word "force" with anything else, and to accept that an advantageous position might not be exactly what you want but a step in that direction.

In the case of the story, if your goal is to stay in the ground then it's much easier to be flexible and move with the wind. You could also develop deeper roots and a stronger trunk but that requires more effort than moving with the wind and eventually a stronger wind could come along and uproot you.

I've used this philosophy to attain personal, and professional, goals which range from losing weight to chaining our process at work - things I previously thought were too laborious to be practical.

For example, at work we use the Scrum methodology and it's not a good fit for us. The development team is increasingly unhappy about the work we do, and the time we get to work on things that aren't part of a sprint. So we read about other methodologies and composed a list of things we would like to borrow from them to improve our process.

We can't just change the process on our own, we have product managers and Scrum masters on which these changes depend on too. So how do we go about implementing those changes?

We could form a document and a slideshow, collect all the evidence and studies to support our proposed process changes, and present everything to management. They are rational people, they will understand and accept our proposal, right?

While this sounds reasonable I can tell you right away that it will most likely fail. And the worst part is that not only will your process be rejected but all the improvements that could be applied to the current process that are bundled with it will be thrown out as well.

The problem is that management often doesn't know, and can't understand, the price of this process - the tech dept piling on, the extra hours, the emergencies on a Friday night which could have been avoided, the endless planning meetings to hack around self imposed system limitations. They either don't see those problems, don't feel them directly, or think they are a badge of honor for you. Your team is shipping value, customers are happy, things get done, so why change something that works and introduce uncertainty the process?

The Judo way to do it would be to point out one issue the team felt during the last sprint at each retrospective meeting and propose the process improvement you want to solve it. This is using Scrum against itself - the retrospective meeting is here to solve process problems so the issues your raised have to be discussed and acted upon. If your solution is accepted, or something similar to it, you will be one step closer to your goal.

New coat of paint

2021-08-09T00:00:00Z

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Abstract color pour by Pawel Czerwinski

Recently I got the urge to write again so I gave this blog a new coat of paint, and decided to give a brief history, the reason I stopped writing and an explanation as to why I moved away from Medium.

I started this blog back in 2016 as a way to share and discuss ideas and solutions to problem's I've tackled as part of my work, life, education and hobbies.

At the time Medium was the hot new, elegant, hosted blog engine with a lot of buzz around it. I jumped on the bandwagon and started my blog there as I didn't have much money at the time, didn't want to build my own blog and didn't want to self-host or pay for WordPress.

The first article here is a story about a hackathon I went to in 2016 after which I basically stopped writing. Then, two years later, I switched jobs and technology stacks which lead me to reevaluate most of the things I knew about web development so I started writing again to crystallize my thoughts.

Then "Do you really need WebSockets", an article I wrote because I was tired of explaining to people that SSE and long-polling are not the same thing, became popular. The feedback was overwhelmingly positive and I got hooked to the high of having a popular article.

Chasing that high made me imitate my previous success. All articles after "Do you really need WebSockets" were in-depth comparisons. But writing those comparisons wasn't fun, it felt like a chore. Even though I was satisfying my own curiosity I didn't enjoy the process of writing down the conclusion. Why? Because I imposed the structure and tone of "Do you really need WebSockets" on those articles to make them more likely to succeed (at least in my mind).

The problem with "Do you really need WebSockets" is that it's written like an academic paper - it's monotone, uses more complicated language just to sound smart and has a predictable structure to every paragraph. When I read those articles now they sound boring and devoid of personality.

The first article I ever wrote in this blog represents me better than the last five.

Once I realized that I started working on this iteration of the blog.

The first step for me was to get away from Medium as the site had many features I didn't want my blog to have.

The feature I wanted to get rid off the most was "claps" (Medium's version of likes). Comparing something by how many likes it has produces feedback loops like the one I got into after "Do you really need WebSockets". And the number of likes doesn't tell me anything of value - I don't know if the person giving the like thinks this article is better than the others I wrote, I just know that they liked this one. Even if I did know they liked this article more then the rest I still wouldn't know why.

The second feature I didn't want to have were comments. Over the years there were many nice comments, good questions and corrections. But, as with any unmoderated comments section on the Internet, there were ten times more comments from people who clearly didn't read the article and made ungrounded claims calling me out as a liar and dimwit.

At first I tried to be kind, explain their claims as false and correct them. But that took more time than I'd like to invest into responding to comments so I tried to ignore those people - this didn't work. Those negative comments influence new readers which then ask me to answer those commenters. In the end I had to answer all comments, which made me bitter, which made my comments sound bitter, which portrayed me as a bitter person.

Therefore, no more comments. If someone wants to comment something they should start a thread somewhere on the Internet and link to an article on this blog. All links to any article will show up in the backlinks section and anyone jump into any of those threads.

So I developed this web app to re-vitalize the old blog, add new features to it and fix the issues I had with Medium:

A custom design that strains the eyes less and that has no accessibility issues.

The new design of stanko.io

Link previews - because having to open links to see if you want to read what they link to is annoying

Link previews

My own analytics - because that way I can respect my readers' privacy and track backlinks to articles

Backlinks & analytics on stanko.io

Visible RSS and ATOM feed for subscriptions

It's by no means perfect, but it was fun to do and it shows off who I am, the way I think, my personality and my values. And now that I can write without self-imposed restriction I'm having fun again.

GraphQL file upload with Shrine

2019-01-02T00:00:00Z

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The GraphQL logo overlayed over a Japanese shrine

At the moment of writing there is no officially supported way to do file upload through GraphQL. Here is a roundup of all available methods to do file upload through it, their pros and cons.

This post grew out of a request on the Shrine issue tracker — you can find the original issue here. It's still useful for other libraries, but the code examples will only apply to Shrine.

Direct file upload

Handling uploads is a resource intensive task for the server (using up IO and bandwidth). To resolve this, we can upload the file directly from the client to our block-storage server (AWS S3 or DigitalOcean Spaces) by using a presigned URL generated on our server.

This is in contrast to the classic approach where we would first upload the client's file to our server, and then from our server to the block-storage server.

The benefit is that our server doesn't have to process the file (which uses up IO) and it doesn't have to upload the file to the block-storage server (more IO usage). It also solves scaling issues related to distributed file caches and distributed resource management.

The downside to this method is apparent when the uploaded file needs to be processed. To process a file our server needs to get the client's form submission containing the URL of the file, download the file, process it and re-upload it to the block-storage server. This uses up bandwidth and IO but it's necessary.

This method should be satisfactory for most use-cases. If your application doesn't require immediate file processing then the downloading and re-uploading can be done in the background or on a separate machine ( using an AWS lambda). On AWS you can get the best of both world using the Serverless image handler.

Shrine supports this feature out-of-the-box through the presign plugin. To get it running you'll need to add the AWS S3 SDK to your Gemfile, configure a new storage for Shrine, and expose the file presign endpoint — this is explained in detail in the documentation.

# This code is license under the MIT license
# Full text: https://opensource.org/licenses/MIT
# Author Stanko K.R. 

# Usage:
# ```ruby
# user.avatar = S3UrlToShrineObjectConverter.call(
#   arguments[:input][:avatar][:url],
#   name: arguments[:input][:avatar][:filename]
# )

require 'uri'

class S3UrlToShrineObjectConverter
  def self.call(*args)
    new(*args).call
  end

  def initialize(url, name: nil)
    @url = url
    @name = name
  end

  def call
    return unless object&.exists?

    {
      id: id,
      storage: storage,
      metadata: {
        size: object.content_length,
        filename: name || id,
        mime_type: object.content_type
      }
    }
  end

  protected

  attr_reader :url
  attr_reader :name

  private

  def object
    @object ||= bucket.object(uri.path)
  end

  def uri
    @uri ||= URI(url)
  end

  def bucket
    @bucket ||= begin
      s3 = AWS::S3.new({}) # TODO: configure this
      s3.buckets['your_bucket_name'] # TODO: load this from the app's config
    end
  end

  def id
    @id ||= uri.path.split('/', 2).last
  end

  def storage
    @storage ||= uri.path.split('/', 2).first
  end
end

Once the client uploads the file it needs to send some kind of reference to the server — the URL of the file, or an ID, or the path of the file (I usually send the original file name as well as the URL).

mutation SetAvatar($url: String, $filename: String) {
  updateUser(
    id: "some_user_uuid"
    input: {
      avatar: {
        filename: $filename
        url: $url
      }
    }
  ) {
    avatar_url
  }
}

The server uses this reference to build a Shrine object and attach it.

Note that you can use the upload_endpoint plugin instead of the presign_endpoint plugin. I advise against this as it enables anyone to upload any content, at any time, to your block-storage. By presiging we solve the "anyone” and "anytime” issues — we have control over who can request presigned URLs to upload or download files.

Base64 encoding

This method is the simplest, yet it's the worst for real-world applications.

Your client can encode the file's data as Base64 and set it as the value of a mutation's field.

mutation SetAvatar($data: String, $filename: String) {
  updateUser(
    id: "some_user_uuid"
    input: {
      avatar: {
        filename: $filename
        data: $data
      }
    }
  ) {
    avatar_url
  }
}

While on the surface this seems harmless, since the image has to be uploaded to the server in one way or another, this approach has many unwanted side-effects.

#!/usr/bin/ruby
user.avatar_data_uri = arguments[:input][:avatar][:data]
user.save

The biggest issue is memory consumption. Since the image is now part of your request's JSON body it will be parsed as a string, which will drive your memory consumption up. E.g. If you upload ten 5MiB images in one mutation you will have a hash that's at least 50MiB in memory. Storing whole files in memory can and will crash your application.

I advise against using this method. If you wish to implement it, you can do so using Shrine's data_uri plugin. After you add the plugin to your uploader you will need to set the <field>_data_uri of your file in the resolver.

If you decide to use this method be careful to remove the image upload field from your logs, otherwise your log file will contain a copy of all uploaded images.

Multipart upload

Multipart uploads are the standard way of uploading files through HTTP. They are used by most browsers to do file uploads through forms. Since GraphQL is just a JSON request we can pass files alongside our request, but we need logic to interpret them.

I'll be referencing jaydenseric's multipart request specification as it's supported by several client and server implementations at this moment.

mutation UploadGalleryImages {
  uploadGalleryImages(
    galleryID: "some_gallery_id"
    images: [null, null]
  ) {
    id
    filename
    url
  }
}

A typical GraphQL request consists of three fields query, variables and operationName. This spec adds a fourth — map. The map contains indices and JSON pointers, each index represents an image and each pointer represents the location of the ActionDispatch::Http::UploadedFile that holds the corresponding image.

Say we want to upload two images to a gallery using this method, our GraphQL mutation might look like this:

Since the server knows that the images field is an array of File types it will match all null values in the array, in order of appearance, with their index in the map field and dereference their JSON pointer. If we were to inspect the input params hash we would see the following:

# {
#   galleryID: "some_gallery_id",
#   images: [
#     #<:http::uploadedfile:0x000055fb90c6dbd0>, @original_filename="b.txt", @content_type="text/plain", @headers="Content-Disposition: form-data; name=\"0\"; filename=\"b.txt\"\r\nContent-Type: text/plain\r\n">,
#     #<:http::uploadedfile:0x000055fb90c6da18>, @original_filename="c.txt", @content_type="text/plain", @headers="Content-Disposition: form-data; name=\"1\"; filename=\"c.txt\"\r\nContent-Type: text/plain\r\n">
#   ]
# }

gallery = Gallery.find(arguemnts[:gallery_id])

arguments[:images].map do |file|
  # Image has a Shrine uploader attached to `file_data`
  Image.create(file: file, gallery: gallery)
end

From here on you can assign those UploadedFile objects to any of your own objects as usual. No Shrine plugins needed.

Conclusion

My suggestion is to use the direct upload method if possible. It's the only method that doesn't require modifications to the GraphQL protocol, it's well understood, and it's used outside of the GraphQL ecosystem (better interoperability with client libraries).

If direct uploads won't work for you I would take a gamble on multipart uploads.

Base64 encoding of upload files is problematic at best and should be avoided in my opinion. Though there are limited use-cases where it's implementation speed out ways the issues it has. Though, those use-cases are rare.

Function composition >> Ruby

2018-12-25T00:00:00Z

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Ruby's new function composition syntax

Last week Proc#<< and Proc#>> got merged into Ruby 2.6. This opens the door for function composition. Here’s my opinion as to why this is a great leap forward for Ruby and what needs to improve.

Composition vs. inheritance

Ruby is an object-oriented language (among others), meaning it has the concept of an object and the class of an objects — which has the concept of inheritance.

With inheritance we are able to create objects with specialized behavior by inheriting the behavior of a more generalized object (the parent) and specializing it. Let’s build a client for an API that consumes both JSON and XML encoded data. It could look like this.

# Wrapper around a HTTP library
class ApiClient; ... end

# Knows how to decode JSON responses from the API
class JSONApiClient < ApiClient; ... end

# Knows how to decode XML responses from the API
class XMLApiClient < ApiClient; ... end

# Exposes an API's endpoints as methods on an object
class MyAppApiClient
  attr_reader :json_client
  attr_reader :xml_client

  def initialize(api_token)
    @json_client = JSONApiClient.new(api_token)
    @xml_client = XMLApiClient.new(api_token)
  end

  def current_account
    json_client.get('/api/current_account')
  end

  def balance
    xml_client.get('/api/current_account/balance')
  end
end

class MyAppApiClient
  attr_reader :json_client
  attr_reader :xml_client

  def initialize(api_token)
    @json_client = ApiClient.new(API_KEY, JSON.method(:parse).to_proc)
    @xml_client = ApiClient.new(API_KEY, XML.method(:parse).to_proc)
  end

  def current_account
    json_client.get('/api/current_account')
  end

  def balance
    xml_client.get('/api/current_account/balance')
  end
end

array = [1,2,3,4,5]

array.map # => #

array.map { |i| i * 2 } # => [2, 4, 6, 8, 10]

This certainly would get the job done. But this code has two issues that we need to address.

First, both JSONApiClient and XMLApiClient are tightly coupled to their parent and provide little functionality besides the functionality they inherited — meaning that any change to the parent class would probably break the classes inheriting from it.

Second, JSONApiClient and XMLApiClient are too specialized. Inheritance is all about specialization, but having too specialized classes can cause headaches — they are hard to extend (when requirements change) and introduce unnecessary complexities (for a human, it’s hard to remember many different kinds of objects and what they do).

We can address both issues by passing the desired parser as an argument to ApiClient. This is in-line with the open-close principle, as now we can create an ApiClient that can parse any kind of data without needing to subclass it.

We took an existing class/service/object/function and combined it with another to get more specialized behavior. This is the basic idea behind function composition.

While the above example is actually an example of dependency injection, it is a great outline for the things that are possible with function composition in Ruby. Also, note that the above approach is limited to the configurable dependencies we accept through the initializer.

We use composition every day without even noticing it. Enumerable#map uses composition to enable us to transform arrays.

The map method is of limited usefulness on it’s own. But when we combine it with a block (another function) it can transform whole datasets.

`Proc#>>` and `Proc#<<`

Proc#>> and Proc#<< were introduced in Ruby 2.6. They provide a substantial quality-of-life improvement when it comes to composing functions.

Proc#>> is similar to Elixir’s pipeline, but instead of returning a result it returns a proc — a callable object.

In mathematical terms, f(x) >> g(x) is the same as g(f(x)).

Let’s go back to our ApiClient example:

f = -> x { x + 2 }
g = -> x { x * 2 }

# h is the composition of f and g
h = f >> g
# h is the same as -> x { (x + 2) * 2 }

[1, 2, 3].map(&h) # => [6, 8, 10]
# This is exactly the same as
[1, 2, 3].map(&f).map(&g) # => [6, 8, 10]

fetch_transactions =
  ApiClient.new(api_token).method(:get)
  >> JSON.method(:parse)
  >> (-> response { response['data']['transactions'] })

fetch_transaction.call('/api/current_user/transactions')

f = -> x { x + 2 }
g = -> x { x * 2 }

# h is the composition of g and f
h = f << g
# h is the same as -> x { (x * 2) + 2 }

[1, 2, 3].map(&h) # => [4, 6, 8]
# This is exactly the same as
[1, 2, 3].map(&g).map(&f) # => [4, 6, 8]

Notice that we didn’t pass anything to ApiClient, we didn’t need to do dependency injection. This solves the configuration problem we had before. And we are able to create highly specialized functions on-the-fly. The above example creates a function that returns all transaction from an API endpoint.

Note, if you want Elixir style pipelines that return a result check out yield_self or the new alias for it then.

On the other hand, Proc#<< is more in-line with Haskell style composition:

Or, in mathematical terms, f(x) << g(x) is the same as f(g(x)).

Both << and >> are basically the same, which one to use is only subject to your preference.

Function composition is very useful in languages that don’t have the concept of classes and inheritance as it enables you to “inherit” the behavior of a function and extend/specialize its behavior. Like in the following example.

Since we do have inheritance in Ruby this kind of composition is less useful. Yet it gives us the ability to create utility functions on-the-fly thus it encouraging us to create methods/classes that can be extended through the open-close principle / dependency injection, and composition-over-inheritance. In my opinion, this is a big step forward for the language.

Conclusion

As its implemented now, composition sticks out like a sore thumb. The ecosystem needs to grow to accommodate for this feature. Constantly calling #method on a class/module is confusing and distracting — a short hand operator for this purpose would be welcome (I would propose &[Json].parse). The map, reduce and other enumerator methods are hard to compose since they are implemented on an instance of an object — an Enum module exposing them would be nice.

require 'net/http'
require 'uri'
require 'json'

fetch =
  self.method(:URI) \
  >> Net::HTTP.method(:get) \
  >> JSON.method(:parse) \
  >> -> response { response.dig('bpi', 'EUR', 'rate') || '0' } \
  >> -> value { value.gsub(',', '') } \
  >> self.method(:Float)

fetch.call('https://api.coindesk.com/v1/bpi/currentprice.json') # => 3530.6782

If somebody shares these pain points, and agrees with me. I’ve created a draft implementation of those features.

Best image uploader for Rails — Revisited

2018-12-25T00:00:00Z

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Hand holding a plastic Ruby gemstone

Three years ago I wrote about how to choose the right uploader gem for your project. Since the time the original article has been published, all mentioned libraries got updated, one got deprecated, and two new libraries have appeared. I feel it's time to revisit this topic.

Why do we use uploader gems?

Rails handles file-upload natively. When a file gets uploaded to your app it's represented by an UploadedFile object, which is a wrapper around the underlying Tempfile object that contains the data sent to the server.

After we get the file we have multiple options what to do with it:

Cache it — if the file is uploaded through a form we want to temporarily store the file in case the form fails to validate, so that the user doesn't have to re-upload it. This lowers bandwidth consumption and improves UX.
Process it — more often than not people want to create smaller or cropped versions of images that users upload. In the case of PDFs / DOCs we might want to create thumbnails. In general we want to process the data somehow.
Store it — we want to store the uploaded file without name clashes and corruption.
Forward the data to a file storage server — to lower bandwidth to our own servers we usually want to store the files on services like Amazon's S3 or DigitalOcean's Spaces. Perhaps we also want to send the data to a CDN (like CloudFlare) so that the file gets served faster to users all over the world.
Stream the file back to users — if our app also handles serving of the file it would be best if we could stream its contents instead of loading it into memory.
Process it on-the-fly — if we don't know what kind of processing our client's require from the data, we can do it on-the-fly. E.g. different phones can have different resolutions, instead of serving all devices the same sized image a device can request the image best suited for it's display. The image will then be processed to the desired dimensions and passed to the device.

Not to re-invent the wheel, and not to introduce bugs into our codebase, we reach for a gem that will do all or most of those things for us.

ActiveStorage & Shrine

Since I published the previous article two new “mainstream” libraries have appeared. Rails' own ActiveStorage and janko-m's Shrine.

ActiveStorage

ActiveStorage now represents the standard (or at least the Rails way) for handling file-upload and storage on file servers. It integrates seamlessly with ActiveRecord, providing an elegant API to upload, download, delete, and process files.

All file processing is done on the fly. This was a gripe I had with Refile in the original article as it made it unusable without a CDN — to cache processed images. ActiveStorage fixed that by storing all created versions after initial processing. Note that only the web app can create a version of the file as it has to sign the file's URL — which enables it to sanitize and rate-limit clients' requests. And at the time of writing there are some difficulties with connecting ActiveStorage to a CDN.

One feature that is lacking is the ability to create custom file processors. In Rails 5.2 ActiveStorage uses the minimagick gem directly to process images, but in the future it will use the image_processing gem which can potentially expand the list of available options. But, it's currently only able to process files to images (it can create PDF and video previews). There is no way to re-encode a video file, or to e.g. count the number of words in a document. But it offers many macros for working with images and vips support is on it's way.

ActiveStorage advocates for direct uploads (uploading directly to the file server), this abolishes the need for caching.

The biggest issue I've experienced is that it's coupled to ActiveRecord. This forces developers' choice of ORM, and makes potential library migrations more difficult. Today it's not uncommon to see ROM or Sequel in an Rails app.

I'm glad that Rails introduced a standard solution. Out-of-the-box file upload was something that was missing in the framework to make it the perfect one-stop-shop. And, in my opinion, ActiveStorage covers 90% of peoples' file uploading and processing needs. With it, you can add file upload to your Rails app in minutes.

Shrine

DISCLAMER: I know the author of the gem and consider him a friend. Therefore my opinion can be interpreted as biased. I'll try to be as objective as possible.

In the previous article I named Carrierwave the Swiss army knife of Rails uploaders. This time around that honor goes to Shrine.

Shrine takes CarrierWave's idea of uploader classes and refines it further by introducing a Roda's / Sequel's plugin system and cleaning up the API.

The plugin system makes it agnostic to many things like ORMs, frameworks, and other. So-much-so that it can be used with any Rack app.

It comes bundled with many plugins which provide a lot of configuration options and compatibility with many libraries. E.g. both ActiveRecord and Sequel are supported out-of-the-box.

There exists the ability for parallel and background file processing. This not only speeds things up, but makes large file processing (like video encoding) easier to handle.

Of course, there's the ability to do direct uploads, same as ActiveStorage. The difference between the two implementation being the underlying JS library. ActiveStorage rolled their own JS library, while Shrine decided to use Uppy (the most-popular direct-upload library at the time).

Different file storage servers, and caching is supported out-of-the-box.

Metadata, such as the original name, extension, file size, checksum and other arbitrary data can be stored alongside the file. This metadata feature also enables the storing of versions (e.g. different sizes/crops of an image or different encodings of a video).

Something that I've only seen is Shrine is the ability to migrate data around. E.g. there is a plugin that enables you to move your files from one file-store to another. Or to copy data over from one model to another.

The only lacking feature is on-the-fly processing. Though, personally, I have never found that to be the issue. But after ActiveStorage did it I'd like to see the same functionality in Shrine. Note that it's possible to achieve on-the-fly processing if you integrate with Dragonfly or Cloudinary, but there is no out-of-the-box solution.

From my experience, people complain that they have to configure Shrine for Rails. I've never found that to be an issue, but to address those people I'd hope the author creates a wrapper gem to ease the integration with sane defaults for most configuration options.

Paperclip

Sadly, with the advent of ActiveStorage we also saw the deprecation of Paperclip.

Paperclip didn't improve much since the last round-up, but it didn't need to. It always was the simplest and quickest solution, and intended only to attach files and light processing.

Much of what it does, ActiveStorage does better and it's a bundled-in part of Rails. Though, note that some features like validation are still missing in ActiveStorage. It was a good decision to deprecate the project as it couldn't compete with Rail's own solution.

CarrierWave, Refile & Dragonfly

When it comes to CarrierWave, Refile and Dragonfly I have to admit I haven't used either of them in quite the while as ActiveStorage and Shrine completely substituted them in my tool belt.

Those libraries haven't changed much in the last two years.

Conclusion

In my opinion, today you have two choices when it comes to file uploaders in Rails — either ActiveStorage or Shrine.

ActiveStorage is the perfect solution if you just need to store a file, do some light processing and forget about it. I'd guess that this is enough for 90% of the projects that exist.

If your applications requires more advanced processing or storage options, or any kind of custom file processing go with Shrine. Its plugins system makes it easily extendable, and there already exist quite a few plugins out there.

CarrierWave, Refile and Dragonfly are by no means dead. They do have their audiances, but I've found that ActiveStorage or Shrine can do most if not more than those libraries can, so I'd recommend those two over the others for new projects.

If you aren't using Rails, then there is only one real option for you — Shrine.

Licensing software

2018-10-20T00:00:00Z

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

My messy desk

Recently I've started working on a small Ruby library. While I was sketching the architecture of it to I was listening to some lectures from Richard M. Stallman which got me thinking about how I should license my library.

Note: I'm by no means a legal expert. Everything written here is what I've found while researching the issue. Some things may not apply to the region you live in.

Why is licensing important?

Unless you provide a license with your code, the code (legally) can't be redistributed by others. It falls under your personal copyright, and only you are allowed to redistribute it, authorize copies, modify it, etc.

I wasn't aware of this until, one day, I got an email from a complete stranger asking if I could add a license to an old project of mine so that his company could fork and modify it to their liking.

This caught me off-guard, I always thought that public, unlicensed, code is free for the taking. Much like things in the public domain. But, it turns out that code (just like essays, books, poems, songs, …) belong to the person who wrote them unless there is a contract specifying otherwise (e.g. a license or a contract with the company you work for). This same issue was covered by Jeff Antwood back in 2007 with the post "Pick a License, Any License".

For that reason you'll often hear something along the lines of "Experienced developers won't touch unlicensed code because they have no legal right to use it."

How to choose a license for your project?

Choosing the right license four your project can be a daunting task. A license determines which rights you hold/give up as the author and under what conditions others can use your project — you can think of it as your business model, of sorts.

Licenses are often written in a confusing to read, but legally correct, manner which means it takes a couple of readings before you understand the full legal implications of the text. Since I've seen at least 20 different software licenses over the years, and I didn't want to spend the time to read all of them through I've decided to narrow my efforts to the most popular ones and research those.

What to choose?

To find out which licenses are most popular in the community I decided to look at GitHub's data on Google's BigQuery platform. GitHub catalogs all repositories by the license they use. It's far from perfect as some projects have multiple-licenses (e.g. Rust) and some have custom licenses (e.g. Ruby) which will be catalogued as unlicensed, but the data should be indicative of a trend.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Representation of licenses in all GitHub projects as a percentage

The above graph represents the percentage of repositories using a particular license.

To get that data from BigQuery I've used the following query to extract the count of projects using a particular license for C, C++, Elixir, Haskell, Java, JavaScript, Kotlin, PHP, Python, Ruby & Rust to construct the data-set.

SELECT licenses.license AS license, COUNT(licenses.repo_name) AS count
FROM [bigquery-public-data:github_repos.licenses] AS licenses
JOIN [bigquery-public-data:github_repos.languages] AS languages ON licenses.repo_name = languages.repo_name
WHERE languages.language.name = ''
GROUP BY license

It's visible form the graph that the five most popular licenses are (in order) MIT, Apache-2.0, GPLv2, GPLv3 and BSD-3-clauses. It's also visible that there is a relatively small amount of unlicensed, custom-licensed and multi-licensed repositories.

Now that our search has been narrowed down to only five licenses I can read and research each one of them and specify a use-case for each.

MIT

The MIT license is by far the most popular license with about 48% of all repositories on GitHub adopting it. It's also the simplest of the five.

DHH (David Heinemeier Hansson) once translated it nicely as "I don't owe you shit".

The license states that the person using the project has to include the license in any derivative work (e.g. forks — like in the case of Roda which is a fork of Cuba), that they are free to do what ever they like with your code and that you as the author can't be held liable for any bugs or unintended side-effects.

That means that people are free to include your code in their projects (provided the copyright/license notice), you can't be held liable for any kind of unwanted behavior, and that nobody is required to upstream or open-source any bug fixes or changes they make.

This license is extremely permissive making it an excellent choice for libraries, as well as applications. It's mostly catered to open-source projects. Though, in my opinion and depending on your desires, GPL may be better for applications. Note that with MIT any one can make a closed-source derivative work, this makes MIT really popular as it doesn't force the end-user's choice.

There is a gotcha' when working with the MIT license, one exploited by Microsoft's Visual Studio Code. MIT only covers the source code, not the compiled or binary version which enabled developers to add additional clauses to the release version of the app/library.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

License popularity by language, expressed as a percentage; EDIT: "Unlicensed" is actually the Unlicense license

The MIT license is extremely popular in all of the languages I sampled but Java, Kotlin & Haskell where Apache-2.0 and BSD-3-clauses were more popular. And even in those languages, MIT was on second place. I'd attribute it's popularity to it's do-what-you-want approach to licensing.

Apache-2.0

Apache 2.0 is more complicated in legal terms than MIT, but basically sets the same expectations. The license must be provided with each copy of the source. The author(s) can't be held liable in any way, shape or form. It grants the end-user full rights regarding patenting on any contributed code given it doesn't infringe on any patents.

The biggest difference from MIT is that any modifications done to the original project have to be redistributed with a notice that the files have been modified. And a warranty can be applied to the project (e.g. for a fee).

It's quite permissive making it an excellent choice for both libraries and applications. Especially given the patents and warrant clause, which makes it more suitable for commercial use.

Same as with MIT, Apache-2.0 allows end-users to modify your project and redistribute it under minimal terms, without the need to open-source or upstream modification.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

License popularity in Java & Kotlin projects, expressed as a percentage

Interestingly enough, Apache-2.0 is the most popular license for Java and Kotlin projects. I'd guess that because the Apache Foundation predominately uses Java for their projects this drives up the number of Java projects with that license. But' I can't explain exactly why these two languages are the outliers.

GPLv2 & GPLv3

Here I'm going to cover only GPLv3, as v2 and v3 are much the same (yet incompatible) expect for patents and a loop hole.

The GPL family of licenses comes from the Free Software Foundation, an organization started by Richard M. Stallman after an, now famous, incident with a printer that often jammed.

This family of licenses is simultaneously loved & frowned upon by many in the Free/Libre and OpenSource community. It's controversial because it requires the end user to open-source, as well as provide build and installation instructions for their project if they use any GPL licensed code in it. Else, the license is quite similar to Apache-2.0. The author(s) can't be held liable, you can patent your work, redistribute and modify the original work (with a notice).

The point of the license is to guarantee the four basic freedoms of software: 1. The freedom to run the program for any purpose 2. The freedom to study how the program works, and change it to make it do what you wish 3. The freedom to redistribute and make copies 4. The freedom to improve the program, and release your improvements to the public

The GPL licenses are also known as Copyleft licenses because they give permission to the end-user to freely distribute and modify the intellectual property but require that the same rights be preserved in any derivative works. Meaning that any software licensed under it preserves the four freedoms for the end-user and any of their derivative works and their end-users as well.

Some people don't agree with the clause that your code has to be open-sourced. This is a topic for another post. This boils down to the difference between free and open-source software. In a nutshell, free software protects and upholds the 4 basic freedoms of software, while open-source software only makes the software's source publicly accessible. In other words, you can view the code of an open-source, but you can't necessarily use, modify, or redistribute it. While free software allows you to do what you want as long as the result is again free software.

A common misconception is that this license prohibits commercial use. The opposite is true, it encourages it, though you have to change your conception of commercial software. Instead of selling the compiled program, you can refocus on selling support, patches, new versions, etc. The essay "The Magic Cauldron" from the book "The Cathedral & the Bazaar" covers the commercial aspect of free & open-source development in great detail, for anyone interested.

The GPLv3 is best applicable for applications. This doesn't mean that it's bad for libraries! Applications can greatly profit from this license as it encourages upstreaming of fixes and distribution of modifications. Libraries can also enjoy all the benefits that applications can, but, because of it's Copyleft nature allications using such libraries must dynamically load them, which may make some end-users reluctant. It depends on your goal and beliefs whether the GPL is for your library. The FSF covers this decision in this article.

The GPL has a semi-permissive license called LGPLv3. It's much the same as MIT, but with the limitation that prominent attribution has to be given to your project in any combined work, that all modification have to be open-sourced (under some conditions) and clearly marked. It still can force the end-user to open-source the whole combined worked if it's statically linked into the derivative, but it can be dynamically linked without repercussions.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

License popularity in C & C++ projects, expressed as a percentage

BSD-3-clauses

BSD-3-clauses is a permissive license. It's quite similar to the MIT license in the sense that the author(s) hold no liability, the license has to be included in all derivative work and similarly to Apache-2.0 it allows a warranty to be applied.

The biggest difference from the other licenses is that it forbids the end user to promote their project based on the use of the licensed project or it's author(s) (without prior permission).

Note that there exists a -patents version of the license which grants the end user the ability to patent their work, same as Apache 2.0. But this license is somewhat controversial after Facebook added very strict patent clauses to their BSD license in the React project, escalating so far that the project got re-licensed to MIT.

BSD-3-clauses is excellent for both libraries and applications. It's not as verbose or popular as Apache-2.0, yet it's as permissive as MIT, the biggest differentiating factor is it's restriction on marketing.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

License popularity in Haskell projects, expressed as a percentage

It's interesting that out of all the sampled languages that Haskell's most popular license is BSD. A friend of mine indicated to me that this could be caused by the fact that their project generator uses it by default.

Clauses

Software licenses can come with clauses which can change the base license. The most recent example of this is the addition of the Commons Clause to some Redis Labs modules/products.

Clauses are envisioned to fulfill the author's additional wishes or change some behavior of the base license.

Perhaps the most well known example is GPL and LGPL. Where LGPL is basically a clause, and requires, the full GPL.

Conclusion

Picking a license is hard. It boils down to what you want other people to do with your work.

If you really don't care and just want to get your project out there for people to use however they wish, even if they choose to sell basically your project — then MIT is for you.

If you also want to grant your users the ability to make patents off your derivative work, Apache-2.0 is the way to go.

If in addition you want to protect the authors'/contributors' privacy and control how people can use your project to market theirs, then BSD is the choice for you.

If you want to protect the users' freedom or you want your project to forever stay free & open-source. Then the GPL family is your best choice.

I usually use the MIT license only for libraries and GPLv3 for applications, as I don't want to force people to open-source their projects, but after doing this research I think I'll choose LGPL for my Ruby library. In my opinion it's a good compromise between staying free & open-source while still giving the user the ability not to open-source their work.

While reasearching this topic I've found a hand full of useful resources to do quick checks, they can't really replace reading the license text, but they give an insightful overview:

I have also stumbled upon a few hilarious (yet functional) licenses:

The data used to generate the statistics can be found here. It was gathered on the 25th of April, 2018 — it shouldn't be affected by the recent migration of projects from GitHub after the acquisition from Microsoft.

Epilogue

https://opensource.org/licenses/alphabetical — list of all known licenses
https://tldrlegal.com/ — quick bullet-point overview of most licenses
https://choosealicense.com/ — license picking decision tree
BeerWare — If you think the software is good, and you meet the author, you can repay them with beer
Do What the Fuck You Want — the name says it all...
The Unlicense — ironically enough, it's a license that aims to abolish any license and copyright clauses

Do you really need WebSockets?

2018-03-29T00:00:00Z

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The Cloud was made by Fabián Alexis (CC BY-SA 3.0), via Wikimedia Commons

Over the years I've had this conversation a couple of times. This post will explain why we use WebSockets, how they can be used, what alternatives exist and when to use them.

Every time I worked on a project where we had to implement any kind of a "real-time" component, usually a chat or an event feed, the word WebSockets started to circulate. Though, most people use them because they either aren't aware of the alternatives, or they blindly follow other people's examples.

Why WebSockets?

WebSockets enable the server and client to send messages to each other at any time, after a connection is established, without an explicit request by one or the other. This is in contrast to HTTP, which is traditionally associated with the challenge-response principle — where to get data one has to explicitly request it. In more technical terms, WebSockets enable a full-duplex connection between the client and the server.

In a challenge-response system there is no way for clients to know when new data is available for them (except by asking the server periodically — polling or long polling), with Websockets the server can push new data at any time which makes them the better candidate for "real-time" applications.

It's important to note that WebSockets convert their HTTP connection to a WebSocket connection. In other words, a WebSocket connection uses HTTP only to do the initial handshake (for authorization and authentification), after which the TCP connection is utilized to send data via the its own protocol.

Animation of a WebSocket connection being established

WebSockets are a part of the HTML5 spec and they are supported by all modern browsers (meaning, there is a JS API to use them natively in the browser). They provide a mechanism to detect dropped (disconnected) clients and can handle up to a 1024 connections per browser, though they aren't compatible with most load balancers out-of-the-box and have no re-connection handling mechanism.

// Create WebSocket connection.
const socket = new WebSocket('ws://localhost:8080');

// Connection opened
socket.addEventListener('open', function (event) {
    socket.send('Hello Server!');
});

// Listen for messages
socket.addEventListener('message', function (event) {
    console.log('Message from server ', event.data);
});

The most common example for WebSockets is either a chat or push notifications. They can be used for those applications, but present an overkill solution to the problem, since in those applications only the server needs to push data to the clients, and not the other way around — only a half-duplex connection is needed.

In Ruby, there are a few gems that add WebSockets to your web app. The one I've mostly used is Faye, though I've been looking at websocket-ruby lately. Rails supports them out-of-the-box since version 5 through ActionCable.

require 'faye/websocket'

App = lambda do |env|
  if Faye::WebSocket.websocket?(env)
    ws = Faye::WebSocket.new(env)

    ws.on :message do |event|
      ws.send(event.data)
    end

    ws.on :close do |event|
      p [:close, event.code, event.reason]
      ws = nil
    end

    # Return async Rack response
    ws.rack_response

  else
    # Normal HTTP request
    [200, {'Content-Type' => 'text/plain'}, ['Hello']]
  end
end

In my opinion, all of those implementations are more-or-less the same — you can't really go wrong. Note that some gems also require their own JS library (mostly to encode and decode data that's being sent or received).

Server-sent events

From my experience, most people don't know that regular old HTTP provides a mechanism to push data from the server to clients via Server-Sent Events (aka. EventSources).

Server-Sent Events utilize a regular HTTP octet streams, and therefore are limited to the browser's connection pool limit of ~6 concurrent HTTP connections per server. But they provide a standard way of pushing data from the server to the clients over HTTP, which load balancers and proxies understand out-of-the-box. The biggest advantage being that, exactly as WebSockets, they utilize only one TCP connection. The biggest disadvantage is that Server-Sent Events don't provide a mechanism to detect dropped clients until a message is sent.

Animation of a Server-Sent Events connection being established

They are standardized via HTML 5, most HTTP servers support them out-of-the-box, and they are available in most browsers except for Internet Explorer and Edge where they are available through a polyfill.

Rails, Roda and Sinatra support them out-of-the-box. And they have a simple protocol — payloads are just prefixed with one of the following keywords data:, event:, id: and retry:. data: is used to push a payload, event: is optional and indicates the type of data being pushed, id: is also optional and indicates the event's ID, finally retry: instructs the client to change it's connection retry timeout — unlike WebSockets, Server-Sent Events have a reconnect mechanism built-in, though this is a feature that most WebSocket libraries add any way.

# frozen_string_literal: true

class App < Roda
  QUEUES = []

  Thread.new do
    loop do
      sleep(60)
      QUEUES.each { |q| q << { heartbeat: true } }
    end
  end

  plugin :streaming
  plugin :render, engine: 'slim'

  route do |r|
    r.root do
      view('root', layout: false)
    end

    r.on 'messages' do
      r.post do
        name = r.params['name']
        message = r.params['message']
        object = { name: name, message: message }
        QUEUES.each { |q| q << object }
        object.to_json
      end
    end

    r.get 'stream' do
      response['Content-Type'] = 'text/event-stream;charset=UTF-8'
      q = Queue.new
      QUEUES << q
      q << { heartbeat: true }
      stream(loop: true, callback: proc { QUEUES.delete(q) }) do |out|
        loop do
          out << "data: #{q.pop.to_json}\n\n"
        end
      end
    end
  end
end

In the example above, to send data to other clients a regular HTTP POST request is made to the server. A heartbeat is kept to keep the connections alive (WebSockets also do that) and to detect dropped clients (since dropped connections can only be detected when data is pushed to them).

Long polling

Before there were Server-Sent Events people usually resorted to long polling to receive "real-time" data from the server. Not to give the wrong impression, long polling is still used today in some scenarios.

Long polling utilizes regular HTTP requests. When a request is made to the server it responds immediately if there is data that can be served. If no data is available, the server drags out its response while the client waits. If in that time new data becomes available it's served to the client. When the client receives data, or its request times-out, it immediately makes a new request to re-establish the connection.

Animation of a Long Polling connection being established

Long polling's biggest advantage is the fact that it works in every environment, and in every browser. It's, arguably, better for applications with large numbers of concurrent users because it doesn't require a constant TCP connection to the server (it's harder to starve the server of TCP connections since they are periodically released). Though they come with the overhead of having to re-authenticate and re-authorize the client on each request. And the server needs to implement some kind of event aggregation to overcome blackouts between re-connects. There are no JS APIs available for this mechanism, there is no re-connection handling, nor dropped client detection.

If data needs to be sent to the server a regular HTTP POST request is made, the same as with Server-Sent Events.

When it comes to pushing data from the server to clients both WebSockets and Server-Sent Event will do the job. There are some subtle differences and incompatibilities, but there are libraries that solve those issues.

If you don't need to send data to the server in "real-time" (e.g. voice/video chat, multiplayer games, …) go with Server-Sent Events. They are the standard HTTP way of pushing data (like notifications, messages or events) to clients. And they can be added just by implementing an additional endpoint in a controller, lowering the need to refactor the existing code base (in a well structured code base the "implementation cost" of both WS and SSE is the same), and making them somewhat faster to implement (and bring the feature to market).

I don't have experience with long polling on projects with large numbers of concurrent users, so I can't attest the claims I've read about it being the better solution for those kinds of applications. Though, while we were reverse engineering Facebook's Messenger to add PGP to it I noticed that they utilize long polling to fetch messages. On smaller projects I would recommend SSE over Long polling since it's easier to implement on both the server and client side.

Conclusion

If you are already using WebSockets or Long polling, don't go and convert them to Server-Sent Events. All solutions are basically the same when it comes to pushing data from the server to clients.

	WebSockets	Server Sent Events	Long Polling
Number of parallel connections from the browser	1024	~6 per server	~6 per server
Load balancing and proxying	Requires additional configuration	Out-of-the-box	Out-of-the-box
Supported on all browsers	Yes (90%)	No (84%)	Yes (100%)
Dropped client detection	Yes	No	No
Reconnection handling	No	Yes	No

RabbitMQ is more than a Sidekiq replacement

2018-03-07T00:00:00Z

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A rabbit dressed up as the karate kid - a cross of RabbitMQ and Sidekiq logos

I've had gripes with Sidekiq because of which I switched to RabbitMQ. Here are my thoughts and experiences after a year of using it in production.

I got inspired to write this post by the overwhelming response I received for my talk at the local Ruby user group.

Why do we need Sidekiq or RabbitMQ?

"Background job" libraries like Sidekiq are used in situations when the result of a process can be yielded, but further side effects still need to be caused.

An example of this are signup forms. When a user signs up, usually, a confirmation email is sent out to confirm their email, but the email is not crucial to the signup process — the user's account will be created even if the confirmation email fails to get sent.

It's unclear what to do if the email fails to get sent. The intuitive solution would be to return an error, but then the user wouldn't be able to create a new account since their email is already taken. We are penalizing the user for something that isn't their fault, something they can't influence and something that has no effect on the action they wanted to undertake. If we decide to retry the action we end up with the same problem as before if it fails again.

To resolve those issues we offload that work to a "background job" library. In the case of the signup example, the user's account would be created and they would be logged in. The "send email" job would be put in a queue, and would eventually get processed. If the job fails we can retry it as many times as we like, or run custom logic on the raised errors. The user isn't penalized for our mistake.

But do we really need Sidekiq for this? No. The same functionality can be accomplished with the standard Queue class and a Thread.

class App < Roda
  JOBS = Queue.new

  Thread.new do
    loop do
      begin
        job = JOBS.pop
        job.call
      rescue => e
        puts "ERROR: #{e}"
        JOBS << job
      end
    end
  end

  route do |r|
    r.on 'sign_up' do
      r.post do
        email = r.params['email']
        JOBS << proc do
          Mailer::ConfirmationMailer.deliver(email)
        end
      end
    end
  end
end

Why do we use background workers then?

The above approach has many downsides. Not to go too deep down the rabbit hole, I'll focus on debuggability (yes, I made that word up), persistence, scaling and lastly fault tolerance.

The above solution is difficult to debug. There is no clear way to inspect the contents of the queue without littering the code with bindings to pry. If the server is stopped (e.g. to add said bindings to pry) all jobs in the queue are lost, which means that we can't even extract the job that caused the issue to replicate it. To solve those issues Sidekiq uses Redis to store it's jobs. Redis is an in-memory key-value store that can store values as many different data types. Jobs are stored in a list as JSON objects. We can inspect the queue's content by connecting to Redis with redis-cli and inspecting the queue list.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Contents of the default queue in Redis

Using Redis as a central job queue also enables us to scale the number of workers to accommodate higher workloads.

Sidekiq's memory problem

There are two concerns regarding fault tolerance, Redis and the worker process. By default, Redis is a volatile store (data may be lost if the store restarts), though that can be changed by utilizing its RDB and AOF features which in conjunction prevent any data loss. There are some caveats, if RDB (which is enabled by default) is used without AOF, data loss may still occur, and if used in conjunction with AOF may cause performance fluctuations.

When it comes to worker failures, Sidekiq handles most issues regarding handling Ruby errors and retry logic. But if a worker crashes or is killed while processing a job, that job is gone.

Animation of how a job can disappear from Sidekiq in the case of a worker restart/crash

The most common cases for job disappearance are deploys (since the application is killed at that point, though Sidekiq offers rolling restarts with it's Pro plan) and VM crashes caused by faulty gem extensions (as they can't be caught).

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Screenshot of Sidekiq's README about error handeling

If you are a business, or have a $1000 spare, you can resolve this issue by buying a Sidekiq Pro license which ensures that a job gets executed and comes with other niceties. Personally, I find this price tag too steep (especially since it's recurring) for small/personal projects which lead me to look for alternatives.

If we ignore the fault tolerance pay wall, Sidekiq still has a memory consumption issue as it uses Redis for its job queue. Redis, is an in-memory data store, and it has no mechanism to offload stale data to disk. This means that all jobs in the queue are kept in-memory all the time.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Redis memory usage plot around peak usage times

The most common solution to this issue is passing IDs of database records instead of values to the job queue. This reduces Redis' memory consumption, but increases Sidekiq's or Ruby's since each Sidekiq instance needs to initialize your application to get access to your models. This can be solved by writing your workers as lightweight Ruby processes, but now we have the issue of managing models and database access information in two separate applications (your worker and your main application). Another solution is to consume your main app's API, but then we are increasing load on our app instead of off-loading work from it.

How RabbitMQ solves those issues

RabbitMQ is a general purpose message queue. To utilize it in a "background worker" backend scenario we need a library to communicate with it. I would highly recommend Sneakers for this purpose. Sneakers handles worker process creation, management, queue creation, and job enqueuing — everything that Sidekiq does, and offers a syntax that resembles Sidekiq's syntax.

class SneakersLogger
  include Sneakers::Worker

  # Defines the queue and it's options
  from_queue 'loggings'

  def work(log_message)
    Logger.log(log_message)
    # Does magic that will be explaned in the next section
    ack!
  end
end

# ---

class SidekiqLogger
  include Sidekiq::Worker

  def perform(log_message)
    Logger.log(log_message)
  end
end

The biggest difference between the two implementations is the ack! on line 10. That line enables Sneakers and RabbitMQ to guarantee that a job has been processed. This is a feature of RabbitMQ's communication protocol — AMQP. In AMQP a message can be popped from a queue in two modes — ack mode and no-ack mode.

Animation of a consumer failing and succeeding in ACK mode

In ack mode the consumer must specify the maximum amount of time for it to process the message. When the consumer pops a message from the queue it's virtually removed from it, but RabbitMQ still keeps a copy of it. If the consumer fails to send an "ack" signal in the specified time period the message is put back at the front of the queue so that another consumer can process it. If the consumer sends an "ack" signal in the specified time period the message is fully removed from RabbitMQ. In no-ack mode no guarantees are given, no time window has to be specified, and no "ack" signal has to be sent.

Another difference is memory consumption. By default, RabbitMQ keeps as many messages in memory as it can, before it reaches a configurable high water mark. At that point it offloads all eligible messages to disk. Though, that is not true for all queues. RabbitMQ also provides a "lazy queue" which keeps all it's messages on disk if possible — it's useful for passing large messages.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Illustration of RabbitMQ resource allocation per message in a normal and a lazy queue

With those features we can create reliable job queues which can handle larger payloads. Now we can write our workers as lightweight Ruby processes which have no database access, as they can now receive all data needed for them to process a job through the queue without a large performance or memory penalty.

There is still one feature of Sidekiq I haven't mentioned — the UI. Sidekiq's UI is useful for monitoring the health of your jobs, and general throughput.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The Sidekiq UI

RabbitMQ also comes with a powerful UI which provides insight not only into the number of failed and succeed jobs, but also into global and per-queue system resource (disk, RAM, …) usage, the ability to consume and publish messages directly from the UI, consumer and user management, and a login screen… which is premium feature in Sidekiq and costs $1950 per year (and there is a limit of 100 workers for that price).

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

RabbitMQ Management Console

Exchanges

AMQP defines the concept of exchanges. Exchanges can be thought of as routers. When a message is published to an exchange, the exchange determines which queues should the message be delivered to. It's important to note that it's impossible to put a message directly into a queue. Even if a message is published directly to a queue, a temporary exchange will be created to deliver it to the queue.

There are four types of exchanges supported by RabbitMQ.

The most commonly used exchange type is the direct exchange. It directly delivers all messages to a single queue bound to it. It's a 1-on-1 mapping of an exchange to a queue. If applied to a chat application, a direct exchange would deliver messages from a chat room to a single user.

Animation of a direct exchange being utilized in a chat app

Another kind of exchange is the fan-out exchange. They deliver messages to all queues bound to them. It's a 1-to-many mapping of an exchange to multiple queues. In the example of the chat application, a fan-out exchange would be used to send a message to all users.

Animation of a fan-out exchange being utilized in a chat app

Then there are topic exchanges. They deliver messages published to them to a bound queue based on the messages tag/topic and the queue's bound topic. In the example of the chat application, a topic exchange would be used to direct messages to their corresponding user.

Animation of a topic exchange being utilized in a chat app

Finally, there are header exchanges. They are a step up from topic exchanges. Instead of looking at a topic, also called a routing key, they look at the message's headers to determine where a message should be delivered. Messages in RabbitMQ can have additional attributes associated with them — called headers. Headers determine different behaviors when handling messages. E.g. an "x-match" header indicates to the exchange that either any or all headers have to match a value for it to get routed to a queue. There is also the "reply-to" header which indicates where the result of processing a message should be published to.

Utilizing exchanges gives many advantages — exactly once delivery, performance, and ease of deprecation. Utilizing an exchange to deliver your messages is much faster and more reliable than using Ruby to handle that logic. There is also the pragmatic reason of not having code to maintain. The logic is handled by RabbitMQ, you only have to configure it (which can be done through code). Exchange-exchange and exchange-queue bindings can be changed on-the-fly by any client which enables one application to change the behavior of other services. Personally, exchanges have helped me deploy applications with little-to-no downtime and deprecate services without having to change other services.

Special features

RabbitMQ adds it's own magic on top of AMQP. I have already mentioned header exchanges, which are a non-standard AMQP feature. A feature I personally use a lot is "direct reply-to". Direct reply-to is a form of synchronous communication between a producer and a consumer. It enables a producer to publish a message and wait for a consumer to process it and return the result directly to the producer. It's useful when the result of a message is used in further processes. E.g. IoT devices usually log a heartbeat signal to their server to indicate that they are connected and configured correctly. If we have a smart lock, we can process its heartbeat asynchronously since the result isn't really important for the server nor for the device. But a pin check is important and should be handled synchronously to avoid access permission errors caused by stale data.

Animation of utilizing a direct reply-to message

Another feature that I often utilize is "dead lettering". It allows for a message to be re-queued automatically in case it gets rejected from an exchange or queue. This feature is useful for error handling, exponential back-off, scheduled message processing …

Animation of utilizing a dead letter queue

"Alternate exchange" is a useful feature for deprecating services. It specifies to which exchange a message should get sent to in the case that the primary exchange rejects it.

Animation of utilizing an alternate exchange

Then there are "priority queues" and "priority consumers". Priority queue are CS standard priority queues. Meaning that messages in the queue have a priority (ranging from 0 to 255), messages with higher priority get processed first. Which is useful for the same reasons as "direct reply-to", but it's asynchronous. While priority consumers are a form of fail-over. Priority consumers will be served messages if they are active. In the case that all priority consumers are inactive, other consumers will get served messages.

Finally there is "TTL". It specifies how long a message lives. If a message outlives its TTL it's automatically rejected from the queue. Though, this feature comes with a caveat — this rule can only be enforced for the message at the front of the queue. E.g. if you put two messages in a queue the first with a TTL of 300 and the second with a TTL of 100. Both would be in the queue until the one with a TTL 300 expires, because it is in front. The moment it expires, the second message is at the front and automatically expires since it's TTL has passed. This seems harmless, but can cause a lot of problems when combined with dead lettering to achieve e.g. offset delivery.

Animation of TTL messages combined with dead lettering

Plugins

For me, this is the most important feature of RabbitMQ. With plugins you can add any functionality you want to RabbitMQ. The best example of this is the management console which is a plugin, and must be enabled before use.

Through plugins, RabbitMQ supports not only AMQP, but STOMP, MQTT and WebSockets as communication protocols.

Then there is the Federation plugin. It enables RabbitMQ to run several isolated clusters or instances which can communicate with one-another. It is similar to the way that Mastodon works. All users, no matter which server they signed up to, can communicate with one-another. Federations are useful for handling large workloads. E.g. if you handle logs from a lot of different machines through RabbitMQ, that can be handled by one federated cluster, while everything else is handled by another federated cluster. That way you can scale those two cluster independently depending on their workload, and you avoid the noisy neighbor problem (when a highly taxed service slows the whole system down).

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Example RabbitMQ Federation for an IoT application

Conclusion

Replacing Sidekiq with RabbitMQ provides many advantages when it comes to debuggability, scaling, fault tolerance and memory consumption. It supports multiple industry-standard message queue protocols and can be used as a drop in replacement for other "background worker" libraries.

If you need a queue that guarantees job execution and persistence, go with RabbitMQ instead of Sidekiq. There are some features that are missing in Rabbit, like cron jobs and unique jobs, but they can be added by the clients. RabbitMQ offers a plethora of features which, if not useful at first, will become useful later as they will help grow a monolith to a services oriented architecture.

To get started take a look at projects like Sneakers (background jobs) and Bunny (AMQP client), read through the basic concepts page, and lastly there is the manual. If you are using Ruby on Rails, Sneakers integrates with ActiveJob which eases the transition.

Sidekiq isn't useless! If your project doesn't require execution or persistence guarantees, or if you hold a Sidekiq Pro license, I would recommend you stick with it for the time being. While I disagree with hiding essential features (like guaranteed execution, and rolling restarts) behind a paywall, a Pro license offers support and additional business oriented features which you won't get with a self-hosted RabbitMQ instance.

Supercharging services architectures with RabbitMQ

2018-03-01T00:00:00Z

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A rabbit with a tachometer in it's silhouette

When I first started using RabbitMQ I didn't understand its usefulness beyond a job queue, but it's helped me to grow and manage services architectures without headaches.

Services Architecture

In web development, a services architecture describes a single application that consists of multiple, smaller, highly specialized, loosely coupled applications that each independently, or in tandem, solve one part of the business domain.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

An example of a services architecture app

This architecture allows for parallel development of features (multiple teams can work on different services, each implementing one part of the business domain), continuous delivery of large applications without downtime (the application, as a whole, should partially work even with one service down), and the freedom to experiment and innovate on the tech stack (a service can be implemented in any technology, framework or language).

A services architecture introduces a new challenge — working with a distributed system.

Problems

The most common problem is state management or data consistency. RabbitMQ doesn't tackle this problem at all. Your application now consists of multiple smaller applications, each of which can have their own state and/or database — on which another service can rely on (e.g. a mailer service needs a user's email from an authentication service). This causes consistency issues. There are many available solutions to this problem, like CQRS, but this topic won't be covered here.

The other common problem is service discovery. You need to have a way to contact the services that compose your application — keep track of their IPs, URIs, and health. This can get out of hand if you have many services that dynamically (horizontally) scale. Traditionally this issue is solved by utilizing load balancers and service registries (like Apache Zookeeper and Consul), but RabbitMQ can partially solve this problem for us.

Lastly there is inter-process communication, or sending messages between your services. RabbitMQ, being a message queue, solves this problem too.

RabbitMQ

As the name implies, it's a message queue. Traditionally, in computer science message queues are used for inter-process communication, which can be asynchronous or synchronous. It uses AMPQ for communicating with clients — it's a wire-level binary protocol that provides mechanisms to ensure message delivery and consumption.

I was drawn to RabbitMQ because of two problems I've had while working on a project, which other message queue services didn't solve.

Some message queues don't have job execution guarantees, e.g. if a message is removed from a queue by a worker it needs to be processed no matter if the job database restarts or if the worker fails. In RabbitMQ this feature is guaranteed by AMPQ. The protocol specifies that, if desired, a message will be removed from a queue only after the consumer acknowledges it has been processed. If a consumer fails to do so in a given time frame, the message is given to another consumer.

The other problem I had with other queues was memory consumption. Some message queues keep the whole queue in memory which can bring even powerful machines to a crawl if the messages are too big or if there is too many of them. In RabbitMQ messages are kept in memory until a (configurable) threshold is reached at which point they will be written to disk. This is not true for all messages — persistent messages (those in durable queues) will be written to disk and kept in-memory as soon as they are enqueued, and there are also "lazy queues" which try to keep everything on disk and nothing in-memory (if possible).

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Queue statistics

RPC / Pub-Sub

Most people I've talked to aren't familiar with RabbitMQ's "direct reply-to" feature. It enables synchronous inter-process communication — you can think of it as an RPC or a Pub-Sub interface. With it, any client can send a message to any queue and get a direct response from any consumer processing messages in that queue. The same rules of memory usage and persistence as described before apply, but message loss protection is not guaranteed with this mechanism.

In my opinion, this method of inter-process communication is better than building your services as HTTP servers. With HTTP servers you re-implement logic that is already present in message queues — e.g. success and failure responses, timeouts, and routing. Oftentimes HTTP servers come with unnecessary overhead for a services architecture like some middleware, session storage and encryption mechanisms which can slow the whole application down. With AMPQ and RabbitMQ you only need to implement your business and serialization logic (in what format will the messages be written to the queue).

Exchanges

In AMPQ, messages aren't published directly to queues, but to exchanges. Exchanges are routers (or post offices) that determine which queues (more than one!) should receive a message.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Topic exchange example

RabbitMQ supports four types of exchanges — direct, fan-out, topic and headers.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Direct exchange example

Direct exchanges deliver messages directly to a single queue, e.g. when you don't want to use exchanges and just want to deliver a message to a queue. For example, if each chat room of an application is represented by a queue, a Ruby Chat Exchange would deliver messages only to the Ruby chat queue.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Fan-out exchange example

Fan-out exchanges deliver messages to all queues bound to them. You can think of them as groups, a message sent to the group will be delivered to all queues in the group. With the previous chat example, a fan-out exchange can be imagined as a broadcast to select channels. E.g. if we want to send a messsage to all chats of awesome languages we can publish a messsage in the Awesome languages exchange which would deliver those messages to both Ruby and Rust chats.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Topic exchange example

Topic exchanges deliver a message to queues tagged with a topic (the above image is an example of such an exchange). E.g. if a message has a routing key (tag) of "Fedex" and it's delivered to the "Shipping" exchange it will be delivered to the "Fedex" queue. Or, with our chat example, a message tagged with text would get delivered to the Chat queue, while a message tagged video would get delivered to the video queue.

Lastly, header exchanges are an continuation on the idea of topic exchanges, they can take different properties of the message being delivered to determine where it should be delivered.

Exchanges come with a multitude of features — dead-lettering (if a message isn't acknowledged or was rejected it gets sent to another exchange), alternate exchanges (a client can specify an exchange to which a message gets routed if the primary exchange rejects it), priority consumers (the ability to specify which consumers to prefer, e.g. consumers on more powerful machines), priority queues (messages can be assigned a priority, higher priority messages get processed first), TTLs (specifies how long a message or queue "lives", though this feature has a caveat — please refer to the manual before using it), and others.

They are not only useful for offloading message delivery logic from your services off to RabbitMQ, but also serve as a way to deprecate services, and reduce downtime. If a service is to be deprecated, its messages can be routed to another service that knows how to process them, or that returns errors interpretable by other services as responses. If a new version of a service is about to be deployed an exchange can be configured to deliver messages simultaneously to the old service's and the new service's queues. Or, when the new service is up, all incoming messages can be redirected to it.

Service discovery

Since RabbitMQ supports multiple producer and multiple consumer queues, exchanges can be utilized as a form of service discovery. Each service knows its name and the names of the services it depends on, therefore it only has to publish messages on their exchanges (or fail if an exchange doesn't exist), and listen for messages on its queue.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

List of available queues

For small services architectures this kind of service discovery is sufficient. All services will communicate through RabbitMQ, and therefore we are able to keep track of them. But sometimes the need arises to communicate with other applications that don't use AMPQ, or keep track of applications in a cluster (e.g. for databases, or RabbitMQ instances) — this kind of service discovery is not suitable for those use-cases and should be handled with the like of Consul or Zookeeper.

Plugins

Personally, I find this to be the "killer feature" of RabbitMQ. If RabbitMQ is missing any kind of functionality you desire it can be added. Out-of-the-box it comes with quite a few useful plugins.

The one plugin I use most often, and the first plugin I introduce people to is the Management plugin. It gives RabbitMQ a full user interface through which you can configure and monitor individual exchanges and queues, monitor system performance, memory and disk usage.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Overview panel in the Management plugin

Through plugins RabbitMQ can support competing protocols to AMPQ — like MQTT, STOMP and WebSockets.

And while clustering is supported, the Federation plugin brings it to a new level. It enables message passing between brokers without clustering (useful since RabbitMQ requires IP addresses of the other instances, instead of URIs, when clustered). E.g. multiple clusters of RabbitMQ can exchange messages between each other. An useful analogy would be Mastodon's and Diaspora's federations which allow users on different instances to communicate as if they were on the same instance. It also comes with a UI that's accessible through the Management plugin.

Lastly, there is a feature called "firehose" that logs all internal messages of RabbitMQ. It's extremely useful for debugging plugin behavior, exchange configuration and even just for logging.

Conclusion

RabbitMQ has helped me manage and scale my services architectures by utilizing plugins, exchanges, queues and the RPC interface. It's become an essential tool for building services.

My biggest issue now is my dependency on it — it's become the backbone of my services architectures. This introduced a large single point of failure, but it's manageable with the Federation plugin and clustering.

If you are struggling with the limitations of your queue service, clustering or inter-process communication — try RabbitMQ. It excels at all those tasks and brings many utilities that can prove helpful for managing your application.

If you are struggling to introduce RabbitMQ to your project. Try it as a simple queue first, and then slowly move more and more services to it

Tips to improve your tests

2016-05-23T00:00:00Z

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

A shoe stuck in the railing of a bridge in Stockholm

Peoples' test suites often get out of hand. Having thousands of tests that run for hours on some CI should be a thing of the past. There doesn't exist code that's too hard to test or code that can't fail. You shouldn't waste your time waiting for your test suite, or chasing some bug you are unable to pinpoint. Writing a test suite that covers 100% of your code base isn't hard. But to do it right is extremely hard.

As I said earlier, all code is 'testable'. There only exists the age-old question of "should I write code that is easy to test?". By now, this discussion has turned into a 'religious war' of sorts. There is no definitive answer to this question.

Writing 'testable' code

You shouldn't write code with your test suite in mind. Write code that makes sense. Write code that is easy to read, and easy to understand. Most of all, write code that is maintainable. If you are having problems coming up with a test suite for the code you wrote then go back and evaluate if your code is as good as it can be. More often than not you will find a way to refactor the code so that it makes more sense, to make it more readable, or to make it maintainable.

You can think of this as a positive feedback loop. Your code becomes easier to test the 'better' code you write. Better in the sense that it's maintainable, readable and sensible. Implementing one pattern (or good practice) will lead you to use other related patterns. And in the end, you will have a solid code base with an excellent test suite.

There are exceptions to this rule. Sometimes, 'good' code is hard to test. That's why you should always write code without your test suite in mind.

Knowing that your application, as a whole, returns expected results for certain scenarios isn't that reassuring. You can never cover all possible combinations of states in an application.

Therefore, it's better to test each module of your application by itself. Rather said — unit test your application. That way you can cover more, if not all possible states. And thus, be confident that your application will do what you expect from it. It also makes refactoring easier as you can detect if you changed an output without intention.

Write more unit tests

Unit tests are generally faster than integration test. Thus, they are cheaper to run. You can have thousands of unit tests that run in a couple of seconds and be assured that your code does what you expect from it. While running the same amount of integration tests might take hours. And they wouldn't tell you if your application performs as expected in all situations.

You should write integrations tests! They assure that all modules of your application work well together. Hence the name — integration tests. Each feature (code execution path) should have at least one integration test. But, there is no rule to determine how many integration tests you should have. My rule of thumb is that, after you run only your integration tests, you should have code coverage greater than zero on each module.

It's only important that your integration tests reassure that the modules of which your application is made of can interact together without a fault.

Stubs and mocks

I often see people letting their 'unit' test call code from a module that isn't their module under test. That effectively turns a unit into an integration test.

The basic idea of stubs is to decouple modules from one another. Stubs, as the name implies, stub out the behaviour of method calls. You can think of stubs as predefined responses to method calls. Stubbing a call to a method never executes the method's code, it just returns the desired value then and there. This is an incredibly powerful tool! Not only can you decouple modules this way, but you can control your program's flow. By stubbing certain methods you can simulate success and failure conditions without the need to craft a complex data set that triggers the same behavior.

Stubs can't assure that a method got called, which is important if that call does something 'mission critical'. Mocks were invented for this kind of situations. Mocks are identical to stubs, but they expect that the mocked method will get called at least once. If the method doesn't get called the test explicitly fails.

Using these two tools you can test all possible scenarios without the need for complex data sets and repercussion checking. This is the first step to making your tests run blazing fast!

Asserting vs. mocking

The question arises of when to assert a value and when to mock a call. We can differentiate two kinds of method calls — queries and commands. Queries are calls that return a value while commands are calls that change the state of the system. For instance, reading a file is a query as it returns the contents of the file. While writing a file is a command as it creates a file with a given content, thus changing the state of the file system.

We also differentiate incoming and outgoing calls. Outgoing calls are calls that your module under test makes to other modules. While incoming calls are calls made to the module under test.

The returned value of outgoing queries should not be asserted. These values will be asserted in the respective module's unit test. Outgoing commands should be mocked. We rely on other module's commands to work. It's only important that we know that they got called. By knowing this, we can stub out the behavior of other modules to simulate the command's repercussions without causing an artificial success state.

The returned value of incoming queries should be asserted. We need to make sure that given an input our method returns an expected result. Incoming commands' repercussions should be asserted for the same reason incoming queries should be asserted.

	Query	Command
Incoming	assert return value	assert side effect
Outgoing	do not assert	assert message sent

There also exist methods that are both queries and commands. A combination of the above rules should be applied when testing them.

Write tests first?

If you know what your module should do, then by all means you should write a test suite first. A test suite should always come first. That way you can ensure that the code you wrote complies to expectations.

There are exceptions to this rule. If you are unsure about the final architecture of the code or think that the code will drastically change during development. Then it's better to write a test suite after the fact.

Conclusion

No matter how you test your software you can never assure that no bugs are present. The methods outlined here are tools to help you write a better test suite. They should be food for thought, not hard rules to follow.

Testing shows the presence, not the absence of bugs

— Edsger W. Dijkstra

Having unit, integration tests, mocks, and spies gives you the assurance that your code does what you expect it to do. Being able to run your test suite fast enables you to iterate quickly. These two together, speed and confidence, enable you to write software at an incredible pace thus giving you time to improve your methods and skills. Resulting in incredible software.

Hacking privacy into Facebook’s Messenger in 24 hours

2016-04-25T00:00:00Z

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

The Copenhagen town hall building

Hackathons are great. When a friend of mine asked me if I wanted to go with him to Copenhacks I had no idea that we would spend 24 hours reverse engineering Facebook's Messenger, let alone win first place.

The journey to Copenhagen

We usually go to a lot of hackathons and coding competitions, but we never went to a hackathon outside Croatia. These are usually either too expensive or at an inconvenient date for us. When we heard of Copenhacks we decided to go, as all of us were free at the time and wanted to see what hackathons looked like in other countries. Hackathons are also a good opportunity to meet recruiters or get internships at one of the sponsors. Finally, let's not forget all the swag that gets handed out!

Copenhacks was a new thing, so we didn't know what to expect. The idea for such a competition came around mid 2015 by a student at the local technical university in the beautiful city of Copenhagen, Denmark. His idea was to create a competition where people could come and build amazing stuff, learn new things and meet fellow hackers from around the world. That's quite different from the other hackathons I went to. Usually the main sponsoring company gives out a specification for an app and then you have a set amount of time to implement it. This was completely different. We could do whatever we wanted. This was also a problem because many questions arose, like 'what could we do?', 'what would impress the judges?', 'what did we have time to implement in 24 hours?'. We had several ideas but none of them felt right. We knew we wanted to make something fun and open-source, so all commercial projects fell out of the picture.

Our flight to Copenhagen was getting closer and closer and we couldn't settle for an idea. The last night before the flight all of us got into a call and started brainstorming. As all of us are cryptography enthusiasts (and perhaps a bit paranoid) and all of us use Messenger all the time. We decided to combine the two together just because it would be convenient and secure. Thus the idea for MessengerPG was born!

MessengerPG should be a browser extension that unobtrusively adds client-side message signing, encryption and decryption with PGP. The cryptography should be done using the local GPG installation. Messenger would only be used as means to transfer the messages from client to client. The extension would collect the public PGP keys for all chat participants from their Facebook profiles. And, yes, you can add your public PGP key to your Facebook profile. The sent messages can only be decrypted with the private keys of each individual participant of the conversation. Everybody else sees only seemingly random strings of letters — this includes Facebook.

More importantly, our extension would be a platform on top of which anybody could build their own extension to Messenger.

The hackathon

My friend once said 'Hackathons are an interesting social experiment'. That's perhaps the shortest way to say it. Hackathons give quite a small amount of time to create something complicated, which causes a lot of stress and leads to sleep deprivation. That's when people start getting into arguments and fights. Then the whole experience turns into a delicate dance of not stepping on each other's toes.

Every time we started getting angry at each other we took a walk, grabbed a soda, played some table-tennis or went to somebody's desk and tried to help them with their problem. This helped, at least me, a lot. Not only did I get to meet people but I saw all the brilliant ideas that others came up with.

Us playing table-tennis at 3 am

Another good piece of advice is to celebrate each victory no matter how small it is as it helps to keep your spirits up. I can recall the moment when we finally managed to intercept incoming and outgoing messages. We got up and started cheering. Each time we managed to solve a problem, we gave each other a high-five.

But the key to getting the best possible product in the smallest amount of time is planning and organization. An hour before the start of the competition we divided our responsibilities.

Marko Božac would be making the presentation website. Luka Strižić would be making an interface between the local GPG installation and our extension. Petar Šegina and I would reverse engineer Messenger and find a way to inject our code. That plan worked out quite well for us, as all of us managed to get three hours of sleep, several meals and we even finished with half an hour to spare.

Reverse engineering Messenger

Finding an entry vector in JS applications usually isn't all that hard because you can read their source and manipulate all objects as they live in your browser. This was a bit harder. Messenger's source is split into several minified files which makes them hard to understand and it's source is bundled with React which adds a lot of code to sift through.

It was obvious that Messenger needs to send and receive data from a server so we opened up Chrome's developer tools and started to monitor incoming and outgoing network traffic. There we noticed that each time you sent a message a request to send_messages.php was being made. This was our entry point. We created an XHR breakpoint with the URL and sent a message. This triggered the breakpoint and showed us the call-stack which we then traversed until we found the getValue function that read text entered into the input field. This was the function we needed to monkey patch to call another function that would process the message before it was sent.

We needed to find a way to get hold of a reference to the object which made the call to getValue and monkey patch it. The window object doesn't have a reference to Messenger's main process but it has a reference to the module loader __d which we then patched to give us a reference to the object we needed.

Receiving messages was a different ball game. We couldn't simply listen for incoming messages and decrypt them before they were passed on to the rendering logic because then only newly received messages would get decrypted. After a page refresh all messages would appear encrypted again. We decided to monkey patch React's render method, as we were sure that it would get called for each and every message that needed to be displayed on the screen. We used the same code injection method as before. The code we added waits for a render call to create a object with a message's CSS class. Then it tries to find a PGP encrypted and signed message and decrypts it.

This all sounds easy enough, but it was a really complicated and frustrating task. We didn't know what Messenger's architecture looked like so we had to figure out how all the things work together. We spent a lot of time trying different approaches and discussing possible solutions. Perhaps the best metric to show how complicated the reverse engineering process was is the fact that two engineers wrote a total of 10 (usable) lines of code per hour.

Many compromises have been made during the development process, some due to time and some due to technical constraints. In order to access the local GPG keychain we built a simple Node.js application that allows us to sign, encrypt and decrypt messages through a simple HTTP API. This led to an unexpected problem with CSP because Facebook prohibits any outgoing communication to non-Facebook approved domains. We tried to use unsafe XMLHTTPRequests but ultimately failed, so a hack was in order. We registered pgp.messenger.com in the local hosts file, and pointed it to 127.0.0.1. This meant that our local API was now located on a CSP trusted domain. However, we still needed to access the API over SSL (as the CSP required https://), so we created a proxy server to our GPG API server, running with locally generated SSL certificates. Getting the browser to trust the generated SSL certificate, being the last hurdle, was not much of a problem.

= 768px) 1040px, 100vw"> = 768px) 1040px, 100vw">

Warning, hack in progress!

I think we all felt a great relief when we implemented the last feature with half an hour to spare. But the relief was short-lived. Soon the presentations started and to the stage came team after team of really talented people. One particular team found a way to misuse Android's accessibility features to create a system wide chat-bot. A different team created a chat-bot that understood natural language and helped sort out arguments between people. Others made games that looked really polished. And then came we, with our short presentation which showed a few messages being exchanged. Plain text on the left, encrypted and signed on the right. Packaged in with a short talk about how we managed to hook in to Messenger and that this can be used not only for encryption but as a platform for all kinds of extensions.

The extension in action — With the extension (left) and without (right)

After the presentations were over none of us believed that we would win anything. But that didn't matter! We created something really cool, traveled together, had an excellent weekend, drank a couple of beers and had fun in general. For us, no matter if we won any prize or not, this was a success. This was what a hackathon should be about!

The anticipation

After a while the room filled with people again, but this time for the prizes to be handed out. One prize was handed out after the other, but our name was never mentioned. Then came the prizes for the top three teams. By this point most of us stopped listening and only waited for the ceremony to be over as we desperately needed sleep. Before the first prize was given out there was a short inspiring speech by the organisers. It explained that they picked first place based on the hacker spirit, problem complexity and general usefulness of the project. Then they called our name. We were in shock. No one expected this to happen. Everybody's faces were white. When it finally hit us that we won, we rushed to the stage to pick up our prize. And so the whole experience was over.

This hackathon was something I will remember, not just because we did something awesome, but because it was really fun.

The organisers did an outstanding job, the food was great, the venue was even better. Thanks to all the sponsors for all the awesome lectures (and all the cool swag). Most of all, thanks to my team which made Copenhagen an experience to remember.

If you are thinking about going to such an event then do it. Don't hesitate. It doesn't matter if you win a prize or not. At the end of the day you will either create something awesome or learn a valuable lesson. Similarly, if you are thinking of organising a hackathon, don't hesitate. You will change somebody's life forever and meet really cool people in the process.

Conclusion — Privacy is hard

If you want to help us make MessengerPG a thing or to build your own extension on top of Messenger, take a look at our Github page. Fork us. Look at the code. Change something. And feel free to open up a pull request. We will gladly take all the help we can get.

Finally, a word of caution. There is currently no way to guarantee that the code you audit will be the code you run next time you open Messenger. This allows malicious servers to serve targeted users with different versions of the code. So, while 99% of users may get clean, non-malicious code, the interesting 1% can get served with code endangering their privacy. This is also a problem with closed-source messaging services which offer end-to-end encryption which cannot be properly audited. Even if the messages being exchanged are encrypted end-to-end, what's stopping the proprietary code from collecting the messages you exchange and exposing them over a different channel?

Privacy is hard. It is not something we should take for granted, but something we should actively try to achieve and protect.