tag:stanko.io,2005:/articles/atomStanko Krtalic Rusendic2024-02-26T07:52:25Ztag:stanko.io,2005:Article/12016-04-25T00:00:00Z2023-11-14T16:28:04ZHacking privacy into Facebook’s Messenger in 24 hours<div class="trix-content">
<div>
<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI4Mj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--a8ee9723390891beb506754c650848217fdd3567" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBaG9CIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--fb9b085a84199807c3d8aeead5201b0f3ff98183/952-NqgY3pKez3-H7I-7EA.jpeg" filename="952-NqgY3pKez3-H7I-7EA.jpeg" filesize="1473096" width="4000" height="2250" previewable="true" caption="The Copenhagen town hall building">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjgyLCJwdXIiOiJibG9iX2lkIn19--0f0881535317aa43cea4356524644ee9ba9be7fd/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/952-NqgY3pKez3-H7I-7EA.jpeg">
<figcaption class="attachment__caption">
The Copenhagen town hall building
</figcaption>
</figure></action-text-attachment>Hackathons are great. When a friend of mine asked me if I wanted to go with him to <a href="http://copenhacks.com/">Copenhacks</a> I had no idea that we would spend 24 hours reverse engineering <a href="https://www.messenger.com/">Facebook's Messenger</a>, let alone win first place.</div><h2>The journey to Copenhagen</h2><div>We usually go to a lot of hackathons and coding competitions, but we never went to a hackathon outside Croatia. These are usually either too expensive or at an inconvenient date for us. When we heard of Copenhacks we decided to go, as all of us were free at the time and wanted to see what hackathons looked like in other countries. Hackathons are also a good opportunity to meet recruiters or get internships at one of the sponsors. Finally, let's not forget all the swag that gets handed out!<br><br>Copenhacks was a new thing, so we didn't know what to expect. The idea for such a competition came around mid 2015 by a student at the local technical university in the beautiful city of Copenhagen, Denmark. His idea was to create a competition where people could come and build amazing stuff, learn new things and meet fellow hackers from around the world. That's quite different from the other hackathons I went to. Usually the main sponsoring company gives out a specification for an app and then you have a set amount of time to implement it. This was completely different. We could do whatever we wanted. This was also a problem because many questions arose, like <em>'what could we do?'</em>, <em>'what would impress the judges?'</em>, <em>'what did we have time to implement in 24 hours?'</em>. We had several ideas but none of them felt right. We knew we wanted to make something fun and open-source, so all commercial projects fell out of the picture.<br><br>Our flight to Copenhagen was getting closer and closer and we couldn't settle for an idea. The last night before the flight all of us got into a call and started brainstorming. As all of us are cryptography enthusiasts (and perhaps a bit paranoid) and all of us use Messenger all the time. We decided to combine the two together just because it would be convenient and secure. Thus the idea for MessengerPG was born!<br><br>MessengerPG should be a browser extension that unobtrusively adds client-side message signing, encryption and decryption with PGP. The cryptography should be done using the local GPG installation. Messenger would only be used as means to transfer the messages from client to client. The extension would collect the public PGP keys for all chat participants from their Facebook profiles. And, yes, <a href="https://www.facebook.com/notes/protect-the-graph/securing-email-communications-from-facebook/1611941762379302/">you can add your public PGP key to your Facebook profile</a>. The sent messages can only be decrypted with the private keys of each individual participant of the conversation. Everybody else sees only seemingly random strings of letters — this includes Facebook.<br><br>More importantly, our extension would be a platform on top of which anybody could build their own extension to Messenger.</div><h2>The hackathon</h2><div>My friend once said <em>'Hackathons are an interesting social experiment'</em>. That's perhaps the shortest way to say it. Hackathons give quite a small amount of time to create something complicated, which causes a lot of stress and leads to sleep deprivation. That's when people start getting into arguments and fights. Then the whole experience turns into a delicate dance of not stepping on each other's toes.<br><br>Every time we started getting angry at each other we took a walk, grabbed a soda, played some table-tennis or went to somebody's desk and tried to help them with their problem. This helped, at least me, a lot. Not only did I get to meet people but I saw all the brilliant ideas that others came up with.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI4MD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--f86123a0fd03f64387951627ec36fc0bfa786821" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBaGdCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--27f7c596a019751a86e0e445eec66111e2556619/wCSUEbT71HObqp90DNlDHA.gif" filename="wCSUEbT71HObqp90DNlDHA.gif" filesize="3346613" width="366" height="206" previewable="true" caption="Us playing table-tennis at 3 am">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjgwLCJwdXIiOiJibG9iX2lkIn19--21e0e63119cbc43bc930ba2420de032bd23681a8/wCSUEbT71HObqp90DNlDHA.gif">
<figcaption class="attachment__caption">
Us playing table-tennis at 3 am
</figcaption>
</figure></action-text-attachment>Another good piece of advice is to celebrate each victory no matter how small it is as it helps to keep your spirits up. I can recall the moment when we finally managed to intercept incoming and outgoing messages. We got up and started cheering. Each time we managed to solve a problem, we gave each other a high-five.<br><br>But the key to getting the best possible product in the smallest amount of time is planning and organization. An hour before the start of the competition we divided our responsibilities.<br><br><a href="https://markobozac.com/">Marko Božac</a> would be making the presentation website. <a href="https://luka.strizic.info/">Luka Strižić</a> would be making an interface between the local GPG installation and our extension. <a href="https://psegina.com/">Petar Šegina</a> and I would reverse engineer Messenger and find a way to inject our code. That plan worked out quite well for us, as all of us managed to get three hours of sleep, several meals and we even finished with half an hour to spare.</div><h2>Reverse engineering Messenger</h2><div>Finding an entry vector in JS applications usually isn't all that hard because you can read their source and manipulate all objects as they live in your browser. This was a bit harder. Messenger's source is split into several minified files which makes them hard to understand and it's source is bundled with <a href="https://facebook.github.io/react/">React</a> which adds a lot of code to sift through.<br><br>It was obvious that Messenger needs to send and receive data from a server so we opened up Chrome's developer tools and started to monitor incoming and outgoing network traffic. There we noticed that each time you sent a message a request to <strong>send_messages.php</strong> was being made. This was our entry point. We created an XHR breakpoint with the URL and sent a message. This triggered the breakpoint and showed us the call-stack which we then traversed until we found the <strong>getValue</strong> function that read text entered into the input field. This was the function we needed to monkey patch to call another function that would process the message before it was sent.<br><br>We needed to find a way to get hold of a reference to the object which made the call to <strong>getValue</strong> and monkey patch it. The window object doesn't have a reference to Messenger's main process but it has a reference to the module loader <strong>__d</strong> which we then patched to give us a reference to the object we needed.<br><br>Receiving messages was a different ball game. We couldn't simply listen for incoming messages and decrypt them before they were passed on to the rendering logic because then only newly received messages would get decrypted. After a page refresh all messages would appear encrypted again. We decided to monkey patch React's render method, as we were sure that it would get called for each and every message that needed to be displayed on the screen. We used the same code injection method as before. The code we added waits for a render call to create a object with a message's CSS class. Then it tries to find a PGP encrypted and signed message and decrypts it.<br><br>This all sounds easy enough, but it was a really complicated and frustrating task. We didn't know what Messenger's architecture looked like so we had to figure out how all the things work together. We spent a lot of time trying different approaches and discussing possible solutions. Perhaps the best metric to show how complicated the reverse engineering process was is the fact that two engineers wrote a total of 10 (usable) lines of code per hour.<br><br>Many compromises have been made during the development process, some due to time and some due to technical constraints. In order to access the local GPG keychain we built a simple Node.js application that allows us to sign, encrypt and decrypt messages through a simple HTTP API. This led to an unexpected problem with <a href="https://developer.mozilla.org/en-US/docs/Web/Security/CSP">CSP</a> because Facebook prohibits any outgoing communication to non-Facebook approved domains. We tried to use <a href="https://wiki.greasespot.net/GM_xmlhttpRequest">unsafe XMLHTTPRequests</a> but ultimately failed, so a hack was in order. We registered <strong>pgp.messenger.com</strong> in the local <strong>hosts</strong> file, and pointed it to 127.0.0.1. This meant that our local API was now located on a CSP trusted domain. However, we still needed to access the API over SSL (as the CSP required https://), so we created a proxy server to our GPG API server, running with locally generated SSL certificates. Getting the browser to trust the generated SSL certificate, being the last hurdle, was not much of a problem.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI4MT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--2dd07dd2e0bfdcda9b4c9e1ba4b56f729c929eab" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBaGtCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--3b4035b8c207a3e344ec43e37894f2a182fec397/w_5AG9SzEJ3--E-27k2yyA.jpeg" filename="w_5AG9SzEJ3--E-27k2yyA.jpeg" filesize="98803" width="700" height="466" previewable="true" caption="Warning, hack in progress!">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjgxLCJwdXIiOiJibG9iX2lkIn19--dc35cc60c6518ce9494e1132317be66dffe016fb/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/w_5AG9SzEJ3--E-27k2yyA.jpeg">
<figcaption class="attachment__caption">
Warning, hack in progress!
</figcaption>
</figure></action-text-attachment>I think we all felt a great relief when we implemented the last feature with half an hour to spare. But the relief was short-lived. Soon the presentations started and to the stage came team after team of really talented people. One particular team found a way to misuse Android's accessibility features to create a system wide chat-bot. A different team created a chat-bot that understood natural language and helped sort out arguments between people. Others made games that looked really polished. And then came we, with our short presentation which showed a few messages being exchanged. Plain text on the left, encrypted and signed on the right. Packaged in with a short talk about how we managed to hook in to Messenger and that this can be used not only for encryption but as a platform for all kinds of extensions.<br><br><a href="https://youtu.be/S-LVSCIzip0">The extension in action — With the extension (left) and without (right)</a><br><br>After the presentations were over none of us believed that we would win anything. But that didn't matter! We created something really cool, traveled together, had an excellent weekend, drank a couple of beers and had fun in general. For us, no matter if we won any prize or not, this was a success. This was what a hackathon should be about!</div><h2>The anticipation</h2><div>After a while the room filled with people again, but this time for the prizes to be handed out. One prize was handed out after the other, but our name was never mentioned. Then came the prizes for the top three teams. By this point most of us stopped listening and only waited for the ceremony to be over as we desperately needed sleep. Before the first prize was given out there was a short inspiring speech by the organisers. It explained that they picked first place based on the hacker spirit, problem complexity and general usefulness of the project. Then they called our name. We were in shock. No one expected this to happen. Everybody's faces were white. When it finally hit us that we won, we rushed to the stage to pick up our prize. And so the whole experience was over.<br><br>This hackathon was something I will remember, not just because we did something awesome, but because it was really fun.<br><br>The organisers did an outstanding job, the food was great, the venue was even better. Thanks to all the sponsors for all the awesome lectures (and all the cool swag). Most of all, thanks to my team which made Copenhagen an experience to remember.<br><br>If you are thinking about going to such an event then do it. Don't hesitate. It doesn't matter if you win a prize or not. At the end of the day you will either create something awesome or learn a valuable lesson. Similarly, if you are thinking of organising a hackathon, don't hesitate. You will change somebody's life forever and meet really cool people in the process.</div><h2>Conclusion — Privacy is hard</h2><div>If you want to help us make MessengerPG a thing or to build your own extension on top of Messenger, take a look at <a href="https://github.com/monorkin/copenhacks-2016">our Github page</a>. Fork us. Look at the code. Change something. And feel free to open up a pull request. We will gladly take all the help we can get.<br><br>Finally, a word of caution. There is currently no way to guarantee that the code you audit will be the code you run next time you open Messenger. This allows malicious servers to serve targeted users with different versions of the code. So, while 99% of users may get clean, non-malicious code, the interesting 1% can get served with code endangering their privacy. This is also a problem with closed-source messaging services which offer end-to-end encryption which cannot be properly audited. Even if the messages being exchanged are encrypted end-to-end, what's stopping the proprietary code from collecting the messages you exchange and exposing them over a different channel?<br><br>Privacy is hard. It is not something we should take for granted, but something we should actively try to achieve and protect.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/22016-05-23T00:00:00Z2023-11-14T16:28:04ZTips to improve your tests<div class="trix-content">
<div>
<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI3Mz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--5b555dc466537d19183395798721137bec224c50" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBaEVCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--afe0513555743a5189ac094470f5c7812d34ceb3/TSnYL5U5EnQJg5xEeysMGg.jpeg" filename="TSnYL5U5EnQJg5xEeysMGg.jpeg" filesize="1192456" width="4000" height="2076" previewable="true" caption="A shoe stuck in the railing of a bridge in Stockholm">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjczLCJwdXIiOiJibG9iX2lkIn19--4d89dc0443dffb9b5017412ba8184e5c73a08e28/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/TSnYL5U5EnQJg5xEeysMGg.jpeg">
<figcaption class="attachment__caption">
A shoe stuck in the railing of a bridge in Stockholm
</figcaption>
</figure></action-text-attachment>Peoples' test suites often get out of hand. Having thousands of tests that run for hours on some CI should be a thing of the past. There doesn't exist code that's too hard to test or code that can't fail. You shouldn't waste your time waiting for your test suite, or chasing some bug you are unable to pinpoint. Writing a test suite that covers 100% of your code base isn't hard. But to do it right is extremely hard.<br><br>As I said earlier, all code is 'testable'. There only exists the age-old question of "should I write code that is easy to test?". By now, this discussion has turned into a 'religious war' of sorts. There is no definitive answer to this question.</div><h2>Writing 'testable' code</h2><div>
<strong>You shouldn't write code with your test suite in mind</strong>. Write code that makes sense. Write code that is easy to read, and easy to understand. Most of all, <strong>write code that is maintainable</strong>. If you are having problems coming up with a test suite for the code you wrote then go back and evaluate if your code is as good as it can be. More often than not you will find a way to refactor the code so that it makes more sense, to make it more readable, or to make it maintainable.<br><br>You can think of this as a positive feedback loop. Your code becomes easier to test the 'better' code you write. Better in the sense that it's maintainable, readable and sensible. Implementing one pattern (or good practice) will lead you to use other related patterns. And in the end, you will have a solid code base with an excellent test suite.<br><br>There are exceptions to this rule. Sometimes, 'good' code is hard to test. That's why you should always write code without your test suite in mind.<br><br>Knowing that your application, as a whole, returns expected results for certain scenarios isn't that reassuring. You can never cover all possible combinations of states in an application.<br><br>Therefore, it's better to test each module of your application by itself. Rather said — unit test your application. That way you can cover more, if not all possible states. And thus, be confident that your application will do what you expect from it. It also makes refactoring easier as you can detect if you changed an output without intention.</div><h2>Write more unit tests</h2><div>Unit tests are generally faster than integration test. Thus, they are cheaper to run. You can have thousands of unit tests that run in a couple of seconds and be assured that your code does what you expect from it. While running the same amount of integration tests might take hours. And they wouldn't tell you if your application performs as expected in all situations.<br><br><strong>You should write integrations tests!</strong> They assure that all modules of your application work well together. Hence the name — integration tests. Each feature (code execution path) should have at least one integration test. But, there is no rule to determine how many integration tests you should have. My rule of thumb is that, after you run only your integration tests, you should have code coverage greater than zero on each module.<br><br>It's only important that your integration tests reassure that the modules of which your application is made of can interact together without a fault.<action-text-attachment content-type="image" url="https://media.giphy.com/media/3o7rbGT9rrEJa4Y5LG/giphy.gif" width="480" height="270"><figure class="attachment attachment--preview">
<img width="480" height="270" src="https://media.giphy.com/media/3o7rbGT9rrEJa4Y5LG/giphy.gif">
</figure></action-text-attachment>
</div><h2>Stubs and mocks</h2><div>I often see people letting their 'unit' test call code from a module that isn't their module under test. That effectively turns a unit into an integration test.<br><br><strong>The basic idea of stubs is to decouple modules from one another</strong>. Stubs, as the name implies, stub out the behaviour of method calls. You can think of stubs as predefined responses to method calls. Stubbing a call to a method never executes the method's code, it just returns the desired value then and there. This is an incredibly powerful tool! <strong>Not only can you decouple modules this way, but you can control your program's flow</strong>. By stubbing certain methods you can simulate success and failure conditions without the need to craft a complex data set that triggers the same behavior.<br><br>Stubs can't assure that a method got called, which is important if that call does something 'mission critical'. Mocks were invented for this kind of situations. Mocks are identical to stubs, but they expect that the mocked method will get called at least once. If the method doesn't get called the test explicitly fails.<br><br>Using these two tools you can test all possible scenarios without the need for complex data sets and repercussion checking. This is the first step to making your tests run blazing fast!</div><h2>Asserting vs. mocking</h2><div>The question arises of when to assert a value and when to mock a call. <em>We can differentiate two kinds of method calls — queries and commands</em>. Queries are calls that return a value while commands are calls that change the state of the system. For instance, reading a file is a query as it returns the contents of the file. While writing a file is a command as it creates a file with a given content, thus changing the state of the file system.<br><br>We also differentiate incoming and outgoing calls. <strong>Outgoing calls are calls that your module under test makes to other modules. While incoming calls are calls made to the module under test</strong>.<br><br>The returned value of outgoing queries should not be asserted. These values will be asserted in the respective module's unit test. Outgoing commands should be mocked. We rely on other module's commands to work. It's only important that we know that they got called. By knowing this, we can stub out the behavior of other modules to simulate the command's repercussions without causing an artificial success state.<br><br>The returned value of incoming queries <em>should be</em> asserted. We need to make sure that given an input our method returns an expected result. Incoming commands' repercussions should be asserted for the same reason incoming queries should be asserted.<br><br><action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQxNT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--8c9e938bf818a4846e7898fdf68b42bc77601c90" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBcDhCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--6ac7fc31bc52b5aa462d71422f1da8ca8a3d5682/Screen%20Shot%202023-04-02%20at%2018.48.49.png" filename="Screen Shot 2023-04-02 at 18.48.49.png" filesize="62127" width="1994" height="402" previewable="true" presentation="gallery" caption="Decision table">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" style="aspect-ratio:1994 / 402;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDE1LCJwdXIiOiJibG9iX2lkIn19--250e0dc8ec86e9b5532a51b86dde431e80b04937/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/Screen%20Shot%202023-04-02%20at%2018.48.49.png">
<figcaption class="attachment__caption">
Decision table
</figcaption>
</figure></action-text-attachment><br><br>There also exist methods that are both queries and commands. A combination of the above rules should be applied when testing them.</div><h2>Write tests first?</h2><div>If you know what your module should do, then by all means you should write a test suite first. A test suite should always come first. That way you can ensure that the code you wrote complies to expectations.<br><br>There are exceptions to this rule. If you are unsure about the final architecture of the code or think that the code will drastically change during development. Then it's better to write a test suite after the fact.</div><h2>Conclusion</h2><div>No matter how you test your software you can never assure that no bugs are present. The methods outlined here are tools to help you write a better test suite. They should be food for thought, not hard rules to follow.</div><blockquote>Testing shows the presence, not the absence of bugs<br><br>— Edsger W. Dijkstra</blockquote><div>Having unit, integration tests, mocks, and spies gives you the assurance that your code does what you expect it to do. Being able to run your test suite fast enables you to iterate quickly. These two together, speed and confidence, enable you to write software at an incredible pace thus giving you time to improve your methods and skills. Resulting in incredible software.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/32018-03-01T00:00:00Z2023-11-14T16:28:05ZSupercharging services architectures with RabbitMQ<div class="trix-content">
<div>
<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI2ND9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--c1607fbf54d43cc16df9e6802418553e82d1c972" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZ2dCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--390cfb28f313a1b0884a788d5431b13f4e891308/bunny.png" filename="bunny.png" filesize="102728" width="2160" height="860" previewable="true" caption="A rabbit with a tachometer in it's silhouette">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjY0LCJwdXIiOiJibG9iX2lkIn19--5466cd017327c96e3e44505f523243405b0fef10/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/bunny.png">
<figcaption class="attachment__caption">
A rabbit with a tachometer in it's silhouette
</figcaption>
</figure></action-text-attachment>When I first started using RabbitMQ I didn't understand its usefulness beyond a job queue, but it's helped me to grow and manage services architectures without headaches.</div><h2>Services Architecture</h2><div>In web development, a services architecture describes a single application that consists of multiple, smaller, highly specialized, loosely coupled applications that each independently, or in tandem, solve one part of the business domain.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI2OT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--9b6e74c1405acfbb8e3bc8388876548c875a2d1a" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZzBCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--b4a90ebe1ed6d868ace47de95222a7a02ad3ca48/zDwh70Y-0ntatt7hwH8cpw.png" filename="zDwh70Y-0ntatt7hwH8cpw.png" filesize="59895" width="1000" height="452" previewable="true" caption="An example of a services architecture app">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjY5LCJwdXIiOiJibG9iX2lkIn19--59daceaa72a2b8c8b94beb7e5c290f9ba7f32450/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/zDwh70Y-0ntatt7hwH8cpw.png">
<figcaption class="attachment__caption">
An example of a services architecture app
</figcaption>
</figure></action-text-attachment>This architecture allows for <strong>parallel development of features</strong> (multiple teams can work on different services, each implementing one part of the business domain), <strong>continuous delivery</strong> of large applications without downtime (the application, as a whole, should partially work even with one service down), and the <strong>freedom to experiment and innovate</strong> on the tech stack (a service can be implemented in any technology, framework or language).<br><br>A services architecture introduces a new challenge — <strong>working with a distributed system.</strong>
</div><h2><strong>Problems</strong></h2><div>The most common problem is state management or <strong>data consistency</strong>. <strong>RabbitMQ doesn't tackle this problem at all.</strong> Your application now consists of multiple smaller applications, each of which can have their own state and/or database — on which another service can rely on (e.g. a mailer service needs a user's email from an authentication service). This <strong>causes consistency issues</strong>. There are many available solutions to this problem, like <a href="https://www.martinfowler.com/bliki/CQRS.html">CQRS</a>, but this topic won't be covered here.<br><br>The other common problem is <strong>service discovery</strong>. You need to have a way to contact the services that compose your application — keep track of their IPs, URIs, and health. This can get out of hand if you have many services that dynamically (horizontally) scale. Traditionally this issue is <a href="https://www.nginx.com/blog/service-discovery-in-a-microservices-architecture/">solved by utilizing load balancers and service registries</a> (like <a href="http://zookeeper.apache.org/">Apache Zookeeper</a> and <a href="https://www.consul.io/">Consul</a>), but <strong>RabbitMQ can partially solve this problem</strong> for us.<br><br>Lastly there is <strong>inter-process communication</strong>, or <a href="https://medium.com/netflix-techblog/fault-tolerance-in-a-high-volume-distributed-system-91ab4faae74a">sending messages between your services</a>. RabbitMQ, being a message queue, solves this problem too.</div><h2>RabbitMQ</h2><div>As the name implies, it's a message queue. Traditionally, in computer science message queues are used for <strong>inter-process communication</strong>, which can be asynchronous or synchronous. It uses <a href="https://en.wikipedia.org/wiki/Message_queue"><strong>AMPQ</strong></a> for communicating with clients — it's a wire-level binary protocol that provides mechanisms to ensure message delivery and consumption.<br><br>I was drawn to RabbitMQ because of two problems I've had while working on a project, <strong>which other message queue services didn't solve</strong>.<br><br>Some message queues <strong>don't have job execution guarantees</strong>, e.g. if a message is removed from a queue by a worker it needs to be processed no matter if the job database restarts or if the worker fails. In RabbitMQ this feature is <a href="https://www.rabbitmq.com/tutorials/amqp-concepts.html"><strong>guaranteed by AMPQ</strong></a>. The protocol specifies that, if desired, a message will be removed from a queue only after the consumer acknowledges it has been processed. If a consumer fails to do so in a given time frame, the message is given to another consumer.</div><div>The other problem I had with other queues was memory consumption. <strong>Some message queues keep the whole queue in memory</strong> which can bring even powerful machines to a crawl if the messages are too big or if there is too many of them. <strong>In RabbitMQ messages are kept in memory until a (configurable) threshold is reached at which point they will be written to disk.</strong> This is not true for all messages — persistent messages (those in durable queues) will be written to disk and kept in-memory as soon as they are enqueued, and there are also "lazy queues" which try to keep everything on disk and nothing in-memory (if possible).<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI2NT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--58f947ff4870b6f2f4f037944e5dc7339d71bc30" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZ2tCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--4d1836198af51bb4a6965fc79586e5f457b4ce38/sgYa89fDzP_j11PRbWhnMg.png" filename="sgYa89fDzP_j11PRbWhnMg.png" filesize="45523" width="1000" height="357" previewable="true" caption="Queue statistics">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjY1LCJwdXIiOiJibG9iX2lkIn19--d4ea51e7ab3e5c51a1d2a528d67216f9c38d1a83/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/sgYa89fDzP_j11PRbWhnMg.png">
<figcaption class="attachment__caption">
Queue statistics
</figcaption>
</figure></action-text-attachment>
</div><h2><strong>RPC / Pub-Sub</strong></h2><div>Most people I've talked to aren't familiar with RabbitMQ's <a href="https://www.rabbitmq.com/direct-reply-to.html"><strong>"direct reply-to"</strong></a><strong> feature</strong>. It enables <strong>synchronous inter-process communication</strong> — you can think of it as an <strong>RPC or a Pub-Sub interface</strong>. With it, any client can send a message to any queue and get a direct response from any consumer processing messages in that queue. The same rules of memory usage and persistence as described before apply, but <strong>message loss protection is not guaranteed with this mechanism</strong>.<br><br>In my opinion, this method of inter-process communication is better than building your services as HTTP servers. With HTTP servers you re-implement logic that is already present in message queues — e.g. success and failure responses, timeouts, and routing. Oftentimes HTTP servers come with unnecessary overhead for a services architecture like some middleware, session storage and encryption mechanisms which can slow the whole application down. <strong>With AMPQ and RabbitMQ you only need to implement your business and serialization logic</strong> (in what format will the messages be written to the queue).</div><h2><strong>Exchanges</strong></h2><div>In AMPQ, messages aren't published directly to queues, but to exchanges. Exchanges are routers (or post offices) that determine which queues (more than one!) should receive a message.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI3MD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--f6df5a12fc6670ae60b557cf64ee8774caa8ad2d" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZzRCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--9841140675bebae013c778461bfd8a23d27d3f0a/Dylm8HewBQraHfs4Wma7EA.png" filename="Dylm8HewBQraHfs4Wma7EA.png" filesize="35154" width="1110" height="430" previewable="true" caption="Topic exchange example">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjcwLCJwdXIiOiJibG9iX2lkIn19--608e49f070acbe273089d1f57dbdfe4e3979d5a5/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/Dylm8HewBQraHfs4Wma7EA.png">
<figcaption class="attachment__caption">
Topic exchange example
</figcaption>
</figure></action-text-attachment>RabbitMQ supports four types of exchanges — direct, fan-out, topic and headers.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI2Nj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--b47ddafcaea5ceb707c60970db33d9347110af5c" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZ29CIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--33ed9da3f9117ef283bf8bb8fe96b3131b1355c3/GAWkJRBOykY4Ly5TsW4cCA.png" filename="GAWkJRBOykY4Ly5TsW4cCA.png" filesize="374293" width="1400" height="786" previewable="true" caption="Direct exchange example">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjY2LCJwdXIiOiJibG9iX2lkIn19--ed60303b73b4135fb36c36272d504045b17c1ea9/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/GAWkJRBOykY4Ly5TsW4cCA.png">
<figcaption class="attachment__caption">
Direct exchange example
</figcaption>
</figure></action-text-attachment><a href="https://www.rabbitmq.com/tutorials/amqp-concepts.html#exchange-direct"><strong>Direct exchanges</strong></a><strong> deliver messages directly to a single queue</strong>, e.g. when you don't want to use exchanges and just want to deliver a message to a queue. For example, if each chat room of an application is represented by a queue, a Ruby Chat Exchange would deliver messages <strong>only</strong> to the Ruby chat queue.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI3MT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--7971b3f9ec6c809767f05e42325a285ef5442180" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZzhCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--3c54d499056b114dcd02134b6a3adce7abd2326e/uEtLTK4ppNItugOfhtEOGA.png" filename="uEtLTK4ppNItugOfhtEOGA.png" filesize="87712" width="700" height="392" previewable="true" caption="Fan-out exchange example">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjcxLCJwdXIiOiJibG9iX2lkIn19--072c93a782400fe33fc5c18d8346b4d6e317448f/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/uEtLTK4ppNItugOfhtEOGA.png">
<figcaption class="attachment__caption">
Fan-out exchange example
</figcaption>
</figure></action-text-attachment><a href="https://www.rabbitmq.com/tutorials/amqp-concepts.html#exchange-fanout"><strong>Fan-out exchanges</strong></a><strong> deliver messages to all queues bound to them</strong>. You can think of them as groups, a message sent to the group will be delivered to all queues in the group. With the previous chat example, a fan-out exchange can be imagined as a broadcast to select channels. E.g. if we want to send a messsage to all chats of awesome languages we can publish a messsage in the Awesome languages exchange which would deliver those messages to both Ruby and Rust chats.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI2Nz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--1692f06f1826c278c1467937e4e8f1b974c11aa4" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZ3NCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--4fb048e4aaa1e4cabf339f067f564da67d376352/wH85oYIoW7LCj4vzhi7x0Q.png" filename="wH85oYIoW7LCj4vzhi7x0Q.png" filesize="87078" width="700" height="392" previewable="true" caption="Topic exchange example">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjY3LCJwdXIiOiJibG9iX2lkIn19--05688dd37c06c443cf6a48e4aac335bc3434dcec/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/wH85oYIoW7LCj4vzhi7x0Q.png">
<figcaption class="attachment__caption">
Topic exchange example
</figcaption>
</figure></action-text-attachment><a href="https://www.rabbitmq.com/tutorials/amqp-concepts.html#exchange-topic"><strong>Topic exchanges</strong></a><strong> deliver a message to queues tagged with a topic</strong> (the above image is an example of such an exchange). E.g. if a message has a routing key (tag) of "Fedex" and it's delivered to the "Shipping" exchange it will be delivered to the "Fedex" queue. Or, with our chat example, a message tagged with text would get delivered to the Chat queue, while a message tagged video would get delivered to the video queue.<br><br>Lastly, <a href="https://www.rabbitmq.com/tutorials/amqp-concepts.html#exchange-headers"><strong>header exchanges</strong></a><strong> are an continuation on the idea of topic exchanges</strong>, they can take different properties of the message being delivered to determine where it should be delivered.<br><br>Exchanges come with a multitude of features — <a href="https://www.rabbitmq.com/dlx.html">dead-lettering</a> (if a message isn't acknowledged or was rejected it gets sent to another exchange), <a href="https://www.rabbitmq.com/ae.html">alternate exchanges</a> (a client can specify an exchange to which a message gets routed if the primary exchange rejects it), <a href="https://www.rabbitmq.com/consumer-priority.html">priority consumers</a> (the ability to specify which consumers to prefer, e.g. consumers on more powerful machines), <a href="https://www.rabbitmq.com/priority.html">priority queues</a> (messages can be assigned a priority, higher priority messages get processed first), <a href="https://www.rabbitmq.com/ttl.html">TTLs</a> (specifies how long a message or queue "lives", though this feature has a caveat — please refer to the manual before using it), and others.<br><br>They are not only useful for offloading message delivery logic from your services off to RabbitMQ, but also serve as a <strong>way to deprecate services, and reduce downtime</strong>. If a service is to be deprecated, its messages can be routed to another service that knows how to process them, or that returns errors interpretable by other services as responses. If a new version of a service is about to be deployed an exchange can be configured to deliver messages simultaneously to the old service's and the new service's queues. Or, when the new service is up, all incoming messages can be redirected to it.</div><h2><strong>Service discovery</strong></h2><div>Since RabbitMQ supports multiple producer and multiple consumer queues, <strong>exchanges can be utilized as a form of service discovery</strong>. Each service knows its name and the names of the services it depends on, therefore it only has to publish messages on their exchanges (or fail if an exchange doesn't exist), and listen for messages on its queue.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI3Mj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--1f30542d0fd662447f744fa06f1d947b285b8d33" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBaEFCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--9b5a7b13ac78138b76b1f83f78e7dc12a323e297/YaTuK3Y5j_6iozZN29ogjg.png" filename="YaTuK3Y5j_6iozZN29ogjg.png" filesize="77634" width="1523" height="532" previewable="true" caption="List of available queues">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjcyLCJwdXIiOiJibG9iX2lkIn19--cecd0eb7d30fc82c87685b308b7df3cdfa4e5308/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/YaTuK3Y5j_6iozZN29ogjg.png">
<figcaption class="attachment__caption">
List of available queues
</figcaption>
</figure></action-text-attachment>For small services architectures this kind of service discovery is sufficient. All services will communicate through RabbitMQ, and therefore we are able to keep track of them. But sometimes the need arises to communicate with other applications that don't use AMPQ, or keep track of applications in a cluster (e.g. for databases, or RabbitMQ instances) — this kind of service discovery is <strong>not suitable for those use-cases</strong> and should be handled with the like of Consul or Zookeeper.</div><h2><strong>Plugins</strong></h2><div>Personally, I find this to be the "<strong>killer feature</strong>" of RabbitMQ. If RabbitMQ is missing any kind of functionality you desire it can be added. Out-of-the-box it comes with quite a few useful plugins.<br><br>The one plugin I use most often, and the first plugin I introduce people to is the <a href="https://www.rabbitmq.com/plugins.html"><strong>Management plugin</strong></a>. It gives RabbitMQ a full user interface through which you can configure and monitor individual exchanges and queues, monitor system performance, memory and disk usage.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI2OD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--7f9dd48461d9231d348b5e91d240003d229e190a" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZ3dCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--3bdf28275149214aceb97d6cb6013dcb1b0cc18d/Zrhh-vWJRIiaTvnPzyJZnQ.png" filename="Zrhh-vWJRIiaTvnPzyJZnQ.png" filesize="294525" width="2000" height="1232" previewable="true" caption="Overview panel in the Management plugin">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjY4LCJwdXIiOiJibG9iX2lkIn19--dc22bc859dff1f6200f69ac8f30a2bac84df27e0/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/Zrhh-vWJRIiaTvnPzyJZnQ.png">
<figcaption class="attachment__caption">
Overview panel in the Management plugin
</figcaption>
</figure></action-text-attachment>Through plugins <strong>RabbitMQ can support competing protocols to AMPQ</strong> — like MQTT, STOMP and WebSockets.<br><br>And while clustering is supported, the <strong>Federation plugin</strong> brings it to a new level. It enables message passing between brokers without clustering (useful since RabbitMQ requires IP addresses of the other instances, instead of URIs, when clustered). E.g. multiple clusters of RabbitMQ can exchange messages between each other. An useful analogy would be Mastodon's and Diaspora's federations which allow users on different instances to communicate as if they were on the same instance. It also comes with a UI that's accessible through the Management plugin.<br><br>Lastly, there is a feature called "<strong>firehose</strong>" that logs all internal messages of RabbitMQ. It's extremely useful for debugging plugin behavior, exchange configuration and even just for logging.</div><h2>Conclusion</h2><div>RabbitMQ has helped me manage and scale my services architectures by utilizing plugins, exchanges, queues and the RPC interface. It's become an essential tool for building services.<br><br><strong>My biggest issue now is my dependency on it</strong> — it's become the backbone of my services architectures. This introduced a large single point of failure, but it's manageable with the Federation plugin and clustering.<br><br>If you are struggling with the limitations of your queue service, clustering or inter-process communication — <strong>try RabbitMQ</strong>. It excels at all those tasks and brings many utilities that can prove helpful for managing your application.<br><br>If you are struggling to introduce RabbitMQ to your project. <strong>Try it as a simple queue first</strong>, and then slowly move more and more services to it</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/42018-03-29T00:00:00Z2023-11-14T16:28:04ZDo you really need WebSockets?<div class="trix-content">
<div>
<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI0Nj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--d47729820c74998bfca0d740fd94b7bec83e8f6b" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZlk9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--e0b5d22bc0800e509ed27c72f44350776cae5115/cloud.png" filename="cloud.png" filesize="29190" width="2000" height="800" previewable="true" caption="The Cloud was made by Fabián Alexis (CC BY-SA 3.0), via Wikimedia Commons">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjQ2LCJwdXIiOiJibG9iX2lkIn19--36e81d7bc30f721d79b9e24366125395f27fe40c/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/cloud.png">
<figcaption class="attachment__caption">
The Cloud was made by Fabián Alexis (CC BY-SA 3.0), via Wikimedia Commons
</figcaption>
</figure></action-text-attachment>Over the years I've had this conversation a couple of times. This post will explain why we use WebSockets, how they can be used, what alternatives exist and when to use them.<br><br>Every time I worked on a project where we had to implement any kind of a "real-time" component, usually a chat or an event feed, the word WebSockets started to circulate. Though, most people use them because they either aren't aware of the alternatives, or they blindly follow other people's examples.</div><h2>Why WebSockets?</h2><div>
<strong>WebSockets enable the server and client to send messages to each other at any time</strong>, after a connection is established, without an explicit request by one or the other. This is in contrast to HTTP, which is traditionally associated with the challenge-response principle — where to get data one has to explicitly request it. In more technical terms, WebSockets enable a <a href="https://en.wikipedia.org/wiki/Duplex_(telecommunications)#Full_duplex">full-duplex connection</a> between the client and the server.<br><br>In a challenge-response system there is no way for clients to know when new data is available for them (except by asking the server periodically — <a href="https://en.wikipedia.org/wiki/Polling_(computer_science)">polling</a> or <a href="https://en.wikipedia.org/wiki/Push_technology#Long_polling">long polling</a>), <strong>with Websockets the server can push new data at any time which makes them the better candidate for "real-time" applications.</strong><br><br><strong>It's important to note that WebSockets convert their HTTP connection to a WebSocket connection.</strong> In other words, a WebSocket connection uses HTTP only to do the initial handshake (for authorization and authentification), after which the TCP connection is utilized to send data via the <a href="https://tools.ietf.org/html/rfc6455#section-5.2">its own protocol</a>.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI0Mz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--66d9d89dc2be9575c46aef48497beacc7323484c" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZk09IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--f4269b2891a266fb4dedc5bb0372c6caaba05e11/v2XywM5pSK_f5VZyWDMPoQ.gif" filename="v2XywM5pSK_f5VZyWDMPoQ.gif" filesize="224261" width="1000" height="200" previewable="true" caption="Animation of a WebSocket connection being established">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjQzLCJwdXIiOiJibG9iX2lkIn19--94071c55e37c8c6a8bb0430d1e5262c67f3d373c/v2XywM5pSK_f5VZyWDMPoQ.gif">
<figcaption class="attachment__caption">
Animation of a WebSocket connection being established
</figcaption>
</figure></action-text-attachment>WebSockets are a part of <a href="https://www.w3.org/TR/websockets/">the HTML5 spec</a> and they are supported by <a href="https://caniuse.com/#feat=websockets">all modern browsers</a> (meaning, there is a JS API to use them natively in the browser). They provide a mechanism to detect dropped (disconnected) clients and can handle up to a 1024 connections per browser, <strong>though </strong><a href="https://www.nginx.com/blog/websocket-nginx/"><strong>they aren't compatible with most load balancers out-of-the-box</strong></a><strong> and have no re-connection handling mechanism.</strong>
</div><pre>#!/usr/bin/node
// Create WebSocket connection.
const socket = new WebSocket('ws://localhost:8080');
// Connection opened
socket.addEventListener('open', function (event) {
socket.send('Hello Server!');
});
// Listen for messages
socket.addEventListener('message', function (event) {
console.log('Message from server ', event.data);
});</pre><div>
<strong>The most common example for WebSockets is either a chat or push notifications. They can be used for those applications, but present an overkill solution to the problem</strong>, since in those applications only the server needs to push data to the clients, and not the other way around — only a half-duplex connection is needed.<br><br><strong>In Ruby, there are a few gems that add WebSockets to your web app.</strong> The one I've mostly used is Faye, though I've been looking at websocket-ruby lately. Rails supports them out-of-the-box since version 5 through ActionCable.</div><pre>#!/usr/bin/ruby
require 'faye/websocket'
App = lambda do |env|
if Faye::WebSocket.websocket?(env)
ws = Faye::WebSocket.new(env)
ws.on :message do |event|
ws.send(event.data)
end
ws.on :close do |event|
p [:close, event.code, event.reason]
ws = nil
end
# Return async Rack response
ws.rack_response
else
# Normal HTTP request
[200, {'Content-Type' => 'text/plain'}, ['Hello']]
end
end</pre><div>In my opinion, <strong>all of those implementations are more-or-less the same — you can't really go wrong.</strong> Note that some gems also require their own JS library (mostly to encode and decode data that's being sent or received).</div><h2>Server-sent events</h2><div>From my experience, most people don't know that regular old <strong>HTTP provides a mechanism to push data from the server to clients via </strong><a href="https://en.wikipedia.org/wiki/Server-sent_events"><strong>Server-Sent Events</strong></a> (aka. <a href="https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events">EventSources</a>).<br><br><strong>Server-Sent Events utilize a regular HTTP octet streams</strong>, and therefore are limited to the browser's connection <a href="https://stackoverflow.com/questions/985431/max-parallel-http-connections-in-a-browser">pool limit of ~6 concurrent HTTP connections per server</a>. But <strong>they provide a standard way of pushing data from the server to the clients over HTTP</strong>, which load balancers and proxies understand out-of-the-box. The biggest advantage being that, exactly as WebSockets, they <strong>utilize only one TCP connection</strong>. The biggest disadvantage is that <strong>Server-Sent Events don't provide a mechanism to detect dropped clients</strong> until a message is sent.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI0ND9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--ad92b2e455e5469936a3449c9a922bc439e142cc" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZlE9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--00c39a62c6ef4dfd4f513ec6caf111c4adfa6ce4/K2J4QncY_0I3gunNKXq2_g.gif" filename="K2J4QncY_0I3gunNKXq2_g.gif" filesize="174301" width="1000" height="200" previewable="true" caption="Animation of a Server-Sent Events connection being established">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjQ0LCJwdXIiOiJibG9iX2lkIn19--0ae1f11b16d7bcd9b21190813edbb4ce72214c17/K2J4QncY_0I3gunNKXq2_g.gif">
<figcaption class="attachment__caption">
Animation of a Server-Sent Events connection being established
</figcaption>
</figure></action-text-attachment><a href="https://html.spec.whatwg.org/multipage/server-sent-events.html#server-sent-events">They are standardized via HTML 5</a>, most HTTP servers support them out-of-the-box, and <a href="https://caniuse.com/#search=server%20sent%20events">they are available in most browsers except for Internet Explorer and Edge</a> where they are <a href="https://github.com/Yaffle/EventSource">available through a polyfill</a>.<br><br><a href="http://edgeapi.rubyonrails.org/classes/ActionController/Live/SSE.html"><strong>Rails</strong></a><strong>, </strong><a href="https://github.com/jeremyevans/roda/blob/master/lib/roda/plugins/streaming.rb"><strong>Roda</strong></a><strong> and </strong><a href="https://gist.github.com/rkh/1476463"><strong>Sinatra</strong></a><strong> support them out-of-the-box.</strong> And they have a simple protocol — payloads are just prefixed with one of the following keywords <strong>data:</strong>, <strong>event:</strong>, <strong>id:</strong> and <strong>retry:</strong>. <strong>data:</strong> is used to push a payload, <strong>event:</strong> is optional and indicates the type of data being pushed, <strong>id:</strong> is also optional and indicates the event's ID, finally <strong>retry:</strong> instructs the client to change it's connection retry timeout — <strong>unlike WebSockets, Server-Sent Events have a reconnect mechanism built-in</strong>, though this is a feature that most WebSocket libraries add any way.</div><pre>#!/usr/bin/ruby
# frozen_string_literal: true
class App < Roda
QUEUES = []
Thread.new do
loop do
sleep(60)
QUEUES.each { |q| q << { heartbeat: true } }
end
end
plugin :streaming
plugin :render, engine: 'slim'
route do |r|
r.root do
view('root', layout: false)
end
r.on 'messages' do
r.post do
name = r.params['name']
message = r.params['message']
object = { name: name, message: message }
QUEUES.each { |q| q << object }
object.to_json
end
end
r.get 'stream' do
response['Content-Type'] = 'text/event-stream;charset=UTF-8'
q = Queue.new
QUEUES << q
q << { heartbeat: true }
stream(loop: true, callback: proc { QUEUES.delete(q) }) do |out|
loop do
out << "data: #{q.pop.to_json}\n\n"
end
end
end
end
end</pre><div>In the example above, <strong>to send data to other clients a regular HTTP POST request is made</strong> to the server. A heartbeat is kept to keep the connections alive (WebSockets also do that) and to detect dropped clients (since dropped connections can only be detected when data is pushed to them).</div><h2>Long polling</h2><div>Before there were Server-Sent Events people usually resorted to long polling to receive "real-time" data from the server. Not to give the wrong impression, long polling is still used today in some scenarios.<br><br><strong>Long polling utilizes regular HTTP requests</strong>. When a request is made to the server it responds immediately if there is data that can be served. If no data is available, the server drags out its response while the client waits. If in that time new data becomes available it's served to the client. When the client receives data, or its request times-out, it immediately makes a new request to re-establish the connection.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI0NT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--d25838f2dd39ca3c9eb54f1e99190ea37dfaf45b" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZlU9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--9217006fd4a519baa1d425b43d1b179baf777d76/PIAkRkmPMripwaFMz83nZQ.gif" filename="PIAkRkmPMripwaFMz83nZQ.gif" filesize="200797" width="1000" height="200" previewable="true" caption="Animation of a Long Polling connection being established">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjQ1LCJwdXIiOiJibG9iX2lkIn19--e471b1c90398ea2a870b61f02c66c8aa5cfad5e4/PIAkRkmPMripwaFMz83nZQ.gif">
<figcaption class="attachment__caption">
Animation of a Long Polling connection being established
</figcaption>
</figure></action-text-attachment><strong>Long polling's biggest advantage is the fact that it works in every environment</strong>, and in every browser. <strong>It's, arguably, better for applications with large numbers of concurrent users</strong> because it doesn't require a constant TCP connection to the server (it's harder to starve the server of TCP connections since they are periodically released). <strong>Though they come with the overhead of having to re-authenticate and re-authorize the client on each request</strong>. And the server needs to implement some kind of event aggregation to overcome blackouts between re-connects. <strong>There are no JS APIs available for this mechanism, there is no re-connection handling, nor dropped client detection.</strong><br><br>If data needs to be sent to the server a regular HTTP POST request is made, the same as with Server-Sent Events.<br><br><strong>When it comes to pushing data from the server to clients both WebSockets and Server-Sent Event will do the job.</strong> There are some subtle differences and incompatibilities, but there are libraries that solve those issues.<br><br><strong>If you don't need to send data to the server in "real-time" (e.g. voice/video chat, multiplayer games, …) go with Server-Sent Events.</strong> They are the standard HTTP way of pushing data (like notifications, messages or events) to clients. And <strong>they can be added just by implementing an additional endpoint in a controller</strong>, lowering the need to refactor the existing code base (in a well structured code base the "implementation cost" of both WS and SSE is the same), and making them somewhat faster to implement (and bring the feature to market).<br><br>I don't have experience with long polling on projects with large numbers of concurrent users, so I can't attest the claims I've read about it being the better solution for those kinds of applications. Though, while we were <a href="https://stanko.io/hacking-privacy-into-facebook-s-messenger-in-24-hours-3da239baf6c5">reverse engineering Facebook's Messenger to add PGP to it</a> I noticed that they utilize long polling to fetch messages. <strong>On smaller projects I would recommend SSE over Long polling since it's easier to implement on both the server and client side.</strong>
</div><h2>Conclusion</h2><div>If you are already using WebSockets or Long polling, don't go and convert them to Server-Sent Events. <strong>All solutions are basically the same when it comes to pushing data from the server to clients.</strong><action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzM2OT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--81a66e9c7f67608b1899d5b879e446b5c62e53c2" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBbkVCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--00cf39b576ec5ee30f502e21d11397a349cf4d18/image.png" filename="image.png" filesize="59366" width="1003" height="521" previewable="true" presentation="gallery" caption="Comparison of all three methods against one another">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" style="aspect-ratio:1003 / 521;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzY5LCJwdXIiOiJibG9iX2lkIn19--2f7ea166ad6aa7ecf69aa67415ce3018ec260a47/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/image.png">
<figcaption class="attachment__caption">
Comparison of all three methods against one another
</figcaption>
</figure></action-text-attachment>
</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/52018-03-07T00:00:00Z2023-11-14T16:28:04ZRabbitMQ is more than a Sidekiq replacement<div class="trix-content">
<div>
<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI2MD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--a25798203fc7d53574aea8d7ab664ea477480854" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZ1FCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--922002215b110196418c13134ab77457dc66b2c2/rabbit_karate.png" filename="rabbit_karate.png" filesize="123940" width="2160" height="1000" previewable="true" caption="A rabbit dressed up as the karate kid - a cross of RabbitMQ and Sidekiq logos">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjYwLCJwdXIiOiJibG9iX2lkIn19--ab73c1736a736f2f85addee81ee655e10ef77f4c/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/rabbit_karate.png">
<figcaption class="attachment__caption">
A rabbit dressed up as the karate kid - a cross of RabbitMQ and Sidekiq logos
</figcaption>
</figure></action-text-attachment>I've had gripes with <a href="https://github.com/mperham/sidekiq">Sidekiq</a> because of which I switched to RabbitMQ. Here are my thoughts and experiences after a year of using it in production.<br><br>I got inspired to write this post by the overwhelming response I received for my <a href="https://github.com/stankec/lectures/tree/master/19-rabbitmq_is_more_than_a_sidekiq_replacement">talk</a> at the <a href="https://www.meetup.com/rubyzg/">local Ruby user group</a>.</div><h2>Why do we need Sidekiq or RabbitMQ?</h2><div>"Background job" libraries like Sidekiq are used in situations when <strong>the result of a process can be yielded, but further side effects still need to be caused.</strong><br><br>An example of this are signup forms. When a user signs up, usually, a confirmation email is sent out to confirm their email, but the email is not crucial to the signup process — the user's account will be created even if the confirmation email fails to get sent.<br><br>It's unclear what to do if the email fails to get sent. The intuitive solution would be to return an error, but then the user wouldn't be able to create a new account since their email is already taken. <strong>We are penalizing the user for something that isn't their fault</strong>, something they can't influence and something that has no effect on the action they wanted to undertake. If we decide to retry the action we end up with the same problem as before if it fails again.<br><br>To resolve those issues we offload that work to a "background job" library. In the case of the signup example, the user's account would be created and they would be logged in. The "send email" job would be put in a queue, and would eventually get processed. If the job fails we can retry it as many times as we like, or run custom logic on the raised errors. <strong>The user isn't penalized for our mistake.</strong><br><br><strong>But do we really need Sidekiq for this?</strong> No. The same functionality can be accomplished with the standard <a href="http://ruby-doc.org/core-2.5.0/Queue.html">Queue class</a> and a <a href="http://ruby-doc.org/core-2.5.0/Thread.html">Thread</a>.</div><pre>#!/usr/bin/ruby
class App < Roda
JOBS = Queue.new
Thread.new do
loop do
begin
job = JOBS.pop
job.call
rescue => e
puts "ERROR: #{e}"
JOBS << job
end
end
end
route do |r|
r.on 'sign_up' do
r.post do
email = r.params['email']
JOBS << proc do
Mailer::ConfirmationMailer.deliver(email)
end
end
end
end
end</pre><h2>Why do we use background workers then?</h2><div>The above approach has many downsides. Not to go too deep down the rabbit hole, I'll focus on <strong>debuggability</strong> (yes, I made that word up), <strong>persistence</strong>, <strong>scaling</strong> and lastly <strong>fault tolerance</strong>.<br><br>The above solution is difficult to debug. There is no clear way to inspect the contents of the queue without littering the code with <a href="https://github.com/pry/pry">bindings to pry</a>. If the server is stopped (e.g. to add said bindings to pry) all jobs in the queue are lost, which means that we can't even extract the job that caused the issue to replicate it. <strong>To solve those issues Sidekiq uses </strong><a href="https://redis.io/"><strong>Redis</strong></a><strong> to store it's jobs.</strong> Redis is an in-memory key-value store that can store values as many different data types. Jobs are stored in a list as JSON objects. We can inspect the queue's content by connecting to Redis with redis-cli and inspecting the queue list.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI2MT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--5839e91d9f4dc729defc4665d903966b42be4a4b" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZ1VCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--2eafd87dd6dfdd9c7e138b8b7489e49f8df588ed/CsKEeXDBB7evp8X129dVWw.png" filename="CsKEeXDBB7evp8X129dVWw.png" filesize="130263" width="1000" height="238" previewable="true" caption="Contents of the default queue in Redis">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjYxLCJwdXIiOiJibG9iX2lkIn19--e32712275e0dd902de8c5107bb8d82f76d5c63b7/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/CsKEeXDBB7evp8X129dVWw.png">
<figcaption class="attachment__caption">
Contents of the default queue in Redis
</figcaption>
</figure></action-text-attachment>Using Redis as a central job queue also enables us to scale the number of workers to accommodate higher workloads.</div><h2>Sidekiq's memory problem</h2><div>There are two concerns regarding fault tolerance, Redis and the worker process. By default, <strong>Redis is a volatile store</strong> (data may be lost if the store restarts), though that can be changed by utilizing its <a href="https://redis.io/topics/persistence">RDB and AOF features</a> which in conjunction prevent any data loss. There are some caveats, if RDB (which is enabled by default) is used without AOF, data loss may still occur, and if used in conjunction with AOF may cause performance fluctuations.<br><br>When it comes to worker failures, Sidekiq handles most issues regarding handling Ruby errors and retry logic. <strong>But if a worker crashes or is killed while processing a job, that job is gone.</strong><action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI0OD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--3b68d9c19bbd7725e1010629d9095aba580fe721" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZmc9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--518d48d1bac4edc981f9d9fd5100550c67985df3/pN5LdFeiowAimYZFQEmmEg.gif" filename="pN5LdFeiowAimYZFQEmmEg.gif" filesize="74898" width="700" height="412" previewable="true" caption="Animation of how a job can disappear from Sidekiq in the case of a worker restart/crash">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjQ4LCJwdXIiOiJibG9iX2lkIn19--ed7cbf351f1c22a79e50e01ddf5d68cd28145af3/pN5LdFeiowAimYZFQEmmEg.gif">
<figcaption class="attachment__caption">
Animation of how a job can disappear from Sidekiq in the case of a worker restart/crash
</figcaption>
</figure></action-text-attachment>The most common cases for job disappearance are <strong>deploys</strong> (since the application is killed at that point, though Sidekiq offers rolling restarts with it's Pro plan) and <strong>VM crashes</strong> caused by faulty gem extensions (as they can't be caught).<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI0OT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--e1ce75965007e35f404e2afab704f1b6ab4bb8b0" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZms9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--841915a88ce25e393f98c41ee1f26729652db817/ildByqYgDM_br_4vv1YG5w.png" filename="ildByqYgDM_br_4vv1YG5w.png" filesize="58156" width="1000" height="153" previewable="true" caption="Screenshot of Sidekiq's README about error handeling">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjQ5LCJwdXIiOiJibG9iX2lkIn19--357d5b02c6c0cdc7ee53ae6459a11e4e93ca8b97/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/ildByqYgDM_br_4vv1YG5w.png">
<figcaption class="attachment__caption">
Screenshot of Sidekiq's README about error handeling
</figcaption>
</figure></action-text-attachment>If you are a business, or have a $1000 spare, you can resolve this issue by buying a <a href="https://sidekiq.org/products/pro.html">Sidekiq Pro license</a> which ensures that a job gets executed and comes with other niceties. Personally, I find this price tag too steep (especially since it's recurring) for small/personal projects which lead me to look for alternatives.<br><br>If we ignore the fault tolerance pay wall, Sidekiq still has a memory consumption issue as it uses Redis for its job queue. Redis, is an in-memory data store, and it has no mechanism to offload stale data to disk. This means that <strong>all jobs in the queue are kept in-memory all the time</strong>.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI1MD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--561c5f1f5512cde92f21845e9e12dac174cc8918" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZm89IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--555e20d335ba653c40071d42d1d9d17c91fa4e80/hZcnzhMtLzR6XBuI5XL7Tw.png" filename="hZcnzhMtLzR6XBuI5XL7Tw.png" filesize="43079" width="1000" height="555" previewable="true" caption="Redis memory usage plot around peak usage times">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjUwLCJwdXIiOiJibG9iX2lkIn19--24da4cb8d111f0884b9c4468b0189503684f1390/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/hZcnzhMtLzR6XBuI5XL7Tw.png">
<figcaption class="attachment__caption">
Redis memory usage plot around peak usage times
</figcaption>
</figure></action-text-attachment>The most common solution to this issue is passing IDs of database records instead of values to the job queue. <strong>This reduces Redis' memory consumption, but increases Sidekiq's or Ruby's</strong> since each Sidekiq instance needs to initialize your application to get access to your models. This can be solved by writing your workers as lightweight Ruby processes, but now we have the issue of managing models and database access information in two separate applications (your worker and your main application). Another solution is to consume your main app's API, but then we are increasing load on our app instead of off-loading work from it.</div><h2>How RabbitMQ solves those issues</h2><div>RabbitMQ is a general purpose message queue. To utilize it in a "background worker" backend scenario we need a library to communicate with it. I would highly recommend <a href="https://github.com/jondot/sneakers">Sneakers</a> for this purpose. Sneakers handles worker process creation, management, queue creation, and job enqueuing — everything that Sidekiq does, and offers a syntax that resembles Sidekiq's syntax.</div><pre>#!/usr/bin/ruby
class SneakersLogger
include Sneakers::Worker
# Defines the queue and it's options
from_queue 'loggings'
def work(log_message)
Logger.log(log_message)
# Does magic that will be explaned in the next section
ack!
end
end
# ---
class SidekiqLogger
include Sidekiq::Worker
def perform(log_message)
Logger.log(log_message)
end
end</pre><div>The biggest difference between the two implementations is the <strong>ack!</strong> on line 10. <strong>That line enables Sneakers and RabbitMQ to guarantee that a job has been processed.</strong> This is a feature of RabbitMQ's communication protocol — AMQP. In AMQP a message can be popped from a queue in two modes — ack mode and no-ack mode.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI1MT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--f0cd928d2a2d42003935b6095e972b14db6493a9" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZnM9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--aa84f0a97f66d77e6ec8cf43034b45000395907d/yEaLpgLdxaLXE0AFCWh79g.gif" filename="yEaLpgLdxaLXE0AFCWh79g.gif" filesize="141245" width="700" height="412" previewable="true" caption="Animation of a consumer failing and succeeding in ACK mode">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjUxLCJwdXIiOiJibG9iX2lkIn19--aa9061fdbfaeaf37f78afa2106622da1d3a87f19/yEaLpgLdxaLXE0AFCWh79g.gif">
<figcaption class="attachment__caption">
Animation of a consumer failing and succeeding in ACK mode
</figcaption>
</figure></action-text-attachment><strong>In ack mode the consumer must specify the maximum amount of time for it to process the message.</strong> When the consumer pops a message from the queue it's virtually removed from it, but RabbitMQ still keeps a copy of it. <strong>If the consumer fails to send an "ack" signal in the specified time period the message is put back at the front of the queue so that another consumer can process it.</strong> If the consumer sends an "ack" signal in the specified time period the message is fully removed from RabbitMQ. <strong>In no-ack mode no guarantees are given</strong>, no time window has to be specified, and no "ack" signal has to be sent.<br><br><strong>Another difference is memory consumption.</strong> By default, RabbitMQ keeps as many messages in memory as it can, before it reaches a configurable high water mark. At that point it offloads all eligible messages to disk. Though, that is not true for all queues. RabbitMQ also provides a "lazy queue" which keeps all it's messages on disk if possible — it's useful for passing large messages.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI1Mj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--59db7511b3ab4a51a455c553e0d71222bbf72cd6" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZnc9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--4c36d63ec9059b758090ccf9f380384984b17f7d/Z_QyC0chosqm-pdy0jVPA.png" filename="Z_QyC0chosqm-pdy0jVPA.png" filesize="56353" width="700" height="412" previewable="true" caption="Illustration of RabbitMQ resource allocation per message in a normal and a lazy queue">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjUyLCJwdXIiOiJibG9iX2lkIn19--e12d57ce69f89dbe5ade4ef4d51c40a937d610f1/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/Z_QyC0chosqm-pdy0jVPA.png">
<figcaption class="attachment__caption">
Illustration of RabbitMQ resource allocation per message in a normal and a lazy queue
</figcaption>
</figure></action-text-attachment><strong>With those features we can create reliable job queues which can handle larger payloads.</strong> Now we can write our workers as lightweight Ruby processes which have no database access, as they can now receive all data needed for them to process a job through the queue without a large performance or memory penalty.</div><div>There is still one feature of Sidekiq I haven't mentioned — the UI. Sidekiq's UI is useful for monitoring the health of your jobs, and general throughput.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI2Mj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--eb3a3bde33e88a337106b4a265e0318bff413cdc" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZ1lCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--d51940136a2bd36b1d9f9f7b88b487aac6076b20/2S8GT8C0hkpfZzL_i6CiDQ.png" filename="2S8GT8C0hkpfZzL_i6CiDQ.png" filesize="537125" width="1000" height="615" previewable="true" caption="The Sidekiq UI">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjYyLCJwdXIiOiJibG9iX2lkIn19--dab03c596b41c6f25c2468d4d3dead312b973b4a/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/2S8GT8C0hkpfZzL_i6CiDQ.png">
<figcaption class="attachment__caption">
The Sidekiq UI
</figcaption>
</figure></action-text-attachment><strong>RabbitMQ also comes with a powerful UI</strong> which provides insight not only into the number of failed and succeed jobs, but also into global and per-queue system resource (disk, RAM, …) usage, the ability to consume and publish messages directly from the UI, consumer and user management, and a login screen… which is premium feature in Sidekiq and costs $1950 per year (and there is a limit of 100 workers for that price).<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI1Mz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--c9e09951e65e196eceaf5bd0b1b7c9a679e5e8c4" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZjA9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--3a4aac3b15de95da35da5011b3108e926a28cab6/ekcsvlnTnmMqu1uwOLOFUw.png" filename="ekcsvlnTnmMqu1uwOLOFUw.png" filesize="123781" width="1000" height="614" previewable="true" caption="RabbitMQ Management Console">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjUzLCJwdXIiOiJibG9iX2lkIn19--9c9cff9bc12db4f542233db55a28c9dad30121e3/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/ekcsvlnTnmMqu1uwOLOFUw.png">
<figcaption class="attachment__caption">
RabbitMQ Management Console
</figcaption>
</figure></action-text-attachment>
</div><h2>Exchanges</h2><div>AMQP defines the concept of exchanges. Exchanges can be thought of as routers. <strong>When a message is published to an exchange, the exchange determines which queues should the message be delivered to.</strong> It's important to note that it's impossible to put a message directly into a queue. Even if a message is published directly to a queue, a temporary exchange will be created to deliver it to the queue.<br><br><strong>There are four types of exchanges supported by RabbitMQ.</strong><br><br>The most commonly used exchange type is the <a href="https://www.rabbitmq.com/tutorials/amqp-concepts.html#exchange-direct"><strong>direct exchange</strong></a>. It directly delivers all messages to a single queue bound to it. <strong>It's a 1-on-1 mapping of an exchange to a queue.</strong> If applied to a chat application, a direct exchange would deliver messages from a chat room to a single user.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI1ND9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--2afe1ca7d596a4ec72ee4ad41f92c3dc4ad267f9" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZjQ9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--3aa98470ab147b3697c134c52cfd0d0b9572fab6/09G-3e6JMVQ5C5bkj-Cydg.gif" filename="09G-3e6JMVQ5C5bkj-Cydg.gif" filesize="31905" width="700" height="412" previewable="true" caption="Animation of a direct exchange being utilized in a chat app">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjU0LCJwdXIiOiJibG9iX2lkIn19--35e928bd5919c1f35087cea1baeffc4437bbe0c1/09G-3e6JMVQ5C5bkj-Cydg.gif">
<figcaption class="attachment__caption">
Animation of a direct exchange being utilized in a chat app
</figcaption>
</figure></action-text-attachment>Another kind of exchange is the <strong>fan-out exchange</strong>. They deliver messages to all queues bound to them. <strong>It's a 1-to-many mapping of an exchange to multiple queues.</strong> In the example of the chat application, a fan-out exchange would be used to send a message to all users.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI1NT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--680204b6d29a38acde0c34fb90ca3eacbfc80acf" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZjg9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--b3867353d7c1d70b23a38d532e5cd09f3e0d6422/2FeZqAfFqpiHAWMV0g8Rxw.gif" filename="2FeZqAfFqpiHAWMV0g8Rxw.gif" filesize="31864" width="700" height="412" previewable="true" caption="Animation of a fan-out exchange being utilized in a chat app">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjU1LCJwdXIiOiJibG9iX2lkIn19--1eb9eccbb0d80b26c0980a97ad557bc464c78f20/2FeZqAfFqpiHAWMV0g8Rxw.gif">
<figcaption class="attachment__caption">
Animation of a fan-out exchange being utilized in a chat app
</figcaption>
</figure></action-text-attachment>Then there are <a href="https://www.rabbitmq.com/tutorials/amqp-concepts.html#exchange-topic"><strong>topic exchanges</strong></a>. <strong>They deliver messages published to them to a bound queue based on the messages tag/topic and the queue's bound topic.</strong> In the example of the chat application, a topic exchange would be used to direct messages to their corresponding user.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI1Nj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--ec1d2584470806b544fc229014373ace64e95abf" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZ0FCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--6933694c26fbce1c718d9550c3a60fe609ff72bb/f2lzM5oK-DyYdFWfk9rexw.gif" filename="f2lzM5oK-DyYdFWfk9rexw.gif" filesize="48341" width="700" height="412" previewable="true" caption="Animation of a topic exchange being utilized in a chat app">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjU2LCJwdXIiOiJibG9iX2lkIn19--2a0c4fe6eac1a994708f7e61458bf1c06ce4875f/f2lzM5oK-DyYdFWfk9rexw.gif">
<figcaption class="attachment__caption">
Animation of a topic exchange being utilized in a chat app
</figcaption>
</figure></action-text-attachment>Finally, there are <a href="https://www.rabbitmq.com/tutorials/amqp-concepts.html#exchange-headers"><strong>header exchanges</strong></a>. They are a step up from topic exchanges. <strong>Instead of looking at a topic, also called a routing key, they look at the message's headers to determine where a message should be delivered.</strong> Messages in RabbitMQ can have additional attributes associated with them — called headers. Headers determine different behaviors when handling messages. E.g. an "x-match" header indicates to the exchange that either any or all headers have to match a value for it to get routed to a queue. There is also the "reply-to" header which indicates where the result of processing a message should be published to.<br><br><strong>Utilizing exchanges gives many advantages</strong> — exactly once delivery, performance, and ease of deprecation. Utilizing an exchange to deliver your messages is much faster and more reliable than using Ruby to handle that logic. There is also the pragmatic reason of not having code to maintain. The logic is handled by RabbitMQ, you only have to configure it (which can be done through code). Exchange-exchange and exchange-queue bindings can be changed on-the-fly by any client which enables one application to change the behavior of other services. <strong>Personally, exchanges have helped me deploy applications with little-to-no downtime and deprecate services without having to change other services.</strong>
</div><h2><strong>Special features</strong></h2><div>RabbitMQ adds it's own magic on top of AMQP. I have already mentioned header exchanges, which are a non-standard AMQP feature. A feature I personally use a lot is <strong>"</strong><a href="https://www.rabbitmq.com/direct-reply-to.html"><strong>direct reply-to</strong></a><strong>"</strong>. Direct reply-to is a form of synchronous communication between a producer and a consumer. <strong>It enables a producer to publish a message and wait for a consumer to process it and return the result directly to the producer.</strong> It's useful when the result of a message is used in further processes. E.g. IoT devices usually log a heartbeat signal to their server to indicate that they are connected and configured correctly. If we have a smart lock, we can process its heartbeat asynchronously since the result isn't really important for the server nor for the device. But a pin check is important and should be handled synchronously to avoid access permission errors caused by stale data.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI1Nz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--2dd1c6ce14af19e7380807402dce721d0dffbbe2" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZ0VCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--c0dc20c2af47dc554839cdc1eda613c35daf9ff0/XCIe9E2HlbFYxRPEIwsWRg.gif" filename="XCIe9E2HlbFYxRPEIwsWRg.gif" filesize="72709" width="700" height="412" previewable="true" caption="Animation of utilizing a direct reply-to message">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjU3LCJwdXIiOiJibG9iX2lkIn19--d97518c107010c242cdc117bfb30339dc29e7467/XCIe9E2HlbFYxRPEIwsWRg.gif">
<figcaption class="attachment__caption">
Animation of utilizing a direct reply-to message
</figcaption>
</figure></action-text-attachment>Another feature that I often utilize is <strong>"</strong><a href="https://www.rabbitmq.com/dlx.html"><strong>dead lettering</strong></a><strong>"</strong>. It allows for a message to be re-queued automatically in case it gets rejected from an exchange or queue. <strong>This feature is useful for error handling</strong>, exponential back-off, scheduled message processing …<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI1OD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--bb261753cccd383b2d35bd4d755b2f841d6e2f86" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZ0lCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--5ad90793b044946d7324edaf946e711461498dd3/fJWeNUxNbDgCzK162fs02Q.gif" filename="fJWeNUxNbDgCzK162fs02Q.gif" filesize="45344" width="700" height="412" previewable="true" caption="Animation of utilizing a dead letter queue">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjU4LCJwdXIiOiJibG9iX2lkIn19--4ddf36fe298f6e3b244750390151a1bbe503b2b3/fJWeNUxNbDgCzK162fs02Q.gif">
<figcaption class="attachment__caption">
Animation of utilizing a dead letter queue
</figcaption>
</figure></action-text-attachment><strong>"</strong><a href="https://www.rabbitmq.com/ae.html"><strong>Alternate exchange</strong></a><strong>"</strong> is a useful feature for deprecating services. <strong>It specifies to which exchange a message should get sent to in the case that the primary exchange rejects it.</strong><action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI1OT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--a6c5e8f8f8d2f1a3b78c04f4637cd22fa3c26399" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZ01CIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--e59b3ce4562efffb86ab73fa8f1e860e17286b20/OgY1IocLzrFHuPrm2FSaPg.gif" filename="OgY1IocLzrFHuPrm2FSaPg.gif" filesize="58502" width="700" height="412" previewable="true" caption="Animation of utilizing an alternate exchange">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjU5LCJwdXIiOiJibG9iX2lkIn19--a93fd9719fe359fe8e237cd4b81d96f9d8d18c69/OgY1IocLzrFHuPrm2FSaPg.gif">
<figcaption class="attachment__caption">
Animation of utilizing an alternate exchange
</figcaption>
</figure></action-text-attachment>Then there are <strong>"</strong><a href="https://www.rabbitmq.com/priority.html"><strong>priority queues</strong></a><strong>"</strong> and <strong>"</strong><a href="https://www.rabbitmq.com/consumer-priority.html"><strong>priority consumers</strong></a><strong>"</strong>. Priority queue are CS standard priority queues. Meaning that messages in the queue have a priority (ranging from 0 to 255), <strong>messages with higher priority get processed first</strong>. Which is useful for the same reasons as "direct reply-to", but it's asynchronous. While <strong>priority consumers are a form of fail-over</strong>. Priority consumers will be served messages if they are active. In the case that all priority consumers are inactive, other consumers will get served messages.<br><br>Finally there is <strong>"</strong><a href="https://www.rabbitmq.com/ttl.html"><strong>TTL</strong></a><strong>"</strong>. It specifies how long a message lives. <strong>If a message outlives its TTL it's automatically rejected from the queue.</strong> Though, this feature comes with a caveat — <strong>this rule can only be enforced for the message at the front of the queue</strong>. E.g. if you put two messages in a queue the first with a TTL of 300 and the second with a TTL of 100. Both would be in the queue until the one with a TTL 300 expires, because it is in front. The moment it expires, the second message is at the front and automatically expires since it's TTL has passed. This seems harmless, but can cause a lot of problems when combined with dead lettering to achieve e.g. offset delivery.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI0Nz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--90c9c8023ebaf8b1b9df62cce6d17fe9a48f337c" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZmM9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--b0cb479bba0b3fa60e52833db2fe822749052947/C9ic606EeV0_Sw5a4DaDpQ.gif" filename="C9ic606EeV0_Sw5a4DaDpQ.gif" filesize="64273" width="700" height="412" previewable="true" caption="Animation of TTL messages combined with dead lettering">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjQ3LCJwdXIiOiJibG9iX2lkIn19--48e0f07f1603e8cd28fceec340f89d38d13ec148/C9ic606EeV0_Sw5a4DaDpQ.gif">
<figcaption class="attachment__caption">
Animation of TTL messages combined with dead lettering
</figcaption>
</figure></action-text-attachment>
</div><h2>Plugins</h2><div>For me, this is the most important feature of RabbitMQ. <strong>With plugins you can add any functionality you want to RabbitMQ</strong>. The best example of this is the management console which is a plugin, and must be enabled before use.<br><br><strong>Through plugins, RabbitMQ supports not only AMQP, but STOMP, MQTT and WebSockets as communication protocols.</strong><br><br>Then there is the <strong>Federation plugin</strong>. It enables RabbitMQ to run several isolated clusters or instances which can communicate with one-another. It is similar to the way that <a href="https://mastodon.social/about">Mastodon</a> works. All users, no matter which server they signed up to, can communicate with one-another. <strong>Federations are useful for handling large workloads.</strong> E.g. if you handle logs from a lot of different machines through RabbitMQ, that can be handled by one federated cluster, while everything else is handled by another federated cluster. That way you can scale those two cluster independently depending on their workload, and you avoid the noisy neighbor problem (when a highly taxed service slows the whole system down).<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI2Mz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--29ad575439d41f7a928bf499377c765e6dec1e9b" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZ2NCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--60b5030550e808099f13021ba0be019efb34597e/wLKNGWu84jQdccQFtUgoRQ.png" filename="wLKNGWu84jQdccQFtUgoRQ.png" filesize="59617" width="700" height="427" previewable="true" caption="Example RabbitMQ Federation for an IoT application">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjYzLCJwdXIiOiJibG9iX2lkIn19--74a9ee28c932844a1212b2dc113dbbfb519acfbc/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/wLKNGWu84jQdccQFtUgoRQ.png">
<figcaption class="attachment__caption">
Example RabbitMQ Federation for an IoT application
</figcaption>
</figure></action-text-attachment>
</div><h2>Conclusion</h2><div>
<strong>Replacing Sidekiq with RabbitMQ provides many advantages when it comes to debuggability, scaling, fault tolerance and memory consumption.</strong> It supports multiple industry-standard message queue protocols and can be used as a drop in replacement for other "background worker" libraries.<br><br><strong>If you need a queue that guarantees job execution and persistence, go with RabbitMQ instead of Sidekiq.</strong> There are some features that are missing in Rabbit, like cron jobs and unique jobs, but they can be added by the clients. RabbitMQ offers a plethora of features which, if not useful at first, will become useful later as they will help grow a monolith to a services oriented architecture.<br><br>To get started take a look at projects like <a href="https://github.com/jondot/sneakers">Sneakers</a> (background jobs) and <a href="https://github.com/ruby-amqp/bunny">Bunny</a> (AMQP client), read through the <a href="https://www.rabbitmq.com/tutorials/amqp-concepts.html">basic concepts page</a>, and lastly there is the <a href="https://www.rabbitmq.com/documentation.html">manual</a>. If you are using Ruby on Rails, <a href="https://github.com/jondot/sneakers/wiki/How-To:-Rails-Background-Jobs-with-ActiveJob">Sneakers integrates with ActiveJob</a> which eases the transition.<br><br><strong>Sidekiq isn't useless!</strong> If your project doesn't require execution or persistence guarantees, or if you hold a Sidekiq Pro license, I would recommend you stick with it for the time being. While I disagree with hiding essential features (like guaranteed execution, and rolling restarts) behind a paywall, <strong>a Pro license offers support and additional business oriented features which you won't get with a self-hosted RabbitMQ instance.</strong>
</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/62018-10-20T00:00:00Z2023-11-14T16:28:04ZLicensing software<div class="trix-content">
<div>
<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI0MD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--cf4639cc101b586a108238dae9418494d8e4a64f" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZkE9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--40044bef25ca697b2a881979f5e21cd7d17721f9/OQBL7rd0Y9yFN92dSkV1Uw.jpeg" filename="OQBL7rd0Y9yFN92dSkV1Uw.jpeg" filesize="1055264" width="4000" height="1734" previewable="true" caption="My messy desk">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjQwLCJwdXIiOiJibG9iX2lkIn19--86bc3ab02581a75cc78ff353e4056d319b7940b0/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/OQBL7rd0Y9yFN92dSkV1Uw.jpeg">
<figcaption class="attachment__caption">
My messy desk
</figcaption>
</figure></action-text-attachment>Recently I've started working on a small Ruby library. While I was sketching the architecture of it to I was listening to some lectures from Richard M. Stallman which got me thinking about how I should license my library.<br><br>Note: I'm by no means a legal expert. Everything written here is what I've found while researching the issue. Some things may not apply to the region you live in.</div><h2>Why is licensing important?</h2><div>
<strong>Unless you provide a license with your code, the code (legally) can't be redistributed by others.</strong> It falls under your personal copyright, and only you are allowed to redistribute it, authorize copies, modify it, etc.<br><br>I wasn't aware of this until, one day, I got an email from a complete stranger asking if I could add a license to an old project of mine so that his company could fork and modify it to their liking.<br><br>This caught me off-guard, I always thought that public, unlicensed, code is free for the taking. Much like things in the public domain. But, it turns out that code (just like essays, books, poems, songs, …) belong to the person who wrote them unless there is a contract specifying otherwise (e.g. a license or a contract with the company you work for). This same issue was covered by Jeff Antwood back in 2007 with the post "<a href="https://blog.codinghorror.com/pick-a-license-any-license/">Pick a License, Any License</a>".<br><br>For that reason you'll often hear something along the lines of <em>"Experienced developers won't touch unlicensed code because they have no legal right to use it."</em>
</div><h2>How to choose a license for your project?</h2><div>Choosing the right license four your project can be a daunting task. <strong>A license determines which rights you hold/give up as the author and under what conditions others can use your project</strong> — you can think of it as your business model, of sorts.<br><br>Licenses are often written in a confusing to read, but legally correct, manner which means it takes a couple of readings before you understand the full legal implications of the text. Since I've seen at least 20 different software licenses over the years, and I didn't want to spend the time to read all of them through <strong>I've decided to narrow my efforts to the most popular ones and research those.</strong>
</div><h2>What to choose?</h2><div>To find out which licenses are most popular in the community I decided to look at GitHub's data on Google's BigQuery platform. GitHub catalogs all repositories by the license they use. It's far from perfect as some projects have multiple-licenses (e.g. <a href="https://github.com/rust-lang/rust/blob/c57deb923e56e04634b7e4b3fdc0f84ea0d41ed4/COPYRIGHT">Rust</a>) and some have custom licenses (e.g. <a href="https://github.com/ruby/ruby/blob/9cbe4553f4e25d0a9beb4b6393b601c1d0ab3a17/COPYING">Ruby</a>) which will be catalogued as unlicensed, but the data should be indicative of a trend.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzIzNz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--182061a85e25744a55286507028be2f44820df2d" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZTA9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--2e75bacdf86298f1f7b5f2ada2cd085babdb25cb/D03sRAEqi3hZGi0v_18dIg.png" filename="D03sRAEqi3hZGi0v_18dIg.png" filesize="24960" width="2664" height="736" previewable="true" caption="Representation of licenses in all GitHub projects as a percentage">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjM3LCJwdXIiOiJibG9iX2lkIn19--abbe6112eaaf076d71063b74f291dc84b5244696/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/D03sRAEqi3hZGi0v_18dIg.png">
<figcaption class="attachment__caption">
Representation of licenses in all GitHub projects as a percentage
</figcaption>
</figure></action-text-attachment><strong>The above graph represents the percentage of repositories using a particular license.</strong><br><br>To get that data from BigQuery I've used the following query to extract the count of projects using a particular license for C, C++, Elixir, Haskell, Java, JavaScript, Kotlin, PHP, Python, Ruby & Rust to construct the data-set.</div><pre>SELECT licenses.license AS license, COUNT(licenses.repo_name) AS count
FROM [bigquery-public-data:github_repos.licenses] AS licenses
JOIN [bigquery-public-data:github_repos.languages] AS languages ON licenses.repo_name = languages.repo_name
WHERE languages.language.name = ''
GROUP BY license</pre><div>It's visible form the graph that <strong>the five most popular licenses are (in order) MIT, Apache-2.0, GPLv2, GPLv3 and BSD-3-clauses.</strong> It's also visible that there is a relatively small amount of unlicensed, custom-licensed and multi-licensed repositories.<br><br>Now that our search has been narrowed down to only five licenses I can read and research each one of them and specify a use-case for each.</div><h2>MIT</h2><div>
<a href="https://opensource.org/licenses/MIT">The MIT license</a> is by far <strong>the most popular license with about 48% of all repositories on GitHub adopting it</strong>. It's also the simplest of the five.<br><br><a href="https://twitter.com/dhh">DHH</a> (David Heinemeier Hansson) once translated it nicely as "<a href="https://www.flickr.com/photos/rooreynolds/243810133/"><strong>I don't owe you shit</strong></a>".<br><br>The license states that the person using the project has to include the license in any derivative work (e.g. forks — like in the case of <a href="https://github.com/jeremyevans/roda/blob/d36844117a6dc33794826b92c6e24f4151bc34cb/MIT-LICENSE">Roda</a> which is a fork of <a href="https://github.com/soveran/cuba">Cuba</a>), that they are free to do what ever they like with your code and that you as the author can't be held liable for any bugs or unintended side-effects.<br><br>That means that people are free to include your code in their projects (provided the copyright/license notice), you can't be held liable for any kind of unwanted behavior, and that nobody is required to upstream or open-source any bug fixes or changes they make.<br><br><strong>This license is extremely permissive making it an excellent choice for libraries, as well as applications.</strong> It's mostly catered to open-source projects. Though, in my opinion and depending on your desires, GPL may be better for applications. Note that with MIT any one can make a closed-source derivative work, this makes MIT really popular as it doesn't force the end-user's choice.<br><br><strong>There is a gotcha' when working with the MIT license</strong>, <a href="https://github.com/VSCodium/vscodium#why-does-this-exist">one exploited by Microsoft's Visual Studio Code</a>. <strong>MIT only covers the source code, not the compiled or binary version</strong> which enabled developers to add additional clauses to the release version of the app/library.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzIzOD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--5b545083b93416896d4358b0a3f4ed60a4d35f59" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZTQ9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--26993dd3184a9ffddb44875ddb4bbe7ebe66d644/y6vD5b6zkq9Da6JXq6tbbw.png" filename="y6vD5b6zkq9Da6JXq6tbbw.png" filesize="279835" width="2000" height="1696" previewable="true" caption='License popularity by language, expressed as a percentage; EDIT: "Unlicensed" is actually the Unlicense license'>
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjM4LCJwdXIiOiJibG9iX2lkIn19--94d316a9ee177ba2a8f7014ea31c71b193048329/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/y6vD5b6zkq9Da6JXq6tbbw.png">
<figcaption class="attachment__caption">
License popularity by language, expressed as a percentage; EDIT: "Unlicensed" is actually the Unlicense license
</figcaption>
</figure></action-text-attachment>The MIT license is extremely popular in all of the languages I sampled but Java, Kotlin & Haskell where Apache-2.0 and BSD-3-clauses were more popular. And even in those languages, MIT was on second place. <strong>I'd attribute it's popularity to it's do-what-you-want approach to licensing.</strong>
</div><h2>Apache-2.0</h2><div>
<a href="https://opensource.org/licenses/Apache-2.0"><strong>Apache 2.0</strong></a><strong> is more complicated in legal terms than MIT, but basically sets the same expectations.</strong> The license must be provided with each copy of the source. The author(s) can't be held liable in any way, shape or form. It grants the end-user full rights regarding patenting on any contributed code given it doesn't infringe on any patents.<br><br>The biggest difference from MIT is that <strong>any modifications done to the original project have to be redistributed with a notice that the files have been modified.</strong> And <strong>a warranty can be applied</strong> to the project (e.g. for a fee).<br><br><strong>It's quite permissive making it an excellent choice for both libraries and applications.</strong> Especially given the patents and warrant clause, which makes it more suitable for commercial use.<br><br>Same as with MIT, Apache-2.0 allows end-users to modify your project and redistribute it under minimal terms, without the need to open-source or upstream modification.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzIzOT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--1b57081a0863a82edd7a8d97d7b5681ea025123c" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZTg9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--239ac3770fc6db977da47016a6e8ee25e8ef8332/pE72mKU20oWLCGSFTVfXVw.png" filename="pE72mKU20oWLCGSFTVfXVw.png" filesize="184730" width="2000" height="1133" previewable="true" caption="License popularity in Java & Kotlin projects, expressed as a percentage">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjM5LCJwdXIiOiJibG9iX2lkIn19--f6b4c5580e771fd67a18223e019582e76ac18b4c/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/pE72mKU20oWLCGSFTVfXVw.png">
<figcaption class="attachment__caption">
License popularity in Java & Kotlin projects, expressed as a percentage
</figcaption>
</figure></action-text-attachment>Interestingly enough, Apache-2.0 is the most popular license for Java and Kotlin projects. I'd guess that because <a href="https://github.com/apache">the Apache Foundation predominately uses Java for their projects</a> this drives up the number of Java projects with that license. But' I can't explain exactly why these two languages are the outliers.</div><h2>GPLv2 & GPLv3</h2><div>Here <strong>I'm going to cover only </strong><a href="https://www.gnu.org/licenses/rms-why-gplv3.html"><strong>GPLv3</strong></a>, <a href="https://www.gnu.org/licenses/rms-why-gplv3.html">as v2 and v3 are much the same (yet incompatible) expect for patents and a loop hole</a>.<br><br>The GPL family of licenses comes from the <a href="https://en.wikipedia.org/wiki/Free_Software_Foundation">Free Software Foundation</a>, an organization started by Richard M. Stallman after an, now famous, <a href="https://en.wikipedia.org/wiki/Richard_Stallman#Events_leading_to_GNU">incident with a printer that often jammed</a>.<br><br>This family of licenses is simultaneously loved & frowned upon by many in the Free/Libre and OpenSource community. <strong>It's controversial because it requires the end user to open-source, as well as provide build and installation instructions for their project</strong> if they use any GPL licensed code in it. Else, the license is quite similar to Apache-2.0. The author(s) can't be held liable, you can patent your work, redistribute and modify the original work (with a notice).<br><br><strong>The point of the license is to guarantee the </strong><a href="https://en.wikipedia.org/wiki/Free_software#Definition_and_the_Four_Freedoms"><strong>four basic freedoms of software</strong></a><strong>:</strong> 1. The freedom to run the program for any purpose 2. The freedom to study how the program works, and change it to make it do what you wish 3. The freedom to redistribute and make copies 4. The freedom to improve the program, and release your improvements to the public<br><br>The GPL licenses are also known as <a href="https://en.wikipedia.org/wiki/Copyleft">Copyleft licenses</a> because they give permission to the end-user to freely distribute and modify the intellectual property but require that the same rights be preserved in any derivative works. Meaning that any software licensed under it preserves the four freedoms for the end-user and any of their derivative works and their end-users as well.<br><br><strong>Some people don't agree with the clause that your code has to be open-sourced.</strong> This is a topic for another post. <strong>This boils down to the difference between free and open-source software.</strong> In a nutshell, free software protects and upholds the 4 basic freedoms of software, while open-source software only makes the software's source publicly accessible. In other words, you can view the code of an open-source, but you can't necessarily use, modify, or redistribute it. While free software allows you to do what you want as long as the result is again free software.<br><br><strong>A common misconception is that this license prohibits commercial use.</strong> The opposite is true, it encourages it, though you have to change your conception of commercial software. Instead of selling the compiled program, you can refocus on selling support, patches, new versions, etc. The essay "The Magic Cauldron" from the book "<a href="https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar">The Cathedral & the Bazaar</a>" covers the commercial aspect of free & open-source development in great detail, for anyone interested.<br><br><strong>The GPLv3 is best applicable for applications. This doesn't mean that it's bad for libraries!</strong> Applications can greatly profit from this license as it encourages upstreaming of fixes and distribution of modifications. <strong>Libraries can also enjoy all the benefits that applications can</strong>, but, because of it's Copyleft nature allications using such libraries must dynamically load them, which may make some end-users reluctant. It depends on your goal and beliefs whether the GPL is for your library. <a href="https://www.gnu.org/licenses/why-not-lgpl.html">The FSF covers this decision in this article</a>.<br><br><strong>The GPL has a </strong><a href="https://opensource.org/licenses/LGPL-3.0"><strong>semi-permissive license called LGPLv3</strong></a>. It's much the same as MIT, but with the limitation that prominent attribution has to be given to your project in any combined work, that all modification have to be open-sourced (under some conditions) and clearly marked. It still can force the end-user to open-source the whole combined worked if it's statically linked into the derivative, but it can be dynamically linked without repercussions.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI0MT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--637c654d322e9fe15d9a0f5c2e5557699b5fba57" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZkU9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--33002893481ddb5cd55037157d487bf63be8125e/KPEKttPShuuYAn0aIXRcDQ.png" filename="KPEKttPShuuYAn0aIXRcDQ.png" filesize="175156" width="2000" height="1143" previewable="true" caption="License popularity in C & C++ projects, expressed as a percentage">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjQxLCJwdXIiOiJibG9iX2lkIn19--f2118bb24410eb82c588bc1cfe3207590df48784/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/KPEKttPShuuYAn0aIXRcDQ.png">
<figcaption class="attachment__caption">
License popularity in C & C++ projects, expressed as a percentage
</figcaption>
</figure></action-text-attachment>
</div><h2>BSD-3-clauses</h2><div>
<a href="https://opensource.org/licenses/BSD-3-Clause"><strong>BSD-3-clauses</strong></a><strong> is a permissive license.</strong> <strong>It's quite similar to the MIT license</strong> in the sense that the author(s) hold no liability, the license has to be included in all derivative work and similarly to Apache-2.0 it <strong>allows a warranty to be applied</strong>.<br><br>The biggest difference from the other licenses is that <strong>it forbids the end user to promote their project based on the use of the licensed project or it's author(s)</strong> (without prior permission).<br><br>Note that <a href="https://opensource.org/licenses/BSDplusPatent">there exists a <strong>-patents</strong> version of the license</a> which grants the end user the ability to patent their work, same as Apache 2.0. But this license is somewhat controversial after <a href="https://hackernoon.com/facebooks-bsd-patents-license-and-how-it-affects-you-66088e052845">Facebook added very strict patent clauses to their BSD license in the React project</a>, escalating so far that the project got re-licensed to MIT.<br><br><strong>BSD-3-clauses is excellent for both libraries and applications</strong>. It's not as verbose or popular as Apache-2.0, yet it's as permissive as MIT, the biggest differentiating factor is it's restriction on marketing.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI0Mj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--2de468e90d0f862a7e4a012b93463932667efb0d" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZkk9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--247b8b91c5ec893d3d272e10f3f43e3b3a7e9546/a4MJwC-Vb9EkCwvyD_KVmQ.png" filename="a4MJwC-Vb9EkCwvyD_KVmQ.png" filesize="89489" width="2000" height="540" previewable="true" caption="License popularity in Haskell projects, expressed as a percentage">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjQyLCJwdXIiOiJibG9iX2lkIn19--e53adb2e1d0ccb774b85777321c7d8227e928bd0/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/a4MJwC-Vb9EkCwvyD_KVmQ.png">
<figcaption class="attachment__caption">
License popularity in Haskell projects, expressed as a percentage
</figcaption>
</figure></action-text-attachment>It's interesting that out of all the sampled languages that Haskell's most popular license is BSD. A friend of mine indicated to me that this could be caused by the fact that their project generator uses it by default.</div><h2>Clauses</h2><div>
<strong>Software licenses can come with clauses which can change the base license</strong>. The most recent example of this is the addition of <a href="https://commonsclause.com/">the Commons Clause</a> to some Redis Labs modules/products.<br><br><strong>Clauses are envisioned to fulfill the author's additional wishes or change some behavior of the base license.</strong><br><br>Perhaps the most well known example is GPL and LGPL. Where LGPL is basically a clause, and requires, the full GPL.</div><h2>Conclusion</h2><div>Picking a license is hard. It boils down to what you want other people to do with your work.<br><br><strong>If you really don't care and just want to get your project out there for people to use however they wish</strong>, even if they choose to sell basically your project — then MIT is for you.<br><br><strong>If you also want to grant your users the ability to make patents off your derivative work</strong>, Apache-2.0 is the way to go.<br><br><strong>If in addition you want to protect the authors'/contributors' privacy and control how people can use your project to market theirs</strong>, then BSD is the choice for you.<br><br><strong>If you want to protect the users' freedom or you want your project to forever stay free & open-source</strong>. Then the GPL family is your best choice.<br><br><strong>I usually use the MIT license only for libraries</strong> and GPLv3 for applications, as I don't want to force people to open-source their projects, but after doing this research I think I'll choose LGPL for my Ruby library. In my opinion it's a good compromise between staying free & open-source while still giving the user the ability not to open-source their work.<br><br>While reasearching this topic I've found a hand full of useful resources to do quick checks, they can't really replace reading the license text, but they give an insightful overview:<br><br>I have also stumbled upon a few hilarious (yet functional) licenses:<br><br>The data used to generate the statistics can be found here. It was gathered on the 25th of April, 2018 — it shouldn't be affected by the recent migration of projects from GitHub after the acquisition from Microsoft.</div><h2>Epilogue</h2><ul>
<li>
<a href="https://opensource.org/licenses/alphabetical">https://opensource.org/licenses/alphabetical</a> — list of all known licenses</li>
<li>
<a href="https://tldrlegal.com/">https://tldrlegal.com/</a> — quick bullet-point overview of most licenses</li>
<li>
<a href="https://choosealicense.com/">https://choosealicense.com/</a> — license picking decision tree</li>
<li>
<a href="https://spdx.org/licenses/Beerware.html">BeerWare</a> — If you think the software is good, and you meet the author, you can repay them with beer</li>
<li>
<a href="http://www.wtfpl.net/">Do What the Fuck You Want</a> — the name says it all...</li>
<li>
<a href="https://spdx.org/licenses/Unlicense.html">The Unlicense</a> — ironically enough, it's a license that aims to abolish any license and copyright clauses</li>
</ul>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/72019-01-02T00:00:00Z2023-11-14T16:28:04ZGraphQL file upload with Shrine<div class="trix-content">
<div>
<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzIyNT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--7f7fa96bda9490c37beda742087490b06bc18834" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZUU9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--d18d985c30cfbec03c3af44eabaae3b92d4c1bbd/9719cecb0e0383439f4d62f48bc1b6f8.jpg" filename="9719cecb0e0383439f4d62f48bc1b6f8.jpg" filesize="764670" width="3456" height="1486" previewable="true" caption="The GraphQL logo overlayed over a Japanese shrine">
<figure class="attachment attachment--preview attachment--jpg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjI1LCJwdXIiOiJibG9iX2lkIn19--322979139475de7248b1e93eb3824faae461539f/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--0ab9331364f3a49361e292e4e525962adf38ddb2/9719cecb0e0383439f4d62f48bc1b6f8.jpg">
<figcaption class="attachment__caption">
The GraphQL logo overlayed over a Japanese shrine
</figcaption>
</figure></action-text-attachment>At the moment of writing there is no officially supported way to do file upload through <a href="https://en.wikipedia.org/wiki/GraphQL">GraphQL</a>. Here is a roundup of all available methods to do file upload through it, their pros and cons.<br><br>This post grew out of a request on the <a href="https://github.com/shrinerb/shrine">Shrine</a> issue tracker — you can find <a href="https://github.com/shrinerb/shrine/issues/244">the original issue here</a>. It's still useful for other libraries, but the code examples will only apply to Shrine.</div><h2>Direct file upload</h2><div>
<strong>Handling uploads is a resource intensive task for the server</strong> (using up IO and bandwidth). To resolve this, <strong>we can upload the file directly from the client to our block-storage server</strong> (<a href="https://aws.amazon.com/s3/">AWS S3</a> or <a href="https://www.digitalocean.com/products/spaces/">DigitalOcean Spaces</a>) by using a presigned URL generated on our server.<br><br>This is in contrast to the classic approach where we would first upload the client's file to our server, and then from our server to the block-storage server.<br><br>The benefit is that <strong>our server doesn't have to process the file</strong> (which uses up IO) and <strong>it doesn't have to upload the file</strong> to the block-storage server (more IO usage). It also solves scaling issues related to distributed file caches and distributed resource management.<br><br><strong>The downside to this method is apparent when the uploaded file needs to be processed</strong>. To process a file our server needs to get the client's form submission containing the URL of the file, download the file, process it and re-upload it to the block-storage server. This uses up bandwidth and IO but it's necessary.<br><br><strong>This method should be satisfactory for most use-cases</strong>. If your application doesn't require immediate file processing then the downloading and re-uploading can be done in the background or on a separate machine ( <a href="https://github.com/ysugimoto/aws-lambda-image">using an AWS lambda</a>). On AWS you can get the best of both world using the <a href="https://aws.amazon.com/solutions/implementations/serverless-image-handler/">Serverless image handler</a>.<br><br><a href="https://github.com/shrinerb/shrine/blob/3c14113ce2fb77d1f3e96bd4e4ff10416651fc7c/doc/direct_s3.md"><strong>Shrine supports this feature out-of-the-box</strong></a> through the <a href="https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/presign_endpoint.rb">presign plugin</a>. To get it running you'll need to add the <a href="https://github.com/aws/aws-sdk-ruby">AWS S3 SDK</a> to your Gemfile, configure a new storage for Shrine, and expose the file presign endpoint — this is explained in detail in the documentation.</div><pre>#!/usr/bin/ruby
# This code is license under the MIT license
# Full text: https://opensource.org/licenses/MIT
# Author Stanko K.R.
# Usage:
# ```ruby
# user.avatar = S3UrlToShrineObjectConverter.call(
# arguments[:input][:avatar][:url],
# name: arguments[:input][:avatar][:filename]
# )
require 'uri'
class S3UrlToShrineObjectConverter
def self.call(*args)
new(*args).call
end
def initialize(url, name: nil)
@url = url
@name = name
end
def call
return unless object&.exists?
{
id: id,
storage: storage,
metadata: {
size: object.content_length,
filename: name || id,
mime_type: object.content_type
}
}
end
protected
attr_reader :url
attr_reader :name
private
def object
@object ||= bucket.object(uri.path)
end
def uri
@uri ||= URI(url)
end
def bucket
@bucket ||= begin
s3 = AWS::S3.new({}) # TODO: configure this
s3.buckets['your_bucket_name'] # TODO: load this from the app's config
end
end
def id
@id ||= uri.path.split('/', 2).last
end
def storage
@storage ||= uri.path.split('/', 2).first
end
end</pre><div>Once the client uploads the file it needs to send some kind of reference to the server — the URL of the file, or an ID, or the path of the file (I usually send the original file name as well as the URL).</div><pre>mutation SetAvatar($url: String, $filename: String) {
updateUser(
id: "some_user_uuid"
input: {
avatar: {
filename: $filename
url: $url
}
}
) {
avatar_url
}
}</pre><div>The server uses this reference to build a Shrine object and attach it.<br><br>Note that you can use the <a href="https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/upload_endpoint.rb">upload_endpoint plugin</a> instead of the <a href="https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/presign_endpoint.rb">presign_endpoint plugin</a>. I advise against this as it enables anyone to upload any content, at any time, to your block-storage. By presiging we solve the "anyone” and "anytime” issues — we have control over who can request presigned URLs to upload or download files.</div><h2>Base64 encoding</h2><div>This method is <strong>the simplest, yet it's the worst</strong> for real-world applications.<br><br>Your client can encode the file's data as Base64 and set it as the value of a mutation's field.</div><pre>mutation SetAvatar($data: String, $filename: String) {
updateUser(
id: "some_user_uuid"
input: {
avatar: {
filename: $filename
data: $data
}
}
) {
avatar_url
}
}</pre><div>While on the surface this seems harmless, since the image has to be uploaded to the server in one way or another, this approach has many unwanted side-effects.</div><pre>#!/usr/bin/ruby
user.avatar_data_uri = arguments[:input][:avatar][:data]
user.save</pre><div>
<strong>The biggest issue is memory consumption</strong>. Since the image is now part of your request's JSON body it will be parsed as a string, which will drive your memory consumption up. E.g. If you upload ten 5MiB images in one mutation you will have a hash that's at least 50MiB in memory. <strong>Storing whole files in memory can and will crash your application</strong>.<br><br><strong>I advise against using this method</strong>. If you wish to implement it, you can do so using Shrine's <a href="https://github.com/shrinerb/shrine/blob/a8ba2510bf01981175eb2abe7421c1a86c1bcc9d/lib/shrine/plugins/data_uri.rb">data_uri plugin</a>. After you add the plugin to your uploader you will need to set the <strong><field>_data_uri</strong> of your file in the resolver.<br><br>If you decide to use this method be careful to remove the image upload field from your logs, otherwise your log file will contain a copy of all uploaded images.</div><h2>Multipart upload</h2><div>Multipart uploads are the standard way of uploading files through HTTP. They are used by most browsers to do file uploads through forms. <strong>Since GraphQL is just a JSON request we can pass files alongside our request</strong>, but we need logic to interpret them.<br><br>I'll be referencing <a href="https://github.com/jaydenseric">jaydenseric</a>'s <a href="https://github.com/jaydenseric/graphql-multipart-request-spec">multipart request specification</a> as it's supported by several client and server implementations at this moment.</div><pre>mutation UploadGalleryImages {
uploadGalleryImages(
galleryID: "some_gallery_id"
images: [null, null]
) {
id
filename
url
}
}</pre><div>A typical GraphQL request consists of three fields <strong>query</strong>, <strong>variables</strong> and <strong>operationName</strong>. This spec adds a fourth — <strong>map</strong>. The map contains indices and JSON pointers, each index represents an image and each pointer represents the location of the <strong>ActionDispatch::Http::UploadedFile</strong> that holds the corresponding image.<br><br>Say we want to upload two images to a gallery using this method, our GraphQL mutation might look like this:<br><br>Since the server knows that the <strong>images</strong> field is an array of <strong>File</strong> types it will match all <strong>null</strong> values in the array, in order of appearance, with their index in the <strong>map</strong> field and dereference their JSON pointer. If we were to inspect the input params hash we would see the following:</div><pre>#!/usr/bin/ruby
# {
# galleryID: "some_gallery_id",
# images: [
# #<:http::uploadedfile:0x000055fb90c6dbd0>, @original_filename="b.txt", @content_type="text/plain", @headers="Content-Disposition: form-data; name=\"0\"; filename=\"b.txt\"\r\nContent-Type: text/plain\r\n">,
# #<:http::uploadedfile:0x000055fb90c6da18>, @original_filename="c.txt", @content_type="text/plain", @headers="Content-Disposition: form-data; name=\"1\"; filename=\"c.txt\"\r\nContent-Type: text/plain\r\n">
# ]
# }
gallery = Gallery.find(arguemnts[:gallery_id])
arguments[:images].map do |file|
# Image has a Shrine uploader attached to `file_data`
Image.create(file: file, gallery: gallery)
end</pre><div>From here on you can assign those <strong>UploadedFile</strong> objects to any of your own objects as usual. No Shrine plugins needed.</div><h2>Conclusion</h2><div>
<strong>My suggestion is to use the direct upload method</strong> if possible. It's the only method that doesn't require modifications to the GraphQL protocol, it's well understood, and it's used outside of the GraphQL ecosystem (better interoperability with client libraries).<br><br><strong>If direct uploads won't work for you I would take a gamble on multipart uploads.</strong><br><br>Base64 encoding of upload files is problematic at best and should be avoided in my opinion. Though there are limited use-cases where it's implementation speed out ways the issues it has. Though, those use-cases are rare.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/82018-12-25T00:00:00Z2023-11-14T16:28:04ZBest image uploader for Rails — Revisited<div class="trix-content">
<div>
<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzIyNj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--8d08066a37d06fbdeda33d124e05982781c2af7b" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZUk9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--925ecdedabf202197b56ff11fcfbd9d569d25c43/ruby.jpeg" filename="ruby.jpeg" filesize="90694" width="1080" height="517" previewable="true" caption="Hand holding a plastic Ruby gemstone">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjI2LCJwdXIiOiJibG9iX2lkIn19--5737419acd136d37ff0ad189425e52be7db53733/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/ruby.jpeg">
<figcaption class="attachment__caption">
Hand holding a plastic Ruby gemstone
</figcaption>
</figure></action-text-attachment>Three years ago I wrote about <a href="https://infinum.co/the-capsized-eight/best-rails-image-uploader-paperclip-carrierwave-refile">how to choose the right uploader gem for your project</a>. Since the time the original article has been published, all mentioned libraries got updated, one got deprecated, and two new libraries have appeared. I feel it's time to revisit this topic.</div><h2>Why do we use uploader gems?</h2><div>
<a href="http://rubyonrails.org/">Rails</a> handles file-upload natively. When a file gets uploaded to your app it's represented by an <a href="https://web.archive.org/web/20171013094615/http://api.rubyonrails.org/classes/ActionDispatch/Http/UploadedFile.html">UploadedFile</a> object, which is a wrapper around the underlying <a href="https://ruby-doc.org/stdlib-2.5.0/libdoc/tempfile/rdoc/Tempfile.html">Tempfile</a> object that contains the data sent to the server.<br><br>After we get the file we have multiple options what to do with it:</div><ul>
<li>
<strong>Cache it</strong> — if the file is uploaded through a form we want to temporarily store the file in case the form fails to validate, so that the user doesn't have to re-upload it. This lowers bandwidth consumption and improves UX.</li>
<li>
<strong>Process it</strong> — more often than not people want to create smaller or cropped versions of images that users upload. In the case of PDFs / DOCs we might want to create thumbnails. In general we want to process the data somehow.</li>
<li>
<strong>Store it</strong> — we want to store the uploaded file without name clashes and corruption.</li>
<li>
<strong>Forward the data to a file storage server</strong> — to lower bandwidth to our own servers we usually want to store the files on services like <a href="https://aws.amazon.com/s3/">Amazon's S3</a> or <a href="https://www.digitalocean.com/products/spaces/">DigitalOcean's Spaces</a>. Perhaps we also want to send the data to a CDN (like <a href="https://www.cloudflare.com/">CloudFlare</a>) so that the file gets served faster to users all over the world.</li>
<li>
<strong>Stream the file back to users</strong> — if our app also handles serving of the file it would be best if we could stream its contents instead of loading it into memory.</li>
<li>
<strong>Process it on-the-fly</strong> — if we don't know what kind of processing our client's require from the data, we can do it on-the-fly. E.g. different phones can have different resolutions, instead of serving all devices the same sized image a device can request the image best suited for it's display. The image will then be processed to the desired dimensions and passed to the device.</li>
</ul><div><strong>Not to re-invent the wheel, and not to introduce bugs into our codebase, we reach for a gem that will do all or most of those things for us.</strong></div><h2>ActiveStorage & Shrine</h2><div>Since I published the previous article two new “mainstream” libraries have appeared. <a href="http://rubyonrails.org/">Rails'</a> own <a href="https://github.com/rails/rails/tree/master/activestorage">ActiveStorage</a> and <a href="https://github.com/shrinerb/shrine">janko-m's Shrine</a>.</div><h2>ActiveStorage</h2><div>ActiveStorage now represents the standard (or at least the Rails way) for handling file-upload and storage on file servers. It integrates seamlessly with <a href="https://github.com/rails/rails/tree/master/activerecord">ActiveRecord</a>, providing an elegant API to upload, download, delete, and process files.<br><br><strong>All file processing is done on the fly.</strong> This was a gripe I had with Refile in the original article as it made it unusable without a CDN — to cache processed images. <strong>ActiveStorage fixed that by storing all created versions after initial processing.</strong> Note that <strong>only the web app can create a version of the file as it has to sign the file's URL</strong> — which enables it to sanitize and rate-limit clients' requests. And at the time of writing <a href="https://github.com/rails/rails/issues/32236">there are some difficulties with connecting ActiveStorage to a CDN</a>.<br><br><strong>One feature that is lacking is the ability to create custom file processors.</strong> In Rails 5.2 ActiveStorage uses the <a href="https://github.com/minimagick/minimagick">minimagick gem</a> directly to process images, but in the future it will use the <a href="https://github.com/janko-m/image_processing">image_processing gem</a> which can potentially expand the list of available options. But, it's currently only able to process files to images (it can create PDF and video previews). There is no way to re-encode a video file, or to e.g. count the number of words in a document. But it offers many macros for working with images and <a href="https://github.com/jcupitt/ruby-vips">vips</a> support is on it's way.<br><br><strong>ActiveStorage advocates for direct uploads</strong> (uploading directly to the file server), this abolishes the need for caching.<br><br>The biggest issue I've experienced is that <strong>it's coupled to ActiveRecord.</strong> This forces developers' choice of ORM, and makes potential library migrations more difficult. Today it's not uncommon to see <a href="https://github.com/rom-rb/rom">ROM</a> or <a href="https://github.com/jeremyevans/sequel">Sequel</a> in an Rails app.<br><br><strong>I'm glad that Rails introduced a standard solution.</strong> Out-of-the-box file upload was something that was missing in the framework to make it the perfect one-stop-shop. And, in my opinion, ActiveStorage covers 90% of peoples' file uploading and processing needs. With it, you can add file upload to your Rails app in minutes.</div><h2>Shrine</h2><div>
<strong>DISCLAMER:</strong> I know the author of the gem and consider him a friend. Therefore my opinion can be interpreted as biased. I'll try to be as objective as possible.<br><br>In the previous article I named Carrierwave the Swiss army knife of Rails uploaders. This time around that honor goes to <a href="http://github.com/janko-m/shrine">Shrine</a>.<br><br><strong>Shrine takes CarrierWave's idea of uploader classes and refines it further</strong> by introducing a <a href="https://janko.io/the-plugin-system-of-sequel-and-roda/">Roda's / Sequel's plugin system</a> and cleaning up the API.<br><br>The plugin system makes it agnostic to many things like ORMs, frameworks, and other. So-much-so that <strong>it can be used with any Rack app.</strong><br><br><a href="http://shrinerb.com/"><strong>It comes bundled with many plugins</strong></a><strong> which provide a lot of configuration options and compatibility with many libraries.</strong> E.g. both ActiveRecord and Sequel are supported out-of-the-box.<br><br>There exists the ability for <strong>parallel and background file processing.</strong> This not only speeds things up, but makes large file processing (like video encoding) easier to handle.<br><br>Of course, <strong>there's the ability to do direct uploads</strong>, same as ActiveStorage. The difference between the two implementation being the underlying JS library. ActiveStorage rolled their own JS library, while Shrine decided to use Uppy (the most-popular direct-upload library at the time).<br><br><strong>Different file storage servers, and caching is supported out-of-the-box.</strong><br><br>Metadata, such as the original name, extension, file size, checksum and other arbitrary data can be stored alongside the file. This metadata feature also enables the storing of versions (e.g. different sizes/crops of an image or different encodings of a video).<br><br><strong>Something that I've only seen is Shrine is the ability to migrate data around.</strong> E.g. there is a plugin that enables you to move your files from one file-store to another. Or to copy data over from one model to another.<br><br><strong>The only lacking feature is on-the-fly processing.</strong> Though, personally, I have never found that to be the issue. But after ActiveStorage did it I'd like to see the same functionality in Shrine. <a href="https://shrinerb.com/rdoc/files/doc/processing_md.html#label-On-the-fly+processing">Note that it's possible to achieve on-the-fly processing if you integrate with Dragonfly or Cloudinary</a>, but there is no out-of-the-box solution.<br><br>From my experience, people complain that they have to configure Shrine for Rails. I've never found that to be an issue, but to address those people I'd hope the author creates a wrapper gem to ease the integration with sane defaults for most configuration options.</div><h2>Paperclip</h2><div>Sadly, with the advent of ActiveStorage we also saw the deprecation of <a href="https://github.com/thoughtbot/paperclip">Paperclip</a>.<br><br>Paperclip didn't improve much since the last round-up, but it didn't need to. It always was the simplest and quickest solution, and intended only to attach files and light processing.<br><br>Much of what it does, ActiveStorage does better and it's a bundled-in part of Rails. Though, note that <a href="https://github.com/rails/rails/issues/31656">some features like validation are still missing in ActiveStorage</a>. It was a good decision to deprecate the project as it couldn't compete with Rail's own solution.</div><h2><strong>CarrierWave, Refile & Dragonfly</strong></h2><div>When it comes to <a href="https://github.com/carrierwaveuploader/carrierwave">CarrierWave</a>, <a href="https://github.com/refile/refile">Refile</a> and <a href="https://github.com/markevans/dragonfly">Dragonfly</a> I have to admit <strong>I haven't used either of them in quite the while as ActiveStorage and Shrine completely substituted them in my tool belt.</strong><br><br>Those libraries haven't changed much in the last two years.</div><h2>Conclusion</h2><div>In my opinion, today <strong>you have two choices when it comes to file uploaders in Rails</strong> — either ActiveStorage or Shrine.<br><br><strong>ActiveStorage is the perfect solution if you just need to store a file, do some light processing and forget about it.</strong> I'd guess that this is enough for 90% of the projects that exist.<br><br><strong>If your applications requires more advanced processing or storage options, or any kind of custom file processing go with Shrine.</strong> Its plugins system makes it easily extendable, and there already exist quite a few plugins out there.<br><br><strong>CarrierWave, Refile and Dragonfly are by no means dead.</strong> They do have their audiances, but I've found that ActiveStorage or Shrine can do most if not more than those libraries can, so I'd recommend those two over the others for new projects.<br><br><strong>If you aren't using Rails</strong>, then there is only one real option for you — Shrine.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/92018-12-25T00:00:00Z2023-11-14T16:28:03ZFunction composition >> Ruby<div class="trix-content">
<div>
<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzIyNz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--3512fa4be17122014f0acf63d412724d5a2050a5" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZU09IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--00f4d71d49f033237a056b8051d3956ddc9f7df7/1den4tFEx_0cCqbh8Iq1mqg.png" filename="1den4tFEx_0cCqbh8Iq1mqg.png" filesize="25448" width="2250" height="750" previewable="true" caption="Ruby's new function composition syntax">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjI3LCJwdXIiOiJibG9iX2lkIn19--fdb6c0facf57a8e2a034a4a6c025fc0d1048e3a6/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/1den4tFEx_0cCqbh8Iq1mqg.png">
<figcaption class="attachment__caption">
Ruby's new function composition syntax
</figcaption>
</figure></action-text-attachment>Last week <strong>Proc#<<</strong> and <strong>Proc#>></strong> got merged into Ruby 2.6. This opens the door for function composition. Here’s my opinion as to why this is a great leap forward for Ruby and what needs to improve.</div><h2>Composition vs. inheritance</h2><div>
<strong>Ruby is an object-oriented language</strong> (among others), meaning it has the concept of an object and the class of an objects — which has the concept of inheritance.<br><br><strong>With inheritance we are able to create objects with specialized behavior by inheriting the behavior of a more generalized object (the parent) and specializing it.</strong> Let’s build a client for an API that consumes both JSON and XML encoded data. It could look like this.</div><pre>#!/usr/bin/ruby
# Wrapper around a HTTP library
class ApiClient; ... end
# Knows how to decode JSON responses from the API
class JSONApiClient < ApiClient; ... end
# Knows how to decode XML responses from the API
class XMLApiClient < ApiClient; ... end
# Exposes an API's endpoints as methods on an object
class MyAppApiClient
attr_reader :json_client
attr_reader :xml_client
def initialize(api_token)
@json_client = JSONApiClient.new(api_token)
@xml_client = XMLApiClient.new(api_token)
end
def current_account
json_client.get('/api/current_account')
end
def balance
xml_client.get('/api/current_account/balance')
end
end
class MyAppApiClient
attr_reader :json_client
attr_reader :xml_client
def initialize(api_token)
@json_client = ApiClient.new(API_KEY, JSON.method(:parse).to_proc)
@xml_client = ApiClient.new(API_KEY, XML.method(:parse).to_proc)
end
def current_account
json_client.get('/api/current_account')
end
def balance
xml_client.get('/api/current_account/balance')
end
end
array = [1,2,3,4,5]
array.map # => #
array.map { |i| i * 2 } # => [2, 4, 6, 8, 10]</pre><div>This certainly would get the job done. But <strong>this code has two issues</strong> that we need to address.<br><br>First, both <strong>JSONApiClient</strong> and <strong>XMLApiClient</strong> are <strong>tightly coupled to their parent and provide little functionality besides the functionality they inherited</strong> — meaning that any change to the parent class would probably break the classes inheriting from it.<br><br>Second, <strong>JSONApiClient</strong> and <strong>XMLApiClient</strong> are <strong>too specialized</strong>. Inheritance is all about specialization, but having too specialized classes can cause headaches — they are hard to extend (when requirements change) and introduce unnecessary complexities (for a human, it’s hard to remember many different kinds of objects and what they do).<br><br><strong>We can address both issues by passing the desired parser as an argument to ApiClient</strong>. This is in-line with the <a href="https://en.wikipedia.org/wiki/Open%E2%80%93closed_principle">open-close</a> principle, as now we can create an <strong>ApiClient</strong> that can parse any kind of data without needing to subclass it.<br><br><strong>We took an existing class/service/object/function and combined it with another to get more specialized behavior.</strong> This is the basic idea behind <a href="https://en.wikipedia.org/wiki/Function_composition_(computer_science)">function composition</a>.<br><br>While the above example is actually an example of <a href="https://en.wikipedia.org/wiki/Dependency_injection">dependency injection</a>, it is a great outline for the things that are possible with function composition in Ruby. Also, note that the above approach is limited to the configurable dependencies we accept through the initializer.<br><br><strong>We use composition every day without even noticing it.</strong> <strong>Enumerable#map</strong> uses composition to enable us to transform arrays.<br><br>The <strong>map</strong> method is of limited usefulness on it’s own. But when we combine it with a block (another function) it can transform whole datasets.</div><h2>Proc#>> and Proc#<<</h2><div>
<strong>Proc#>></strong> and <strong>Proc#<<</strong> were introduced in Ruby 2.6. <strong>They provide a substantial quality-of-life improvement when it comes to composing functions.</strong><br><br><strong>Proc#>></strong> is similar to Elixir’s pipeline, but instead of returning a result it returns a proc — a callable object.<br><br>In mathematical terms, <strong>f(x) >> g(x)</strong> is the same as <strong>g(f(x))</strong>.<br><br>Let’s go back to our <strong>ApiClient</strong> example:</div><pre>#!/usr/bin/ruby
f = -> x { x + 2 }
g = -> x { x * 2 }
# h is the composition of f and g
h = f >> g
# h is the same as -> x { (x + 2) * 2 }
[1, 2, 3].map(&h) # => [6, 8, 10]
# This is exactly the same as
[1, 2, 3].map(&f).map(&g) # => [6, 8, 10]
fetch_transactions =
ApiClient.new(api_token).method(:get)
>> JSON.method(:parse)
>> (-> response { response['data']['transactions'] })
fetch_transaction.call('/api/current_user/transactions')
f = -> x { x + 2 }
g = -> x { x * 2 }
# h is the composition of g and f
h = f << g
# h is the same as -> x { (x * 2) + 2 }
[1, 2, 3].map(&h) # => [4, 6, 8]
# This is exactly the same as
[1, 2, 3].map(&g).map(&f) # => [4, 6, 8]</pre><div>Notice that we didn’t pass anything to <strong>ApiClient</strong>, we didn’t need to do dependency injection. <strong>This solves the configuration problem we had before. And we are able to create highly specialized functions on-the-fly.</strong> The above example creates a function that returns all transaction from an API endpoint.<br><br>Note, if you want Elixir style pipelines that return a result check out <strong>yield_self</strong> or the new alias for it <strong>then</strong>.<br><br>On the other hand, <strong>Proc#<< is more in-line with Haskell style composition:</strong><br><br>Or, in mathematical terms, <strong>f(x) << g(x)</strong> is the same as <strong>f(g(x))</strong>.<br><br>Both <strong><<</strong> and <strong>>></strong> are basically the same, <strong>which one to use is only subject to your preference.</strong><br><br><strong>Function composition is very useful in languages that don’t have the concept of classes and inheritance</strong> as it enables you to “inherit” the behavior of a function and extend/specialize its behavior. Like in the following example.<br><br><strong>Since we do have inheritance in Ruby this kind of composition is less useful.</strong> Yet it gives us the ability to create utility functions on-the-fly thus <strong>it encouraging us to create methods/classes that can be extended through the open-close principle / dependency injection, and </strong><a href="https://en.wikipedia.org/wiki/Composition_over_inheritance"><strong>composition-over-inheritance</strong></a><strong>.</strong> In my opinion, this is a big step forward for the language.</div><h2>Conclusion</h2><div>As its implemented now, composition sticks out like a sore thumb. <strong>The ecosystem needs to grow to accommodate for this feature.</strong> Constantly calling <strong>#method</strong> on a class/module is confusing and distracting — a short hand operator for this purpose would be welcome (I would propose <strong>&[Json].parse</strong>). The <strong>map</strong>, <strong>reduce</strong> and other enumerator methods are hard to compose since they are implemented on an instance of an object — an <strong>Enum</strong> module exposing them would be nice.</div><pre>#!/usr/bin/ruby
require 'net/http'
require 'uri'
require 'json'
fetch =
self.method(:URI) \
>> Net::HTTP.method(:get) \
>> JSON.method(:parse) \
>> -> response { response.dig('bpi', 'EUR', 'rate') || '0' } \
>> -> value { value.gsub(',', '') } \
>> self.method(:Float)
fetch.call('https://api.coindesk.com/v1/bpi/currentprice.json') # => 3530.6782</pre><div>If somebody shares these pain points, and agrees with me. <a href="https://gist.github.com/monorkin/7e1a597afb0ad295d5b8ecfb951d60af">I’ve created a draft implementation of those features</a>.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/102021-08-09T00:00:00Z2023-11-14T16:28:03ZNew coat of paint<div class="trix-content">
<div>
<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzIyMT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--a6da6035ccdb5e438b9a7501ea53f8063cecbe52" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZDA9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--ca547c8f90d535fb25f94d5f7b8cb8dae8484ab9/paint.png" filename="paint.png" filesize="769850" width="1203" height="502" previewable="true" caption="Abstract color pour by Pawel Czerwinski">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjIxLCJwdXIiOiJibG9iX2lkIn19--bfb9ff970a1962e391b3b74629676e139c444ac6/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/paint.png">
<figcaption class="attachment__caption">
Abstract color pour by Pawel Czerwinski
</figcaption>
</figure></action-text-attachment>Recently I got the urge to write again so I gave this blog a new coat of paint, and decided to give a brief history, the reason I stopped writing and an explanation as to why I moved away from <a href="https://medium.com/">Medium</a>.<br><br>I started this blog back in 2016 as a way to share and discuss ideas and solutions to problem's I've tackled as part of my work, life, education and hobbies.<br><br>At the time Medium was the hot new, elegant, hosted blog engine with a lot of buzz around it. I jumped on the bandwagon and started my blog there as I didn't have much money at the time, didn't want to build my own blog and didn't want to self-host or pay for WordPress.<br><br>The first article here <a href="/hacking-privacy-into-facebook-s-messenger-in-24-hours-3da239baf6c5">is a story about a hackathon I went to in 2016</a> after which I basically stopped writing. Then, two years later, I switched jobs and technology stacks which lead me to reevaluate most of the things I knew about web development so I started writing again to crystallize my thoughts.<br><br>Then <a href="do-you-really-need-websockets-343aed40aa9b">"Do you really need WebSockets"</a>, an article I wrote because I was tired of explaining to people that SSE and long-polling are not the same thing, became popular. The feedback was overwhelmingly positive and I got hooked to the high of having a popular article.<br><br>Chasing that high made me imitate my previous success. All articles after "Do you really need WebSockets" were in-depth comparisons. But writing those comparisons wasn't fun, it felt like a chore. Even though I was satisfying my own curiosity I didn't enjoy the process of writing down the conclusion. Why? Because I imposed the structure and tone of "Do you really need WebSockets" on those articles to make them more likely to succeed (at least in my mind).<br><br>The problem with "Do you really need WebSockets" is that it's written like an academic paper - it's monotone, uses more complicated language just to sound smart and has a predictable structure to every paragraph. When I read those articles now they sound boring and devoid of personality.<br><br>The first article I ever wrote in this blog represents me better than the last five.<br><br>Once I realized that I started working on this iteration of the blog.<br><br>The first step for me was to get away from Medium as the site had many features I didn't want my blog to have.<br><br>The feature I wanted to get rid off the most was "claps" (Medium's version of likes). Comparing something by how many likes it has produces feedback loops like the one I got into after "Do you really need WebSockets". And the number of likes doesn't tell me anything of value - I don't know if the person giving the like thinks this article is better than the others I wrote, I just know that they liked this one. Even if I did know they liked this article more then the rest I still wouldn't know why.<br><br>The second feature I didn't want to have were comments. Over the years there were many nice comments, good questions and corrections. But, as with any unmoderated comments section on the Internet, there were ten times more comments from people who clearly didn't read the article and made ungrounded claims calling me out as a liar and dimwit.<br><br>At first I tried to be kind, explain their claims as false and correct them. But that took more time than I'd like to invest into responding to comments so I tried to ignore those people - this didn't work. Those negative comments influence new readers which then ask me to answer those commenters. In the end I had to answer all comments, which made me bitter, which made my comments sound bitter, which portrayed me as a bitter person.<br><br>Therefore, no more comments. If someone wants to comment something they should start a thread somewhere on the Internet and link to an article on this blog. All links to any article will show up in the backlinks section and anyone jump into any of those threads.<br><br>So I developed this web app to re-vitalize the old blog, add new features to it and fix the issues I had with Medium:<br><br>A custom design that strains the eyes less and that has no accessibility issues.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzIyMj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--c36ba7e3b5aaef23310b7c4526ad6f0b9d1c2cac" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZDQ9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--e28c62b2df557cf3e1299d0b6fd7047ac8f9f4ce/new_design.gif" filename="new_design.gif" filesize="3261996" width="1252" height="787" previewable="true" caption="The new design of stanko.io">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjIyLCJwdXIiOiJibG9iX2lkIn19--3e32a96a665ba6162f057ea1a5b77a60ba773c71/new_design.gif">
<figcaption class="attachment__caption">
The new design of stanko.io
</figcaption>
</figure></action-text-attachment>Link previews - because having to open links to see if you want to read what they link to is annoying<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzIyMz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--96dd9cd9ed03354af6acae3e408ee14f8083831d" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZDg9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--2b0458cb976dbafb408f2b0d6b316f996580e2ef/link_previews.gif" filename="link_previews.gif" filesize="960201" width="1252" height="787" previewable="true" caption="Link previews">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjIzLCJwdXIiOiJibG9iX2lkIn19--dfc22c4fbae7b54c7d8716c7c66a8b1ec145f07d/link_previews.gif">
<figcaption class="attachment__caption">
Link previews
</figcaption>
</figure></action-text-attachment>My own analytics - because that way I can respect my readers' privacy and track backlinks to articles<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzIyND9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--6f70f0a52da377378392b8827763d30c41ea4dd6" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZUE9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--3346245a03e249db2dbf646a721ebc018e7d9871/backlinks_and_analytics.gif" filename="backlinks_and_analytics.gif" filesize="2843572" width="2442" height="750" previewable="true" caption="Backlinks & analytics on stanko.io">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjI0LCJwdXIiOiJibG9iX2lkIn19--da15fa5785e1dc81c229d703c2cc9e5ca18ad4c5/backlinks_and_analytics.gif">
<figcaption class="attachment__caption">
Backlinks & analytics on stanko.io
</figcaption>
</figure></action-text-attachment>Visible RSS and ATOM feed for subscriptions<br><br>It's by no means perfect, but it was fun to do and it shows off who I am, the way I think, my personality and my values. And now that I can write without self-imposed restriction I'm having fun again.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/112021-08-28T00:00:00Z2023-11-14T16:28:03ZThe Judo way<div class="trix-content">
<div>
<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzIzNj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--e9a1e1c8fcf743a36437b8f1f99f7f1995e28030" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZXc9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--9652982111f8b0ec6476e1f5c268dfd8ad57ccc2/lake_walkway.png" filename="lake_walkway.png" filesize="2346232" width="1279" height="751" previewable="true" caption="A walkway at Plitvice Lakes, Croatia">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjM2LCJwdXIiOiJibG9iX2lkIn19--4d458290d2064f43b49075cac2651457c9457f4b/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/lake_walkway.png">
<figcaption class="attachment__caption">
A walkway at Plitvice Lakes, Croatia
</figcaption>
</figure></action-text-attachment>As a kid I practiced Judo which is a martial art as well as a philosophy. And my Judo teacher gave us a philosophical story which I came to understand only recently and which helped me achieve more than I though was possible. So I wanted to share it.<br><br>The story was about a small tree in a strong wind. The strong wind could uproot big trees but couldn't do the same to small trees. How so?<br><br>The teacher explained that the small trees are flexible so they can bend and move with the wind, therefore the wind can never exercise it's full force on them. But the big trees aren't as flexible and their only way to take the force of the wind is to develop deeper roots, stronger trunks and branches. So you don't have to be a big strong tree to take on an opponent such as the strong wind if you can move with the wind.<br><br>This analogy completely flew over my 11 year old head at the time, but I remembered the story and came to understand it a few years ago.<br><br>The point of the story, and the philosophy of Judo, is to achieve maximum efficiency with minimum effort (similar to the <a href="https://en.wikipedia.org/wiki/Pareto_principle">Pareto principle or the 80/20 rule</a>). The best way to do so is not to fight the incoming force but to use or steerer it in a direction that puts you in an advantageous position. The key to understanding the power of this philosophy is to understand that you can replace the word "force" with anything else, and to accept that an advantageous position might not be exactly what you want but a step in that direction.<br><br>In the case of the story, if your goal is to stay in the ground then it's much easier to be flexible and move with the wind. You could also develop deeper roots and a stronger trunk but that requires more effort than moving with the wind and eventually a stronger wind could come along and uproot you.<br><br>I've used this philosophy to attain personal, and professional, goals which range from losing weight to chaining our process at work - things I previously thought were too laborious to be practical.<br><br>For example, at work we use the Scrum methodology and it's not a good fit for us. The development team is increasingly unhappy about the work we do, and the time we get to work on things that aren't part of a sprint. So we read about other methodologies and composed a list of things we would like to borrow from them to improve our process.<br><br>We can't just change the process on our own, we have product managers and Scrum masters on which these changes depend on too. So how do we go about implementing those changes?<br><br>We could form a document and a slideshow, collect all the evidence and studies to support our proposed process changes, and present everything to management. They are rational people, they will understand and accept our proposal, right?<br><br>While this sounds reasonable I can tell you right away that it will most likely fail. And the worst part is that not only will your process be rejected but all the improvements that could be applied to the current process that are bundled with it will be thrown out as well.<br><br>The problem is that management often doesn't know, and can't understand, the price of this process - the tech dept piling on, the extra hours, the emergencies on a Friday night which could have been avoided, the endless planning meetings to hack around self imposed system limitations. They either don't see those problems, don't feel them directly, or think they are a badge of honor for you. Your team is shipping value, customers are happy, things get done, so why change something that works and introduce uncertainty the process?<br><br>The Judo way to do it would be to point out one issue the team felt during the last sprint at each retrospective meeting and propose the process improvement you want to solve it. This is using Scrum against itself - the retrospective meeting is here to solve process problems so the issues your raised have to be discussed and acted upon. If your solution is accepted, or something similar to it, you will be one step closer to your goal.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/122022-07-01T12:00:00Z2023-11-14T16:28:04ZHow I stumbled upon Strada while forwarding an email<div class="trix-content">
<div>I wanted to forward an email one evening, so I opened up the Hey app on my phone, found the email, tapped on the “More” button, and just before I hit “Forward” I noticed a “Share or print…” button at the bottom of the screen. I hit “Share” and to my surprise was greeted by a share sheet. It offered me to send a link to the email via SMS, Slack, Signal, and others.<br><br>If it was any other app this wouldn’t surprise me, but Hey is a hybrid app — its views are server-rendered HTML and some JavaScript wrapped in an app that is essentially a web browser. Showing a share sheet isn’t something you can do with HTML and JS, so how did the team at Hey do this?<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzIzND9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--a3195539072eb92ebd807203254bc62fb37ecb12" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZW89IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--45aa0edae6219c93988dfdcdef399377df617ddd/strada_demo.gif" filename="strada_demo.gif" filesize="3277647" width="450" height="800" previewable="true" caption="Opening a share sheet in Hey">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjM0LCJwdXIiOiJibG9iX2lkIn19--19f6b4915cb0c6f60ef109bb06dbf2c8651f8b51/strada_demo.gif">
<figcaption class="attachment__caption">
Opening a share sheet in Hey
</figcaption>
</figure></action-text-attachment>I assumed that the Hey mobile app used the same HTML and JS as the web version. So I opened the Hey web app on my computer and started poking around.<br><br>The first thing I noticed was that all the buttons that were visible in the mobile app were present in the web version. But they were hidden unless the frontend was running in a browser that used a special User-Agent header. So I changed my User-Agent and reloaded the page. The UI changed — the header row was gone, some buttons looked like they were disabled and new buttons showed up.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzIzNT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--4a180dcb8bbaa47312856609442421d0108a9b5e" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZXM9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--a6c02144c2085d9affda8abb4b6ebc146de67632/2022-06-13_15-11_3.png" filename="2022-06-13_15-11_3.png" filesize="244096" width="852" height="1010" previewable="true" caption="Left: Hey web app in Firefox with the default User-Agent. Right: Impersonating Hey iOS app by changing the User-Agent">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjM1LCJwdXIiOiJibG9iX2lkIn19--70aa93da264e080b3e709a1833e193027159ac67/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/2022-06-13_15-11_3.png">
<figcaption class="attachment__caption">
Left: Hey web app in Firefox with the default User-Agent. Right: Impersonating Hey iOS app by changing the User-Agent
</figcaption>
</figure></action-text-attachment>I thought the layout changes were related to Turbo Native and the disabled buttons seemed like a CSS issue, so l focused on the “Share and print…” button.<br><br>Inspecting it showed that it was controlled by a <strong>bridge/share</strong> Stimulus controller. So I opened that controller and saw that it only had a <strong>component</strong> static variable and a <strong>#share</strong> method. The static variable was hard-coded to the value “share”. The <strong>#share</strong> method just passed the current URL to a <strong>#send</strong> method it inherited from <strong>bridge/base_component_controller</strong>.</div><pre>#!/usr/bin/node
// bridge/share.js
import BaseComponentController from "bridge/base_component_controller"
class ShareController extends BaseComponentController {
static component = "share"
share(event) {
event.preventDefalt()
this.send("share", { url: window.location })
}
}</pre><div>I followed the code and opened the <strong>bridge/base_component_controller</strong>. Its <strong>#send</strong> method took an event name, a data object, and a callback function. It built a <strong>message</strong> object with the event name, data, callback and the controller’s component. Then it forwarded everything to the <strong>window.webBridge.send</strong> method and stored the returned value as the ID of the message.</div><pre>#!/usr/bin/node
// bridge/base_component_controller.js
// Pseudocode for BaseComponentController's send method
send(event, data, callback) {
let message = { component: this.constructor.component, event, data, callback }
let messageId = window.webBridge.send(message)
}</pre><div>But where did <strong>window.webBridge</strong> come from? I searched all the files and found it was initialized right after a class named “Strata” was loaded. From the name of the class I knew I was looking at Strada — the, yet unreleased, framework from Hotwired.<br><br>This got me excited! I started reading the code of the class to figure out what it does.</div><pre>#!/usr/bin/node
// strata.js
// Pseudocode for Strata's send method
send(originalMessage) {
if (!this.supportsComponent(message.component)) return null
let id = this.generateID()
let message = {
id: id,
component: originalMessage.component,
event: originalMessage.event,
data: originalMessage.data || {}
}
this.adapter.receive(message)
if (originalMessage.callback) this.callbacks[id] = originalMessage.callback
return id
}
receive(message) {
const callback = this.callbacks[message.id]
if (callback) callback(message.data)
}
supportsComponent(component) {
return this.adapter.supportsComponent(component)
}</pre><div>On load, it initialized itself and was assigned to <strong>window.webBridge</strong>. Then it fired a <strong>web-bridge:ready</strong> on the document.<br><br>It had an <strong>adapter</strong> variable which was called in most of its methods.<br><br>Its <strong>#send</strong> method generated an ID for each message, then it appended the ID to the message, stored the callback and forwarded the rest to the adapter.<br><br>There was also a <strong>#receive</strong> method which seemed to take a message object and execute any callback method associated with its ID. That explained why the callbacks were stored when a message was sent.<br><br>Then there was the <strong>spportsComponent</strong> method that queried the adapter for components it could support. Probably to avoid sending messages that the adapter couldn't process.<br><br><strong>Now I started looking for the adapter implementation but I couldn’t find it. This was where the code ended.</strong><br><br>And then it hit me! This would usually run within a browser view controlled by the native app. The app can inject JavaScript into it. That’s why I couldn’t find the adapter implementation — it was in the native app.<br><br>But why would the app inject the adapter into the frontend? This didn’t make sense at first. Then I remembered that in browsers controlled by a native app there is a special <strong>window.external</strong> object (on iOS it’s <strong>window.webkit</strong>). The app can expose functions on that object that when called from JS invoke a handler function in the app.<br><br>So, the app probably registered an adapter by injecting JS into the browser. That would explain why <strong>web-bridge:ready</strong> fires when Strada loads — so that the app knows when it can register the adapter. And through that adapter, the frontend can send messages to the app.</div><pre>#!/usr/bin/swift
// AppDelegate.swift
// Example of how you could register an adapter with Strada from iOS
let supportedComponents = ["share"]
let supportedComponentsJSON = String(
data: JSONSerialization.data(
withJSONObject: supportedComponents,
options: []
),
encoding: String.Encoding.utf8
)
let registerAdapterJS = """
window.registerStradaAdapter = () => {
const supportedComponents = \(supportedComponentsJSON)
window.webBridge.setAdapter({
platform: "ios",
supportsComponent: (name) => return supportedComponents.include(name),
supportedComponents: supportedComponents,
receive: (message) => window.webkit.messageHandlers.strada.receive(message))
})
}
if (window.webBridge) {
window.registerStradaAdapter()
}
else {
document.addEventListener("web-bridge:ready", window.registerStradaAdapter)
}
"""
// Inject the script right after the document is loaded
webView.configuration.userContentController.addUserScript(
WKUserScript(
source: registerAdapterJS,
injectionTime: WKUserScriptInjectionTime.AtDocumentEnd,
forMainFrameOnly: true
)
)</pre><div>And the app could send messages back to the frontend by injecting them into the browser like JavaScript.</div><pre>#!/usr/bin/swift
// AppDelegate.swift
// Example of how an iOS app could add a
// `window.webkit.messageHandlers.strada.receive`
// function and process calls to it.
// Somewhere in AppDelegate after you have created your
// browser view (webView) and have initialized Turbo Native
let strada = Strada()
let contentController = webView.configuration.userContentController
contentController.addScriptMessageHandler(strada, name: "strada")
// Strada.swift
// Example message handler that responds to calls to
// `window.webkit.messageHandlers.strada.receive`
class Strada: WKScriptMessageHandler {
func userContentController(userContentController: WKUserContentController!,
didReceiveScriptMessage message: WKScriptMessage!) {
// The `name` attribute holds the name of the function that was invoked
if (message.name != "receive") {
return
}
if (message.body["component"] == "share" &&
message.body["event"] == "share") {
// show share sheet
}
}
}
// Strada.swift
// Example of how you could call the adapter from iOS
let message: [String: Any] = [
"id": "7",
"data": ["foo": "bar"]
]
let messageJSON = String(
data: JSONSerialization.data(
withJSONObject: message,
options: []
),
encoding: String.Encoding.utf8
)
webView.evaluateJavaScript(
"window.webBridge.receive(\(messageJSON))",
completionHandler: nil
)</pre><div>
<strong>I realized that Strada was a bridge between the native app and the frontend.</strong><br><br>And through that bridge one can patch in functions that the platform supports but the browser doesn’t — things like showing a share sheet.<br><br>The mystery of the share sheet was solved. But now my head was buzzing with possibilities.<br><br>Most native apps are functionally the same as, or a subset of, their web counterpart. Without Strada, a native app would have to re-implement some of the views and logic of its web counterpart. This means that a team would have to build the same app at least twice — once for the web and another time for iOS, Android, Mac, or Windows — just so they could use some platform-specific features. This is even more wasteful if we take into account that most web apps are already optimized to work on phones, tablets, and PCs.<br><br>But with Strada, any existing web app can become a native app.<br><br>Many thanks to Karla, Marko, Hrvoje & Vlado for reviewing drafts of this article.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/132022-07-11T12:00:00Z2023-11-14T16:28:04ZMisguided Mark misrepresents Micro-services: A story about paradigms<div class="trix-content">
<div>Tom was brewing tea in preparation for a 5 o'clock meeting he knew little about. The agenda was so vague it might as well have been left out - it read “Mark introduces a new system for controlling 3rd party IoT water sprinklers”. That was standard practice for CorpCorp - the company Tom worked for - everything was a meeting and things were only ever presented through meetings with at least 10 attendees, even when an email would be more than enough.<br><br>But Tom wanted to attend this meeting. He was the head of the Web development department and 3rd-party integrations were his responsibility. His team was short-staffed to the point that they couldn’t start working on this integration until the next quarter.<br><br>When Mark, a developer from the in-house IoT sprinkler department, offered to develop this integration Tom immediately accepted - Mark used to work as a Web developer before he joined CorpCorp and Tom thought that maybe he wanted to switch departments.<br><br>5 o’clock rolls around, Tom joins the meeting from his home office and takes a big sip of tea.<br><br>On his screen is Mark, standing in front of a nice looking slide deck that in bold letters says “Controlling 3rd party water sprinklers: The future of CorpCorp”.<br><br>Mark greeted everybody and started to present. He explained that he had spent two weeks planning out this integration and came to the conclusion that it will be built using micro-services - 13 of them - even though most of CorpCorp’s Web services were in a monolithic Web app.<br><br>Mark explained that monoliths were a thing of the past - they didn’t scale, and always ended up too complex to maintain. Micro-services, on the other hand, scaled infinitely and were super easy to maintain because having everything separate improved encapsulation and separation of concerns.<br><br>Mark stopped, looked at all the confused faces, and asked if there were any questions. Immediately a few managers that didn’t listen to half of the presentation asked simple questions that Mark already explained during his presentation - this was also standard practice at CorpCorp.<br><br>Tom had a working theory that the managers get bored by the technical jibber-jabber and doze off, but when the meeting ends they want to prove that they actually listened and that they were a useful addition to the meeting, so they ask a question about the last thing that they remember. Further experiments and observations were required to prove this theory, but even knowing if that was the case wouldn’t solve the problem - that the meeting should have been an email.<br><br>Once the flurry of questions ended Tom started to talk. He thanked Mark for taking the time to research this integration, and confirmed that it would be a good solution to the problem, but that he shouldn’t move ahead with its implementation because the system would cause too many problems down the line. He then added that Mark couldn’t see how misguided he was about monoliths because he clinged too hard to his paradigm to think level-headed.<br><br>The managers woke up out of a sudden - they didn’t want to miss the drama.<br><br>Tom started to address each point of Mark’s presentation. He began by asking Mark ih he knew why CorpCorp developed a Monolith in the first place. Mark just shrugged and guessed that it was because they were at the peak of their popularity when the project was started. Tom refuted that and said that a monolith was conscious decision by the department because:<br><br><strong>MONOLITHS ARE EASIER TO MAINTAIN FOR SMALLER TEAMS</strong><br><br>Tom’s department counted 12 people, but those 12 people were enough to handle most of CorpCorp’s business. Only in recent months, after one employee retired and another quit, did Tom’s department start to struggle with meeting business demands.<br><br>Tom’s employees built a monolith and decided upon a convention for going about their most common tasks. This allowed them to be flexible - anyone could work on any part of the system because they knew how everything was supposed to be organized. And over time, as features were added or removed, and bugs fixed, everybody became familiar with most of the system. So anybody was able to add a feature or fix a bug anywhere in the system.<br><br>You could achieve the same by implementing a convention on top of micro-services, but there is nothing stopping you from implementing multiple conventions. The more people that work on your micro-services, the more opinions there are on how they should be implemented, the likelier the chance that convention will be broken or changed.<br><br>In fact that is a big selling point of micro-services - you can use any technology and any convention you want for each individual service. And that is a desirable trait for mammoth companies that suck up developers just so the competition won't get them - they can keep them happy by letting them do what they want in their own micro-cosmos. This sometimes works out and sometimes it doesn’t, but for these kinds of companies that is just part of doing business.<br><br>In general, it is better for individuals and small teams, to be intentional with what they do and how they go about it. Especially in software development - where the question isn’t “is it doable?”, because the answer is always “yes”, but “how much time will it take?”.<br><br>And a monolith is great at ensuring that - one vision, one codebase and everybody is aligned.<br><br><strong>MONOLITHS ARE EASIER TO SCALE, BUT THEY DON’T SCALE AS WELL AS MICRO-SERVICES</strong><br><br>Tom decided to use an analogy here so that the managers wouldn’t doze off again. A monolith is like a hamburger, it is a meal consisting of a bun, a patty and some salad. While micro-services are a patty, some bread and a salad as three separate meals.<br><br>With a monolith, if you want to double the throughput you double the number of running instances. This is the equivalent of ordering two hamburgers when you are very hungry. The good thing is that it’s simple - you just double everything and you are done, the bad part is that it can be wasteful.<br><br>In micro-services, if you want to double the throughput you first have to identify what the bottleneck is, and then you double that service and some of its upstream services. This is the equivalent of ordering two patties, a regular salad and a bun when you are very hungry - you only pay for what you really want.<br><br>But figuring out what the bottleneck is can be hard, both in monoliths and micro-services.<br><br><strong>COMPLEXITY COMES FROM BUSINESS NOT CODE</strong><br><br>The notion that monoliths are complex to maintain and micro-services aren’t is a myth. In a monolith and in micro-services complexity comes from the business domain - what the app does. If the business domain is complex or not thought out - if there are no clear rules, if the process is dependent on many factors, if the process has many interdependent steps, if the process is convoluted to humans - the code will reflect that.<br><br>The code you write mirrors how you think about a problem. If you don’t think clearly, or if the problem is inherently muddy, then you are out of luck - architecture has little to do with that.<br><br><strong>MICRO-SERVICES IMPROVING ENCAPSULATION OR SEPARATION OF CONCERNS IS A MYTH</strong><br><br>Tom had to resort to another analogy here to keep the managers listening. Encapsulation is the act of bundling together things that belong together. It is in a way similar to doing laundry - imagine you have two laundry baskets: one for colored clothes and another for white & black clothes.<br><br>Separating the clothes in their baskets is encapsulation - each basket encapsulates only its set of clothes. But where would you put dark blue underwear?<br><br>On the one hand it is blue - a color - so it should go into the color basket. But on the other hand it is dark blue, which is close to black - so it should go into the white & black basket. Depending on who you ask you will get a different answer.<br><br>But people who argue that micro-services improve encapsulation & separation of concerns argue that, somehow, the problem will solve itself if you put the laundry baskets really far apart.<br><br>If you are bad at separating concerns or encapsulating code, you will be equally as bad at it in a monolith and in micro-services.<br><br>Tom stopped for a moment to look at everyone’s faces, the managers were more alert than they have ever been while Mark was embarrassed. Tom felt bad, and decided to give a moral to this story to lighten the mood.<br><br>He said that this was a common mistake that young developers make - they get too attached to their paradigm and become unable to understand why someone would do things differently. They think that the existence of a new paradigm - a new school of thought - must invalidate the old one. Why else would then the new school exist? Why would micro-services exist if monoliths were good?<br><br>People form an understanding of the world through paradigms. There is no one perfect paradigm, no one ultimate school of thought, just like there is no one perfect theory of how the world works..<br><br>Look at Newton’s and Einstein's model of gravity. Einstein’s model is superior to Newton’s - it is better at explaining how objects will move when they are going slow and when they are going close to the speed of light.<br><br>This precision comes at a cost - it is much harder to compute Einstein’s gravity equations than Newton’s. But is this computational cost worth it? When the object is moving slowly, Newton’s equations are a fraction of a percent off compared to Einstein’s equations.<br><br>Many civil engineers say the trade off isn’t worth it. For their day to day work they use Newton’s equations because they can iterate bridge, road, and building designs quickly and find the best solution. But NASA engineers would beg to differ - a precise result is much more important to them.<br><br>Each paradigm is good at something and bad at something else. It is up to you to understand the trade off and pick what works best for you.<br><br>P.S. The style of writing in this essay was inspired by a book I recently read - <a href="https://geraldmweinberg.com/Site/AYLO.html">Are your light on?</a> by Donald C. Gause and Gerald M. Weinberg. Its a fantastic book about problem solving and thinking about problems.<br><br>Many thanks to Marko I. and <a href="https://shime.sh/">Hrvoje S.</a> for reading drafts of this article and providing feedback</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/142022-07-15T12:00:00Z2023-11-14T16:28:05ZScrum is for time estimates, not projects<div class="trix-content">
<div>From my experience Scrum is the prevalent project management framework in software development. Most of the teams I was part of used it to develop, deliver and maintain their projects. Despite its prevalence I always felt like Scrum was bogs me down - that it was more harmful than useful.<br><br>I knew what was bothering me the most - the daily standup. But even in teams that had a standup once a week I felt like Scrum didn’t make the project better. Yet we were burning two work-hours per person every week on its ritual.<br><br>One day while reading about Control Theory and PID controllers, I realized that Scrum was a time estimation tool.<br><br>To understand this, you first have to know what a PID controller is and how it works. In a nutshell, it’s a device that observes a dimension of a process - like the speed of a wheel - and tries to keep it at a desired value.<br><br>If the wheel goes too slow the controller speeds it up, if it goes too fast the controller slows it down. The magic of PID controllers is that they, once tuned, don’t over or undershoot the desired value. Their magic consists of 3 steps - P, I & D.<br><br>P, the proportional, corrects the difference between the desired and actual value. I, the integral, looks at past over or undershoots and corrects for them. D, the derivative, nudges the result closer to the ideal value.<br><br>That is exactly what Scrum does. Standups correct problems as they occur. Planning corrects over or undershoots in the committed amount of work. Retros nudge the process towards the ideal outcome.<br><br>If Scrum is a PID controller what dimension does it control?<br><br>The only one that has a fixed value - the duration of the sprint. All other values are in flux - the number of story points, the number of tasks in a sprint, the number of people in a team.<br><br>By controlling the duration of a sprint you can estimate the duration of a project. If you chop up a project into 30 tasks, and the team solves about 8 tasks every sprint, you can be somewhat certain that the project will take 4 to 6 sprints. If a sprint is 2 weeks long, then the project will take 12 weeks.<br><br>This chopping up of a project into tasks is why Scrum bogs me down - it kills the vision. Without vision you can’t know what you are building and therefore you can’t utilize your expertise to make it better.<br><br>Building a project task by task can produce something that doesn’t work as a whole. Each task makes sense on its own but all of them together don’t form a whole - they don’t accomplish the goal of the project.<br><br>Because tasks don’t carry a sense of how they fit together, any sense of wholeness is accidental. One feature can be split into multiple tasks which can produce a different result depending on which one is implemented first. New tasks can be added with no regard as to how they incorporate into the vision, which can produce nonsensical features. When a feature is broken into multiple tasks, changing one task without updating the others leads to a disjointed feature.<br><br>The worst possible case is when someone creates tasks that build on top of each other. Working on such tasks is like being led by a carrot on a stick through what someone else thinks is the best way to build the project. Instead of letting you, the expert at building software, decide what the best way to build it is. This is frustrating and demotivating, and you can’t prevent it because you can’t see the bigger picture.<br><br>Scrum isn’t a framework for people building the project, it’s a framework for people concerned with timelines and roadmaps.<br><br>Builders are concerned with how things are supposed to work, how they connect together and interact with one another. They are concerned with what they are building, with the vision.<br><br>Scrum has no mechanism to help you build your project. None of its rituals can help you shape a project. They can’t help you form a vision for what you are about to build, or help you figure out what is still undefined. Scrum assumes that this is already figured out when you start grooming and planning the project. The best you can ever do is say “I have no clue what we are building” and throw a sprint out.<br><br>Yet the burden of making Scrum work falls on the ones it doesn’t help - the builders - the ones concerned with building something great. It only helps them adhere to a timeline. But a bad project delivered on time is still a bad project.<br><br><em>P.S. Many thanks to </em><a href="https://shime.sh/"><em>Hrvoje S.</em></a><em> for reviewing drafts of this article.<br><br>If you are looking for a framework that does help you shape your project and build something better, but has a looser concept of timelines, take a look at </em><a href="https://basecamp.com/shapeup/webbook"><em>Shape Up</em></a><em>. I recommend that book even if you aren't looking for a new framework, it describes excellent tools to help you identify if your project is well defined - like breadboards and fat marker sketches.</em>
</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/152022-08-21T12:00:00Z2023-11-14T16:28:04ZKeep it boring, don’t surprise me<div class="trix-content">
<blockquote>I’ve spent a lot of my life worried that people will think I don’t know enough. Sometimes, that worry has made me use big words when I didn’t need to. <br><br>–Randall Munroe in the foreword to the Thing Explainer</blockquote><div>I used to be a stickler for organizing code by what it was. Models, decorators, form objects, view objects, value objects, service objects - all lived in their own separate directory. I felt that the code was well organized that way, that everybody knew what went where, and what the purpose of everything was just by how the code was organized.<br><br>My obsession with grouping things that way made me blind to what I was actually doing - writing surprising code.<br><br>When I think of the reasons why I did things that way only three things come to mind.<br><br>First, I wanted to show off how much I knew about patterns, algorithms and programming. I wanted to be the best programmer, and prove that to my peers by organizing code in a manner that showed off my knowledge.<br><br>Second, I didn’t know any better. The code was already organized into models, views and controllers so it was natural to me to continue this trend of having every kind of object separate.<br><br>Third, I didn’t want to have “fat” models or controllers because that has bitten me in the past.<br><br>Today I consider most things to be a model, with a few exceptions. Why? Because decorators, form objects, view objects, value objects, service objects, and others are just different ways to model data. Models aren’t just objects persisted in a database, a model is any object that represents a piece of data. Through them your app represents the world so that it can work with its data, similar to how <a href="https://en.wikipedia.org/wiki/Mental_model">you have mental models about the world around you</a> (e.g. what is night and what is day).<br><br>Think about it. A model backed by a database - like User - just represents a row in that database that holds data about a person. It takes that row and wraps it into an object you can interact with - e.g. to get the person’s name or email.<br><br>Decorators, form, value, service and view objects do the same. They take one or more pieces of data and give you an object to interact with that data in different ways.<br><br>This might be a reductionist way of thinking because controllers and views do the same. But they are novel concepts that are used so often, and that are so important, that separating them out makes building the application easier.<br><br>For instance, I work on an IoT application. One of the most common actions we do is to send messages to a devices. Because it is so novel and so common, we have extracted “device messengers” out of models into their own thing. That way developers don’t have to search for them and we communicate that they are an important part of the app.<br><br>Treating most things as models, and making models heavier, has made the code I write much less surprising.<br><br>Having super skinny objects produces surprising code because you have to be very verbose. Something boring like <strong>user.purchase(line_items).charge(authorization_code: params[:authorization_code])</strong> becomes a convoluted mess like <strong>PurchaseChargerService(purchase: PurchaseBuilderService.new(line_items: line_items).call, form: purchase_form, user: current_user).call</strong>.<br><br>Nobody expects things like <strong>PurchaseChargerService</strong> and <strong>PurchaseBuilderService</strong> to exist, yet alone that you have to call those in succession with a form object and the current user to make a purchase. While most people working on a web shop expect that a user can just purchase something, and that a purchase can be charged.<br><br>By writing skinny objects you are pushing the complexity onto the programmer instead of having it in code.<br><br>I’m not saying you should get rid of <strong>PurchaseChargerService</strong> or <strong>PurchaseBuilderService</strong> (ok, maybe the builder is pushing it a bit, I'd get rid of that one). I’m saying that you should make them as boring as possible.<br><br>For all you know <strong>user.purchase</strong> might call those same objects internally. You don’t have to know about that to make a purchase. It’s boring and it’s beautiful.<br><br>How to make things boring?<br><br>First, embrace the domain you are working with.<br><br>Naming something a service and putting it in its own directory solves no problems, but it can create some. Naming something after what it does and putting it next to the thing it works with tells a story that can help people understand what’s going on.<br><br>Instead of <strong>PurchaseChargerService</strong> that lives in a service objects directory, try naming it <strong>Purchase::Charger</strong> and putting it under the Purchase model (e.g. <strong>app/model/purchase/charger.rb</strong>). You will know just by looking at the file that it has something to do with purchases and charging them.<br><br>Second, write code that mimics how people think about the domain you are working in.<br><br>Expose methods for common actions in your domain. If a <strong>User</strong> can purchase line items, then add a <strong>purchase</strong> method to the User model that accepts line items.<br><br>Don’t push the complexity of figuring out how to make a purchase onto other people, solve it once in code, and explain it well so that others can build on top of your work. Don’t try to be too smart, most often people won’t care how the purchase is made so don’t try to expose it if it’s not needed.<br><br>Third, extract actions into models to tell a story and form new domains.<br><br>Don’t create new classes just to keep things DRY. If your model has many small or a single large method that deal with a single action, extract it into its own model.<br><br>If the <strong>User</strong> model has 5 methods related to purchasing or a single large <strong>purchase</strong> method, extract that into a <strong>Purchase</strong> model and delegate to it. If the <strong>Purchase</strong> model has 10 methods that deal with charging credit cards, extract those into a <strong>Purchase::Charger</strong> model and delegate to it.<br><br>Nobody will think less of you for writing boring code.<br><br>Nobody will fuss about a method, module or class that is understandable but is too pedestrian.<br><br>Nobody wants to work with code that requires a manual to figure out how to do the most basic actions.<br><br>Keep it simple, keep it boring, and don’t surprise other.<br><br><strong><em>Update on 2022-08-27</em></strong><br><br>After sharing this article with a few friends I got a wonderful question which I want to address.<br><br><strong>Q:</strong> If you put everything in <strong>app/models</strong> isn’t that surprising too? People will expect an ActiveRecord-like object and they can get anything.<br><br><strong>A:</strong> Yes that would be surprising, but I don’t see a situation where someone would write <strong>Purchase::Charger.new(purchase: Purchase.all.sample).save</strong> and hit Enter out of the blue.<br><br>If you see <strong>Purchase::Charger</strong> you’d at least be curious why the model name isn’t a noun - the name implies that it’s different.<br><br>Why would someone create and try to save an object they know nothing about? That implies that they are just passing random arguments to random methods expecting things to just work.<br><br>I don't see this as a realistic scenario, I expect people to know what they are doing and if they don't to read up until they do. In other words, I trust them to do good work.<br><br>But I agree that there are cases where it’s beneficial to have a similar interface for some objects. Therefore, I make most objects in <strong>app/models</strong> inherit from <strong>ActiveModel::Model</strong>. There are many reasons for this which I think I’ll cover in a separate article.<br><br>P.S. Many thanks to <a href="https://shime.sh/">Hrvoje S.</a> for reviewing drafts of this article.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/162022-08-27T12:00:00Z2023-11-14T16:28:05ZTake a break<div class="trix-content">
<div>At my first job I used to take a break every time I got stuck on a problem until one day my boss caught me looking out of the window.<br><br>He didn’t say anything the first time he saw me, but when he saw me in the same position some 15 min later he yelled “At least sit by your desk and have your laptop open so that I think you are working instead of slacking off”.<br><br>As I switched jobs, I learned that “at least make it look like you are working” is a prevalent philosophy in most offices. So I adapted and started scrolling through Reddit while on break to seem busy. But this didn’t work out as my boss had hoped for - I started taking more breaks because I couldn’t focus more often.<br><br>Thankfully, I didn’t have to work in an office for the last 4 years so I was able to return to my old routine. But why was this method so much more effective than scrolling Reddit for 15min?<br><br>Rachel and Stephen Kaplan developed a theory that might explain why - <a href="https://en.wikipedia.org/wiki/Attention_restoration_theory">Attention Restoration Theory or ART for short</a>. ART differentiates two kinds of attention humans can give - direct attention and effortless attention.<br><br>We use direct attention when we focus on something - like trying to solve a problem, searching for something, following rules… When giving direct attention we consciously block out most distractions.<br><br>Direct attention is a finite resource - if you use it too much you will run out of it and won’t be able to focus anymore - but it replenishes if it’s not used for a while. You can think of it as a muscle - if you use it too much it becomes sore and weak, but if you give it some rest it can work for much longer.<br><br>When you run out of direct attention you start suffering from attention fatigue - a state in which you are easily irritable, tired and stressed out.<br><br>In contrast, effortless attention is… well… effortless.<br><br>The Kaplans tell us that nature is full of fascinating things to experience and give our effortless attention to - like the shapes of clouds, the wind on your skin, rustling of leaves, the sun breaking through the treetops, the chirps of birds, the sound of water rushing in a creek... So by looking at or being in nature we can replenish our direct attention because we will mostly use effortless attention to process it.<br><br>The reason I took more breaks while scrolling Reddit was over use of direct attention. Reading through posts and commenting takes direct attention to do, therefore it doesn’t replenish during the break and I couldn’t focus as well when I started working again.<br><br>But there is another compounding problem - <a href="https://www.sciencedirect.com/science/article/abs/pii/S0749597809000399">attention residue</a>.<br><br>Attention residue is an effect that prevents humans from switching their attention between tasks effortlessly. When you switch tasks your mind will still pay attention to the previous task for a while and prevent you from fully focusing on the new task.<br><br>Whatever posts I read on Reddit during my break would linger in my mind and steal attention away from the problem I was trying to solve when I started working again. And when I took a break any problem I was working on before would still invade my thoughts and steal attention from what I was doing during my break.<br><br>The problem with scrolling through Reddit while on break was that my attention was depleted before the break, it couldn’t replenish during the break, and then it was split between Reddit and work after the break.<br><br>To be clear, Reddit isn’t the problem here - being unnecessarily busy is. I would have probably felt the same way if I read the news, a book, scrolled through Twitter, solved a Sudoku or played chess.<br><br>Today, when I feel like I’m having a hard time focusing, I go out for a 15min walk if the weather is nice. If the weather doesn’t play along I make myself a cup of tea, look out of the window and enjoy the birds, the clouds passing by, the falling snow, or the sound of wind and rain.<br><br>A proper 15 min break can keep you happy, focused and rested for hours.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI3ND9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--5aeca059d6a552f44001eec588f39913fb1dbb69" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBaElCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--4fcbb988ce8d018dd1cee23ce6ded3dc644749f5/image.jpeg" filename="image.jpeg" filesize="539434" width="1200" height="1600" previewable="true" caption="The sun shining through the tree tops in a forest">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6Mjc0LCJwdXIiOiJibG9iX2lkIn19--6a7dc7b9c42d5bce95edce431b7bfcd20fe0b9cc/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/image.jpeg">
<figcaption class="attachment__caption">
The sun shining through the tree tops in a forest
</figcaption>
</figure></action-text-attachment>P.S. Many thanks to Marko I. for reviewing drafts of this article.<br><br>My inspiration for this article was a conversation I had with a co-worker who didn’t want to take a break when he was stressed out, after which I started encouraging all my co-workers to take a break when they start to feel attention fatigue. This is a cautionary tale for them.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/172022-09-22T12:00:00Z2023-11-14T16:28:05ZThe humble ActiveModel<div class="trix-content">
<div>ActiveModel is one of my most used tools in Rails applications. I use it in service objects, form objects and objects that represent external entities.<br><br>Why? Because it provides a nice interface for validating inputs and results, it can have callbacks for pre and post-processing data, and it integrates well into various Rails conventions.<br><br>A simple login form is my go to example for explaining how nice ActiveModel is to work with.<br><br>I have seen people do login flows in controllers, in User models and in interactors. But no other approach is as straightforward as creating a Login model.</div><pre>#!/usr/bin/env ruby
# /app/models/login.rb
class Login
include ActiveModel::Model
include ActiveModel::Validations::Callbacks
attr_accessor :username, :password
validates :username, presence: true
validates :password, presence: true
validate :validate_user_exists!
validate :validate_password_for_user!
before_validation do
@user = nil
end
def validate_user_exists!
return if user.present?
errors.add(:username, "not found")
end
def validate_password_for_user!
# `authenticate` comes from Rails
# Reference: https://api.rubyonrails.org/classes/ActiveModel/SecurePassword/ClassMethods.html#method-i-has_secure_password
return if user&.authenticate(password)
errors.add(:password, "is invalid")
end
def user
@user ||= User.find_by(username: username)
end
end</pre><div>Which can then be used in a controller like any other model:</div><pre>#!/usr/bin/env ruby
class LoginsController < ApplicationController
def new
@login = Login.new
end
def create
permitted_params = params.require(:login).permit(:username, :password)
@login = Login.new(permitted_param)
if @login.valid?
session[:current_user_id] = @login.user.id
redirect_to root_path, notice: "Welcome back!"
else
render :new, status: :unprocessable_entity
end
end
end</pre><div>You can use <strong>form_for</strong> without extra arguments or configuration, <strong>I18n</strong> works for the attributes and error messages just like it does for ActiveRecord models, and if the login fails the fields get repopulated.<br><br>It looks and feels like you are working with an ActiveRecord model, which most of us know and love.<br><br>When it comes to Service objects, the ActiveModel approach offers a way to communicate success and failure through validations. You can check the inputs with <strong>valid?</strong>, or you can check the result, and return a nice error message.</div><pre>#!/usr/bin/env ruby
class Device::WarehouseReservator < ApplicationModel
attr_accessor :device, :user
validates :device, presence: true
validates :user, presence: true
after_initialize do
self.user ||= device.creator
end
def reserve!
return false if invalid?
response = HTTP.post("http://device.inventory.com/api/v1/register", data: payload)
if response.ok?
errors.add(:device, “failed to register”)
return false
end
self.warehouse_reference = device.create_warehouse_reference(reservation_id: JSON.parse(response.body).fetch(:id))
end
def payload
{ device_id: device.id, name: device.name, reservation_holder: user.name }
end
end
reservator = Device::WarehouseReservator.new(user: User.all.sample, device: Device.all.sample.reserver!
reservator.valid? # => true
reservator.reserver!
Reservator.warehouse_reference.id # => 1
reservator = Device::WarehouseReservator.new(user: nil, device: nil)
reservator.valid? # => false
reservator.reserver! # => false
reservator.errors.full_messages # => ["User missing", "Device missing"]
<br><br></pre><div>This is a much nicer way to communicate what went wrong than raising and catching errors, returning symbols or custom Error objects.</div><ul>
<li>validate and coerce inputs by type through <a href="https://api.rubyonrails.org/classes/ActiveModel/Attributes/ClassMethods.html">ActiveModel::Attributes</a>
</li>
<li>handle password storage and checking through <a href="https://api.rubyonrails.org/classes/ActiveModel/SecurePassword/ClassMethods.html#method-i-has_secure_password">has_secure_password</a>
</li>
<li>add custom callbacks with <a href="https://api.rubyonrails.org/classes/ActiveModel/Callbacks.html">ActiveModel::Callbacks</a> (I usually implement an <strong>after_initialize_callback</strong> in <strong>ApplicationModel</strong>)</li>
<li>track changes to attributes using <a href="https://api.rubyonrails.org/classes/ActiveModel/Dirty.html">ActiveModel::Dirty</a>
</li>
</ul><div>ActiveModel is quite a versatile tool for that alone, but there is much more that it can do like:<br><br>P.S. Many thanks to <a href="https://shime.sh/">Hrvoje S.</a> For reviwing this article.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/182022-09-30T12:00:00Z2023-11-14T16:28:05Z“Having a monolith is a single point of failure”<div class="trix-content">
<div>I recently took part in a discussion that brought to light the most unusual argument against having a monolith that I have ever heard - that a monolith is a single point of failure.<br><br>I want to make clear that I consider monoliths and microservices neither good nor bad, or universally better or worse at any particular job. To me these architectures are just tools that work better in some settings and worse in others. As I explained in <a href="https://stanko.io/misguided-mark-misrepresents-micro-services-a-story-about-paradigms-XBMbcWf6RRkz">“Misguided Mark misrepresents Micro-services“</a>, to me they are just paradigms - different ways to see and explain the same thing.<br><br>The details of the discussion aren’t important for debunking the argument. The important thing is that the discussion was centered around should a new feature be added to a monolith or should it be a microservice.<br><br>We covered quite a few common arguments for microservices all of which fell apart when examined. And then the other side brought up that having a monolith is a single point of failure.<br><br>This argument shows a deep misunderstanding of what microservices are and how they work.<br><br>People usually think about monoliths in terms of classes and methods (or modules and functions), but they think about microservices in terms of services and APIs.<br><br>Services and APIs are the same thing as classes and methods.<br><br>They are two sides of the same coin - on one side everything is in one program, on the other each class is its own program.<br><br>In other words, they are two paradigms through which people build and think about applications. They are talking about the same thing just from different perspectives.<br><br>If you have a monolith in which a call to the process method on the class Payment internally calls the authorize method on class User, then you have two places where something can go wrong. Either Payment#process can error or User#authorize can error. In both cases the call to Payment#process has erred.<br><br>If you have microservices in which a request to the process_payment API on the Payment microservice internally makes a request to the authorize_token API on the User microservice, then you have two places where something can go wrong. Either the request to process_payment on the Payment service returns a 500 or the request to authorize_token on the User service returns a 500. In both cases the request to process_payment on the Payment microservice has erred.<br><br>The number of failure points is the same.<br><br>Though in reality, the microservices example has two more failure points because each request to a service can also fail due to various problems - like a misconfigured load balancer, or the machine running out of file descriptors, or a congested router in the data center, or any other reason a network request might fail…<br><br>So where does this single point of failure argument come from?<br><br>I believe that Netflix and a simple case of not thinking with your own head are the source of this odd argument.<br><br>As most programmers know, Netflix is a big proponent of microservices. They have been making headlines across various sites popular in the IT industry how microservices have enabled them to weather any outage. They even made a tool called <a href="https://github.com/netflix/chaosmonkey">Chaos Monkey</a> that turns off services at random to test how resilient your system is.<br><br>So if it works for Netflix why wouldn’t it work for us? Right?<br><br>Let’s apply Chaos Monkey to our example with two services and see.<br><br>If all services are on, payment processing works. But if one or both are turned off, payment processing fails. It can process payments only 25% of the time. A monolith would perform much better, it can either be on or off meaning that it can process payments 50% of the time.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI3NT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--adde4441a0c6daecf02a090d948c0bbe2de482c1" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBaE1CIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--c90f3e9d6e357f6174afa89a83bffa69811f1dd5/example_1.gif" filename="example_1.gif" filesize="56094" width="1024" height="559" previewable="true" caption="Animation that shows in which scenarios the system is up or down depending on which of the two services is on or off">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6Mjc1LCJwdXIiOiJibG9iX2lkIn19--be2e4046683c754ac34d5ef16ce3853f0deb8d83/example_1.gif">
<figcaption class="attachment__caption">
Animation that shows in which scenarios the system is up or down depending on which of the two services is on or off
</figcaption>
</figure></action-text-attachment>What gives? Maybe if we had two of every service it would be more resilient?<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI3Nj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--968db05b682832325d7484708a70747b920b57cc" content-type="image/gif" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBaFFCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--bc2e1b3be58f546e4fc83efd5741f6be9ccfc339/example_2.gif" filename="example_2.gif" filesize="282926" width="1024" height="559" previewable="true" caption="Animation that shows in which scenarios the system is up or down depending on which of the four services is on or off">
<figure class="attachment attachment--preview attachment--gif">
<img loading="lazy" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6Mjc2LCJwdXIiOiJibG9iX2lkIn19--6444a4bfe14b5be8f03dbf37864b2dde13bffe7c/example_2.gif">
<figcaption class="attachment__caption">
Animation that shows in which scenarios the system is up or down depending on which of the four services is on or off
</figcaption>
</figure></action-text-attachment>It would, but only 50% of the time. A monolith would perform much better, as it would fail to process the payment only when all instances are off which means it would process payments 75% of the time.<br><br>The point of Chaos Monkey is to test how resilient your architecture is. It works with services because Netflix is invested in microservices and they have built a tool to test the resiliency of their architecture.<br><br>But architecture is so much more than microservices or monoliths. It’s how flexible the system is, how easily it can be extended, how easily a team can maintain it, and how resilient it is to failure.<br><br>The fact that Netflix has built a resilient architecture with microservices doesn’t mean that you can’t achieve the same with other architectures.<br><br>Resilience to failure can be addressed both in microservices and in monoliths through the same patterns - <a href="https://github.com/yammer/circuitbox">circuit breakers</a>, <a href="https://en.wikipedia.org/wiki/Strategy_pattern">policies/strategies</a>, and others.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/192022-10-07T12:00:00Z2023-11-14T16:28:05Z140 million rows later<div class="trix-content">
<div>At work, as part of a project I’m working on, I wanted to add a new reference to a table. ”Simple enough” - I thought. Spoiler alert, it wasn’t.<br><br>Adding a reference from one table to another is straight forward in Rails.<br><br>You create a migration using <strong>rails generate</strong> and then write in it something like <strong>add_reference :device_event_logs, :door, foreign_key: true, null: true</strong>. And then you run <strong>rails migrate</strong>. Boom. Done!<br><br>This is where my problems began.<br><br>The table I was adding the reference to holds event logs for all devices. Each event represents some kind of user interaction with the device. I know that there are a decent number of devices in the field, and a decent number of users in the system.<br><br>Since adding a reference can lock the table (prevent reading or writing to it) until the migration is done I wanted to be sure that it wouldn’t cause any problems.<br><br>I opened the DB console, entered <strong>SELECT COUNT(1) FROM device_event_logs</strong> and hit enter. A few seconds passed and nothing happened - “there must be a lot of records in the table” I thought. Then 30 more seconds passed and still no result - this was a bad omen. After 45 sec the query returned a count of 140 million rows.<br><br>Yikes. This means that adding the reference would take quite a while and use up a decent chunk of storage space.<br><br>Next I went to check how often this table is written to - to determine how big of an impact a few minutes of downtime would have. So I checked the statistics tables in MariaDB and the number of requests to the controller that the devices call to log events. There were around 30 inserts per second.<br><br>Yikes again. This meant that 5 min of downtime would cause 9000 failed requests - potentially 9000 unhappy customers.<br><br>I wanted to find a way to add this reference quickly, without locking the table and without using too much storage space.<br><br>Digging through the Rails documentation for <strong>add_reference</strong> I didn’t find anything that could help me solve this, so I resorted to adding the reference manually, step by step.<br><br>In Rails, a reference like <strong>add_reference :device_event_logs, :door, foreign_key: true, null: true</strong> creates a column to store the ID of the row being referenced in the other table, creates and index on that column and adds a foreign key constraint from that column to the referenced table’s primary key.<br><br>Now I read through MariaDB’s documentation to see how I could optimize each of these three steps.<br><br>Turns out that adding the column is already optimized out of the box so I could just add it through <strong>add_column :device_event_logs, :door_id, :integer, null: true</strong>. That would create the column in less than a second and wouldn’t lock the table.<br><br>When creating an index, the MariaDB docs say that one can specify an algorithm with which the index is generated. Some algorithms are locking, some aren’t, some create a copy of the whole table, some don’t. But if you don’t pass an algorithm MariaDB will try to use the fastest, least locking, and most space efficient one possible.<br><br>Sounds great, but as I have been burned before by documentation leading to incorrect assumptions I decided to verify this.<br><br>So I ran <strong>CREATE INDEX foobar ON device_event_logs (door_id)</strong> on our staging database. It took about 20 min to complete, increased the database’s size by nearly double and I couldn’t insert or select from the table while the index was being created.<br><br>Good thing I checked.<br><br>Then I tired specifying the algorithm explicitly with <strong>CREATE INDEX foobar ON device_event_logs (door_id) ALGORITHM inplace</strong> and it erred out with a message saying that <strong>INPLACE</strong> isn’t supported on this table. I also tried all other algorithms none of which were supported, except for the slow and locking one - <strong>COPY</strong>.<br><br>This threw me down a rabbit hole. I spent a day trying to figure out why the better algorithms weren’t supported until I finally found a paragraph in MariaDB’s documentation explaining that tables created prior to MariaDB version 10.something.something can’t use the newer algorithms. So I checked when the staging table was created, and sure enough, it was way before that version was released.<br><br>Since the staging environment was setup after the production environment I’d surely run into the same problem there. What now?<br><br>After some more digging I found out that this problem can be solved by running <strong>OPTIMIZE TABLE device_event_logs</strong>. Which will do all sorts of magic, fairly quickly and with a lock on the table, to get the table working with all features of the current MariaDB version. So I ran it, and a second later it was done.<br><br>Now I could add the index with <strong>add_index :device_event_logs, :door_id, algorithm: :inplace</strong>. The algorithm option isn’t necessary, but I wanted the migration to fail if it would lock the table - in case I forgot to optimize it.<br><br>When creating a foreign key constraint, the MariaDB docs say that, you can pass an algorithm (just like with an index), and a <strong>LOCK=NONE</strong> option. But <strong>add_foreign_key</strong> in rails accepts neither of these options, so I had to resort to writing SQL.<br><br>But when I tried to run the migration it failed, stating that an index on the <strong>doors</strong> table’s <strong>id</strong> column was missing. What? How? It’s the primary key!<br><br>After some more digging I found out that the column types in a foreign key have to match, else you get that error.<br><br>I added an integer column to <strong>device_event_logs</strong> thinking it would automatically use <strong>bigint</strong> which is the default primary key type in Rails, but it didn’t. So I went back and changed the add column to <strong>add_column :device_event_logs, :door_id, :bigint, null: true</strong>.<br><br>Finally I could add the foreign key, but it was very slow. I had a hunch that it was due to MariaDB checking if the key actually exists in the other table.<br><br>To speed it up I disabled referential integrity by wrapping the SQL command in a <a href="https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/AbstractAdapter.html#method-i-disable_referential_integrity"><strong>disable_referential_integrity</strong> block</a> after which the foreign key was added in less than a second.<br><br>The downside of disabling referential integrity is that a foreign key might point to something that doesn’t exist, but that wasn’t a problem in this migration since the column the foreign key was added to was completely empty.<br><br>My migration looked like this in the end<br><br>The production migration took 15 min, caused no locks or downtime, and increased the size of the database by a gigabyte.<br><br>All in all, a big win and learning experience.</div><pre># `add_reference :device_event_logs, :door, foreign_key: true, null: true` is equuivalent to
add_column :device_event_logs, :door_id, :bigint, null: true
add_index :device_event_types, :door_id
add_foreign_key :device_event_types, :doors
<br><br>execute(
<<~SQL
ALTER TABLE device_event_logs
ADD CONSTRAINT fk_rails_53a1f7c81d
FOREIGN KEY IF NOT EXISTS (door_id)
REFERENCES doors (id),
ALGORITHM = INPLACE ,
LOCK = NONE;
SQL
)
<br><br>add_column :device_event_logs, :door_id, :bigint, null: true
add_index :device_event_logs, :door_id, algorithm: :inplace
# https://api.rubyonrails.org/classes/ActiveRecord/Migration.html#method-i-reversible
reversible do |dir|
dir.up do
disable_referential_integrity do
execute(
<<~SQL
ALTER TABLE device_event_logs
ADD CONSTRAINT fk_rails_53a1f7c81d
FOREIGN KEY IF NOT EXISTS (door_id)
REFERENCES doors (id),
ALGORITHM = INPLACE,
LOCK = NONE;
SQL
)
end
end
end
<br><br></pre>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/202022-10-10T12:00:00Z2023-11-14T16:28:05ZHow an index made rendering slow<div class="trix-content">
<div>I noticed that a view I was working on was rendering much slower than I would expect it to. The view showed a list of events, together with the person that generated the event and the device that the event belongs to. It took nearly half a second to render 25 events, while other similar pages render in a couple of milliseconds.<br><br>At first I thought that I had an N+1 - that each rendered event went to fetch its person and device from the database, generating 2 database queries per rendered row.<br><br>But in the controller the associations were clearly <a href="https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-preload">preloaded</a>, and even <a href="https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-strict_loading">marked for strict loading</a>. Not only was an N+1 problem avoided, but the system would raise an error if one would occur.</div><pre>#!/usr/bin/ruby
def index
@events = Device::EventLog.all
.preload(:user, :device)
.strict_loading
.page(params[:page])
.per(25)
end</pre><div>
<strong>So why did the view render so slow?</strong><br><br>I went to look at the development server logs - which, in Rails, show a lot of useful information for debugging problems like these - and I saw this:</div><pre>Started GET "/events" for 172.23.0.1 at 2022-10-10 08:57:51 +0000
Processing by EventsController#index as HTML
Parameters: {}
...
Device::EventLog Load (2.8ms) SELECT `device_event_logs`.* FROM `device_event_logs` LIMIT 25 OFFSET 0
↳ /app/views/events/index.html.erb:10:in `_app_views_events_index_html_erb___1845829941033837665_155040'
...
User Load (79.75ms) SELECT `users`.* FROM `users` WHERE `users`.`deleted_at` IS NULL AND `users`.`id` IN (97212, 2526, ...)
↳ /app/views/events/index.html.erb:10:in `_app_views_events_index_html_erb___1845829941033837665_155040'
...
Device Load (79.95ms) SELECT `devices`.* FROM `devices` WHERE `devices`.`deleted_at` IS NULL AND `devices`.`id` IN (8368, 137, ...)
↳ /app/views/events/index.html.erb:10:in `_app_views_events_index_html_erb___1845829941033837665_155040'
...
Completed 200 OK in 251ms (Views: 78.7ms | ActiveRecord: 162.5ms | Allocations: 226640)</pre><div>It was clearly visible that the preload worked. ActiveRecord made three queries to the database - one to get the events, another to get the users, and a third to get the devices.<br><br><strong>But where does the WHERE deleted_at IS NULL come from?</strong><br><br>I looked at the <strong>User</strong> and <strong>Device</strong> models. Both invoke the <strong>acts_as_paranoid</strong> method which the <a href="https://github.com/rubysherpas/paranoia">paranoia gem</a> adds to <strong>ActiveRecord::Record</strong>.<br><br>We use that gem at work to soft delete records in the database - the data is kept in the table, it just isn’t returned unless specifically requested.<br><br>Paranoia knows which records are deleted by checking a <strong>deleted_at</strong> column that has to be added to each model that can act as paranoid. When the record is deleted, <strong>deleted_at</strong> is set to the current timestamp. A record can be restored (un-deleted) by setting <strong>deleted_at</strong> back to <strong>nil</strong>.<br><br>Paranoia filters out deleted records by adding <strong>WHERE deleted_at IS NULL</strong> to all queries.<br><br><strong>Why are the queries to fetch the users and devices so slow?</strong><br><br>In situations like these I reach for <a href="https://api.rubyonrails.org/classes/ActiveRecord/Relation.html#method-i-explain">the <strong>explain</strong> method on an ActiveRecord relation</a>. It tells the database to explain how it will execute the query. This shows important information like the indices it plans to use, in which order it wants to filter rows, and more.<br><br>Here is the explain result for one of the slow queries:</div><pre>#!/usr/bin/ruby
User.where(id: [
97212, 2526, 15027, 11982, 10846,
17091, 19528, 10731, 7785, 23240,
23238, 7244, 7221, 6993, 11959,
16931, 38234, 7006, 8993, 6766,
39565, 31681, 7088, 7786, 34029
]).explain
# User Load (79.75ms) SELECT `users`.* FROM `users` WHERE `users`.`deleted_at` IS NULL AND `users`.`id` IN (97212, 2526, 15027, 11982, 10846, 17091, 19528, 10731, 7785, 23240, 23238, 7244, 7221, 6993, 11959, 16931, 38234, 7006, 8993, 6766, 39565, 31681, 7088, 7786, 34029)
# => EXPLAIN for: SELECT `users`.* FROM `users` WHERE `users`.`deleted_at` IS NULL AND `users`.`id` IN (97212, 2526, 15027, 11982, 10846, 17091, 19528, 10731, 7785, 23240, 23238, 7244, 7221, 6993, 11959, 16931, 38234, 7006, 8993, 6766, 39565, 31681, 7088, 7786, 34029)
# +----+-------------+-------+------+-----------------------------------+---------------------------+---------+-------+------+-----------------------+
# | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
# +----+-------------+-------+------+-----------------------------------+---------------------------+---------+-------+------+-----------------------+
# | 1 | SIMPLE | users | ref | PRIMARY,index_users_on_deleted_at | index_users_on_deleted_at | 6 | const | 25 | Using index condition |
# +----+-------------+-------+------+-----------------------------------+---------------------------+---------+-------+------+-----------------------+
# 1 row in set (0.00 sec)</pre><div>The result shows that the database sees two indices with which it can filter the table to generate the result - <strong>PRIMARY</strong> and <strong>index_users_on_deleted_at</strong>.<br><br>Out of these two it choose to use <strong>index_users_on_deleted_at</strong>, which is a bit odd since the query mostly filters by IDs.<br><br><strong>Why did it choose that index instead of using the primary key? And why is that so slow?</strong><br><br>I ran <strong>SHOW INDEX</strong>, an introspection tool in MariaDB that shows statistics about indices on a table. The query planner uses those same statistics to plan how to filter the requested rows. Understanding what the query planner sees can explain why MariaDB picked <strong>index_users_on_deleted_at</strong> instead of <strong>PRIMARY</strong><br><br>These are the statistics for the <strong>users</strong> table:</div><pre>+---------+------------+-----------------------------+--------------+--------------+-----------+-------------+----
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | ...
+---------+------------+-----------------------------+--------------+--------------+-----------+-------------+-----
| "users" | 0 | "PRIMARY" | 1 | "id" | "A" | 3202800 | ...
| "users" | 1 | "index_users_on_deleted_at" | 1 | "deleted_at" | "A" | 4003 | ...
+---------+------------+-----------------------------+--------------+--------------+-----------+-------------+-----</pre><div>The <strong>Cardinality</strong> column shows that the <strong>PRIMARY</strong> index has millions of values indexed, while <strong>index_users_on_deleted_at</strong> only has a few thousand.<br><br><strong>But why is that index so slow?</strong><br><br>The decision by the query planner to use <strong>index_users_on_deleted_at</strong> felt off. It seemed like a mistake so I dug deeper.<br><br>MariaDB offers query optimizer tracing, which is fancy term for a “detailed explanation why it did what it did”.<br><br>I opened a DB console and ran <strong>SET optimizer_trace='enabled=on';</strong> to enable tracing. Then I ran the slow query and right after that I checked to see the optimizer trace with <strong>SELECT * FROM information_schema.optimizer_trace LIMIT 1;</strong>.<br><br>This returned a large JSON object, which showed every decision that the query planner made and explained why it chose the approach it did.<br><br>The trace showed that the <strong>PRIMARY</strong> index had a much lower execution cost (was much faster) than <strong>index_users_on_deleted_at</strong>, but it chose to use <strong>index_users_on_deleted_at</strong> for “rowid filtering”. I had no clue what “rowid filtering” was so I <a href="https://mariadb.com/kb/en/rowid-filtering-optimization/">went to look it up in the documentation</a>.<br><br>Row ID filtering is an optimization. The basic idea is this - if you have two indices on a table, an you are filtering by both; more often than not it’s quicker to filter by the more restrictive index first, and then scan the result to filter by the less restrictive one.<br><br>Or in other words, it’s better to pick the index that eliminates most values first so that you have fewer rows to work with, and then scan through the result and filter by any other filters. Makes sense.<br><br>MariaDB, unlike PostgreSQL, doesn’t support conditional indexing. This means that each index you add to a table indexes the whole table.<br><br>This is great when you have many rows with different values in the indexed column, but it can backfire when you have a table that has many rows with the same value.<br><br>In my case the <strong>deleted_at</strong> column has around 4k different values in about 3 million rows. But a single value - <strong>NULL</strong> - of those 4k is associated with nearly all 3 million rows. This means that the database will load and scan through 3 million records to find the ones matching the <strong>IN</strong> clause - which is extremely slow.<br><br><strong>How do I make the query fast?</strong><br><br>Luckily MariaDB, unlike PostgreSQL, provides an escape hatch for situations like this. Through the <strong>USE INDEX</strong> option, it allows you to specify the index it should use when filtering.<br><br>ActiveRecord doesn’t have a method for <strong>USE INDEX</strong> but it can be added by passing a string to the <strong>from</strong> method.<br><br>Telling the planner to use the <strong>PRIMARY</strong> index speeds up the query drastically. The execution time fell from 80 ms to 1 ms.</div><pre>#!/usr/bin/ruby
User.from('users USE INDEX(PRIMARY)')
.where(id: [97212, 2526, 15027, 11982, 10846, 17091, 19528, 10731, 7785, 23240, 23238, 7244, 7221, 6993, 11959, 16931, 38234, 7006, 8993, 6766, 39565, 31681, 7088, 7786, 34029])
.explain
# User Load (0.7ms) SELECT `users`.* FROM users USE INDEX(PRIMARY) WHERE `users`.`deleted_at` IS NULL AND `users`.`id` IN (97212, 2526, 15027, 11982, 10846, 17091, 19528, 10731, 7785, 23240, 23238, 7244, 7221, 6993, 11959, 16931, 38234, 7006, 8993, 6766, 39565, 31681, 7088, 7786, 34029)
# => EXPLAIN for: SELECT `users`.* FROM users USE INDEX(PRIMARY) WHERE `users`.`deleted_at` IS NULL AND `users`.`id` IN (97212, 2526, 15027, 11982, 10846, 17091, 19528, 10731, 7785, 23240, 23238, 7244, 7221, 6993, 11959, 16931, 38234, 7006, 8993, 6766, 39565, 31681, 7088, 7786, 34029)
# +----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
# | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
# +----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
# | 1 | SIMPLE | users | range | PRIMARY | PRIMARY | 8 | NULL | 25 | Using where |
# +----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
# 1 row in set (0.00 sec)</pre><div>But this doesn’t solve my problem. I can’t call the <strong>from</strong> method in a preload query - that query is built and executed by ActiveRecord internally.<br><br>Though, now that I understand the problem I can trick the query planner into doing what I want.<br><br>If I replace <strong>preload</strong> with <strong>eager_load</strong> I can force the query planner to use at least one <strong>PRIMARY</strong> index and thereby speed things up considerably.</div><pre>#!/usr/bin/ruby
Device::EventLog.all
.eager_load(:user, :device)
.strict_loading
.explain
# Device::EventLog Load (2.7ms) ...
# +----+-------------+-------------------+--------+-------------------------------------+---------+---------+-----------------------------------------+------+-------------+
# | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
# +----+-------------+-------------------+--------+-------------------------------------+---------+---------+-----------------------------------------+------+-------------+
# | 1 | SIMPLE | device_event_logs | ALL | NULL | NULL | NULL | NULL | 25 | Using where |
# | 1 | SIMPLE | users | eq_ref | PRIMARY,index_users_on_deleted_at | PRIMARY | 8 | development.device_event_logs.user_id | 25 | Using where |
# | 1 | SIMPLE | devices | eq_ref | PRIMARY,index_devices_on_deleted_at | PRIMARY | 8 | development.device_event_logs.device_id | 25 | Using where |
# +----+-------------+-------------------+--------+-------------------------------------+---------+---------+-----------------------------------------+------+-------------+
<br><br></pre><div>And it worked! Both <strong>PRIMARY</strong> indices were used when filtering, and the query returns in 3 ms.<br><br><strong>Why did this work?</strong><br><br>I had a hunch that a <strong>LEFT OUTER JOIN</strong> would force the optimizer to use at least one <strong>PRIMARY</strong> index on the joined tables. Because <strong>eager_load</strong> does the same thing as <strong>preload</strong>, in this case, but using a <strong>LEFT OUTER JOIN</strong> it was the perfect solution.<br><br>And I was right. The query optimizer trace for the <strong>eager_load</strong> query showed that the optimizer tries to apply the same optimization as before, but its cost is higher than using the <strong>PRIMARY</strong> index.<br><br>The <strong>Device::EventLog</strong> table doesn’t have any more restrictive index that is also used in the query so it can’t apply the same optimization. While the joined tables are implicitly filtered by so many IDs that the cost of using the <strong>PRIMARY</strong>index goes way down compared to the optimization.<br><br>At least that’s what I think is going on.<br><br>Either way, sometimes replacing <strong>preload</strong> with <strong>eager_load</strong> can utilize indices better and therefore speed up query execution.<br><br>P.S. Because of this problem I took a deep dive into query planners and optimizers in both PostgreSQL and MariaDB. This is a fascinating topic, a science on its own, and a world with very different opinions about what the “correct” solution is. It made me appreciate the “no hints” approach that PostgreSQL takes, but at the same time showed me the value of escape hatches that MariaDB provides. It made me see that there really is no one-size-fits-all solution for SQL databases - as always in software, you stick with what works for you.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/212022-10-23T12:00:00Z2023-11-14T16:28:05ZA week in Helsinki<div class="trix-content">
<div>Last Wednesday my girlfriend and I woke up at 4 AM and headed to the Zagreb Airport to catch a flight to Helsinki.<br><br>I haven’t been in Helsinki since <a href="https://www.hackjunction.com/">Junction 2016</a>, but with <a href="https://euruko.org/">EuRuKo 2022</a> (the European Ruby Konference - this is not a typo) being there this was the prefect opportunity to visit again.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI5MT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--bfb348c937f1a2d8d16a8fe779da39ce34dec006" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBaU1CIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--7bcbe31b29b9cdc16ed919cb9637835389bbb07b/231529FF-ACE2-41AC-9B92-17DF332FF25F.jpeg" filename="231529FF-ACE2-41AC-9B92-17DF332FF25F.jpeg" filesize="1097912" width="3024" height="4032" previewable="true" caption="Above the clouds somewhere between Zagreb and Munich">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjkxLCJwdXIiOiJibG9iX2lkIn19--2ce32c250112e69ab46c198eb2a089ef800ab4ec/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/231529FF-ACE2-41AC-9B92-17DF332FF25F.jpeg">
<figcaption class="attachment__caption">
Above the clouds somewhere between Zagreb and Munich
</figcaption>
</figure></action-text-attachment>One hour after boarding our flight in Zagreb we were in Munich where we had a 40min layover. I though we would have to run through the airport to catch the next flight, but to my surprise there was a person with a “Helsinki” sign waiting for us just as we stepped off the airplane.<br><br>They took us with a van from one part of the airport to a small border crossing to check our passports. This confused me at first, but it occurred to me later that it was due to the Schengen area rules. Croatia isn’t a member yet so we have to go through passport control.<br><br>After the passport check the man took out a key and started unlocking a door that until that point I didn’t even notice existed. This was the first in a series of doors that he opened. Each lead to a hallway that connected to a staircase or elevator and then to another locked door that eventually spit us out right next to our gate. It felt like we were going through a pocket dimension or a worm hole.<br><br>A trip that would take about 40 to 50 min of running through the airport was cut short to just 30 min of inter-dimensional airport travel.<br><br>The next flight wasn’t as exciting as the previous one. We landed in Helsinki some two hours later and went to the baggage clam to find that our luggage went on trip of it’s own, but neither we nor Lufthansa knew where to. Thanks Lufthansa.<br><br>A taxi ride later we were in our hotel. With nothing to unpack we met up with a few friends and went for a walk towards the city center.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzMwNz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--27a81e2f3950b3f2c5e51bfed8306f36e4f1997d" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBak1CIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--fca6581bac6e25e0346a3a0616d2e2053711a24b/8DC90C0D-A958-423E-AF94-84005DC5E9A9.jpeg" filename="8DC90C0D-A958-423E-AF94-84005DC5E9A9.jpeg" filesize="4741592" width="3024" height="4032" previewable="true" caption="Statues at the Helsinki railway station">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzA3LCJwdXIiOiJibG9iX2lkIn19--93c5dddd86e9eb8d1aafac12db3cbe4ee625fc2b/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/8DC90C0D-A958-423E-AF94-84005DC5E9A9.jpeg">
<figcaption class="attachment__caption">
Statues at the Helsinki railway station
</figcaption>
</figure></action-text-attachment>The walk turned into lunch at the Zetro - a tractor (farming machinery) themed restaurant. Inside which everything is dimly lit, there are tractor tables, and all kinds of farming equipment. When you sit at a table the waiters light a few candles without which it would be hard to read the menu.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzMwOD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--44aba93898246563f79ae3715f57a43542be98a9" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBalFCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--06ec68eb1548720189cd6dda6fb07893415486f0/E932F629-A185-495F-9E8B-5429BAD99FFC.jpeg" filename="E932F629-A185-495F-9E8B-5429BAD99FFC.jpeg" filesize="1136309" width="2132" height="3530" previewable="true" caption="A tractor turned into a dining table in a room dimly lit by red light">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzA4LCJwdXIiOiJibG9iX2lkIn19--708dbef7e65a85d26936055882ec82c1347b125e/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/E932F629-A185-495F-9E8B-5429BAD99FFC.jpeg">
<figcaption class="attachment__caption">
A tractor turned into a dining table in a room dimly lit by red light
</figcaption>
</figure></action-text-attachment>Everything on the menu has a cheeky name and a funny description. I ordered the “Momma’s boy meet balls”, while my girlfriend ordered a steak that is allegedly cooked on the bonnet of a Zetro tractor. Both meals were amazing as was the service.<br><br>The following day was the first day of EuRuKo.<br><br>The organizers really outdid themselves. The venue was the <a href="https://en.wikipedia.org/wiki/Paasitorni">Paasitorni</a>, a beautiful and ornate workers’ house from the early 1900s. Registration was quick. There weren’t queues for food, water or hot beverages - what more can you ask for?<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzMwOT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--0dd26d6e42995b1e6e717fdaf0beea22dfd02af9" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBalVCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--33d93843dbdcd4c8f21f13712c8cc3464346da51/4319989F-6CE8-44CD-B105-D9C52271C8A2.jpeg" filename="4319989F-6CE8-44CD-B105-D9C52271C8A2.jpeg" filesize="2897524" width="2887" height="3926" previewable="true" caption="Inside of the ornate room where the conference was held">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzA5LCJwdXIiOiJibG9iX2lkIn19--a255db6a7d9861e09fed7b8d300b5126cfa0c980/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/4319989F-6CE8-44CD-B105-D9C52271C8A2.jpeg">
<figcaption class="attachment__caption">
Inside of the ornate room where the conference was held
</figcaption>
</figure></action-text-attachment>All talks are <a href="https://2022.euruko.org/videos/">already available for everyone to watch</a>.<br><br>My favorites are <a href="https://www.youtube.com/watch?v=MtweO89OsUM">How music works, using Ruby by Thijs Cadier</a>, <a href="https://www.youtube.com/watch?v=2aVyTtxs0GU">Implementing Object Shapes in CRuby by Jemma Issroff</a><strong>,</strong> <a href="https://www.youtube.com/watch?v=cNLscH0sGz4">The Technical and Organizational Infrastructure of the Ruby Community by Adarsh Pandit</a> and <a href="https://www.youtube.com/watch?v=gnX1o8GBed0">Looking Into Peephole Optimizations by Maple Ong</a>.<br><br>But the magic on conferences is meeting people. I had many wonderful conversations with various people about how they run their applications (thanks Shopify team), and about the internals of the Ruby virtual machine. Exciting times are ahead for Ruby with version 3.2 and beyond.<br><br>After the conference we went out for dinner and found a newly opened Korean barbecue named Oppa. This was our first time in a Korean BBQ and it was an experience. We had so much fun, and the food was so delicious, that we came back the next evening with a whole group of friends.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzMxMD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--5e35bfce2da04cbdc94f06edbfd1d39c6d4df76e" content-type="video/mp4" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBallCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--778c9c3e098c54c0771774545f1f50b369b5f18d/koreanbbq2.mp4" filename="koreanbbq2.mp4" filesize="19401846" previewable="true" caption="A few pieces of meat on a Korean barbecue plate">
<figure class="attachment attachment--preview attachment--mp4">
<video controls="controls" autoplay="autoplay" loop="loop" muted="muted" loading="lazy" poster="/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzEwLCJwdXIiOiJibG9iX2lkIn19--4f79370eafdf519450910a8eb387fc8c3e7ef884/eyJfcmFpbHMiOnsiZGF0YSI6eyJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--21cb38a28c52d6c3cf41ca159781d10834673b37/koreanbbq2.mp4" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzEwLCJwdXIiOiJibG9iX2lkIn19--4f79370eafdf519450910a8eb387fc8c3e7ef884/koreanbbq2.mp4"></video>
<figcaption class="attachment__caption">
A few pieces of meat on a Korean barbecue plate
</figcaption>
</figure></action-text-attachment>On Saturday we took a ferry to <a href="https://en.wikipedia.org/wiki/Suomenlinna">Suomenlinna</a>, an island some 10 min away from the city center.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzMxMT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--8e0589d7c84e12540fc2120233b395898b249bf3" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBamNCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--4f9611eb40b8e6bdf86ed7836c94420fee9e735a/2DA6F8A1-576F-46C8-9937-51E4EDD3F5AA.jpeg" filename="2DA6F8A1-576F-46C8-9937-51E4EDD3F5AA.jpeg" filesize="1074528" width="1536" height="2048" previewable="true" caption="A view of Helsinki from the harbor">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzExLCJwdXIiOiJibG9iX2lkIn19--ee41c1e6f2e6d5b6dc5cd482915cd84f94cfcc28/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/2DA6F8A1-576F-46C8-9937-51E4EDD3F5AA.jpeg">
<figcaption class="attachment__caption">
A view of Helsinki from the harbor
</figcaption>
</figure></action-text-attachment>We saw a few small islands with a single house along the way. It must be wonderful to live in such a beautiful, secluded, wooden house, only 10 min away from anything you might need or want to visit.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzMxMj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--b04561fed7f8d61cafc49762be76709ed4a83634" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBamdCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--85d3c5983ef5802c82235605852f5d1c15af0506/509D914E-75C1-401E-BF39-579088CBDA86.jpeg" filename="509D914E-75C1-401E-BF39-579088CBDA86.jpeg" filesize="955793" width="2048" height="1536" previewable="true" caption="A beautiful, red painted, wooden house on a small island right outside the Helsinki harbor">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzEyLCJwdXIiOiJibG9iX2lkIn19--c2807a9ca6ba8dbc4fa1d2aa8d031bb8290bac87/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/509D914E-75C1-401E-BF39-579088CBDA86.jpeg">
<figcaption class="attachment__caption">
A beautiful, red painted, wooden house on a small island right outside the Helsinki harbor
</figcaption>
</figure></action-text-attachment>As we stepped off the ferry we saw a sign on a pinkish-red building that said the island is an UNESCO World Heritage site. It houses the naval academy, a former ship yard, a former prison and a fortress that used to protect the city.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzMxMz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--47fd660256147df44627239001386e0abf4c5e46" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBamtCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--fe994e75e5ffa4bad157f4ba128ed712f5c67d81/DE8FA187-7DBE-4642-AC2A-1699C91CC2AC.jpeg" filename="DE8FA187-7DBE-4642-AC2A-1699C91CC2AC.jpeg" filesize="5681729" width="2975" height="4032" previewable="true" caption="A pinkish-red tower of a building with a clock in it">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzEzLCJwdXIiOiJibG9iX2lkIn19--5f5bf532913e83354050fb51ec1b48aab0962ec1/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/DE8FA187-7DBE-4642-AC2A-1699C91CC2AC.jpeg">
<figcaption class="attachment__caption">
A pinkish-red tower of a building with a clock in it
</figcaption>
</figure></action-text-attachment>The first thing that stood out on this gloomy autumn morning was the bright yellow naval academy. So we went to it first and then returned to the harbor.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzMxND9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--32071b5637f721beeb1d37480514741c08569683" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBam9CIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--d5a2540d975523fa52ef0ed334a815510312396d/0134EBCC-6453-4947-B2ED-5C49323C49C7.jpeg" filename="0134EBCC-6453-4947-B2ED-5C49323C49C7.jpeg" filesize="4525627" width="3024" height="4032" previewable="true" caption="Two bright-yellow buildings of the naval academy">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzE0LCJwdXIiOiJibG9iX2lkIn19--da706c1abc81f473693586574869ca1106c760e4/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/0134EBCC-6453-4947-B2ED-5C49323C49C7.jpeg">
<figcaption class="attachment__caption">
Two bright-yellow buildings of the naval academy
</figcaption>
</figure></action-text-attachment>The harbor and coastal regions of the island are littered with small wooden homes with a few brick and mortar buildings thrown in here and there. Many of the houses are private, some are hostels, museums, cafes and shops.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzMwMz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--bd38bb3ca0811bf0d78dd37c681c4f943c2000ae" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBaThCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--4c4f22031e93ca8ae526298db9fa65fe23582b59/B087AE75-5014-4173-BB57-3B3D841ECF39.jpeg" filename="B087AE75-5014-4173-BB57-3B3D841ECF39.jpeg" filesize="1731128" width="1536" height="2048" previewable="true" caption="A brick and mortar building from the center of the island.">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzAzLCJwdXIiOiJibG9iX2lkIn19--7a12cc72245b6f2ae26c5c1d9c1da8a086f2d9ae/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/B087AE75-5014-4173-BB57-3B3D841ECF39.jpeg">
<figcaption class="attachment__caption">
A brick and mortar building from the center of the island.
</figcaption>
</figure></action-text-attachment>We visited the military history museum which is split across two buildings (keep the ticket when you enter the first building so that you can enter the next).<br><br>One building is full of military uniforms and equipment all the way from the early 1900s to WWII. While the second building is full of modern military equipment.<br><br>In the second building you can wear some of the historic uniforms and sit in the cockpit of a jet fighter simulator. Needles to say we had a fashion show.<br><br>And there is even a submarine!<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI5Nz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--73a9c7faa6c8bb366ac000ac3fa6bce9de805180" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBaWtCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--3d5710fbfb64354511d31f2847cfaf0142ed1142/71E79C51-1865-4568-9624-52BF6F67A3B1.jpeg" filename="71E79C51-1865-4568-9624-52BF6F67A3B1.jpeg" filesize="942990" width="1536" height="2048" previewable="true" caption="A submarine in a dry dock">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6Mjk3LCJwdXIiOiJibG9iX2lkIn19--7dbef4e0313fd6966ab42e6d3b07ddf18e6a358c/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/71E79C51-1865-4568-9624-52BF6F67A3B1.jpeg">
<figcaption class="attachment__caption">
A submarine in a dry dock
</figcaption>
</figure></action-text-attachment>After the museum we went for a walk around the island - which is full of hills, most of which have been hollowed out to make bunkers, caches and armories.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzMwND9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--3124b4549f076d6d11d15fecfb1d4748f0cd9cda" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBakFCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--347e63767db4228daef52e1da8b0b7b7ad1aa2c2/D757BF2A-7099-43B2-9F03-592B95477073.jpeg" filename="D757BF2A-7099-43B2-9F03-592B95477073.jpeg" filesize="2988215" width="2972" height="3963" previewable="true" caption="The inside of a hollowed out hill">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzA0LCJwdXIiOiJibG9iX2lkIn19--49cbd901e36fd40cb9e7a05e8d7f9dee586a0fd2/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/D757BF2A-7099-43B2-9F03-592B95477073.jpeg">
<figcaption class="attachment__caption">
The inside of a hollowed out hill
</figcaption>
</figure></action-text-attachment>To me the landscape was magical. I never imagined that an island could be covered with beech and oak trees on rolling green hills that end with granite rock cliffs in the dark blue sea. My mental image of an island or the seaside is Dalmatia - steep mountainous terrain with sharp white rocks covered in pine and olive forests ending in a teal-azure sea with pebbled beaches.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI5OD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--47524ac0326114151585c69d1fe7987ced2c88e1" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBaW9CIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--56c9f24b5010413c4dc24cc4cfc1bb61306e9923/44D7DE29-E92C-4CD2-A8A9-CF12FA142692.jpeg" filename="44D7DE29-E92C-4CD2-A8A9-CF12FA142692.jpeg" filesize="5062488" width="4032" height="3024" previewable="true" caption="A cliff covered with in green moss, grass, and trees, looking over the sea">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6Mjk4LCJwdXIiOiJibG9iX2lkIn19--ce1353cab0012053eac36c1b6468737ce821920e/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/44D7DE29-E92C-4CD2-A8A9-CF12FA142692.jpeg">
<figcaption class="attachment__caption">
A cliff covered with in green moss, grass, and trees, looking over the sea
</figcaption>
</figure></action-text-attachment>We went past the huge defensive walls, the canons looking over the harbor, through the fortress, all the way to the king’s gate and then back to the harbor building - where we sat down in a brewery to wait for the ferry back.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzMwNT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--cee4b39f492025f5c37d03b0046a88b81f580ff6" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBakVCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--2a7ed3ac96ecff3c0c4bde6e2f0e1653b9d380db/99956906-1E18-48AA-95EB-00D3CE3ED208.jpeg" filename="99956906-1E18-48AA-95EB-00D3CE3ED208.jpeg" filesize="5835033" width="2954" height="4030" previewable="true" caption="A green bench in front of a large defensive wall made of many gray, dark red and dark blue granite rocks topped off with a brick and mortar gable">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzA1LCJwdXIiOiJibG9iX2lkIn19--44221ccd291049a8c281f6606a613664d1bdaefe/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/99956906-1E18-48AA-95EB-00D3CE3ED208.jpeg">
<figcaption class="attachment__caption">
A green bench in front of a large defensive wall made of many gray, dark red and dark blue granite rocks topped off with a brick and mortar gable
</figcaption>
</figure></action-text-attachment>Back in Helsinki we went to the cathedral and then went back to the hotel to pack our bags for the trip back home.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzI5OT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--a026dda34f5ef849250358e3a25de00a404dfedb" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBaXNCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--d40a30e51c5cb40776a571047d27fb525b271e52/266F023D-1751-4CC4-8D93-18349911407D.jpeg" filename="266F023D-1751-4CC4-8D93-18349911407D.jpeg" filesize="850217" width="1536" height="2048" previewable="true" caption="The Helsinki Cathedral - a tall, white palace with green domed roofs decorated with golden stars">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6Mjk5LCJwdXIiOiJibG9iX2lkIn19--5e350f1dc5c3a3a16387953856c589e680a2d9a3/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/266F023D-1751-4CC4-8D93-18349911407D.jpeg">
<figcaption class="attachment__caption">
The Helsinki Cathedral - a tall, white palace with green domed roofs decorated with golden stars
</figcaption>
</figure></action-text-attachment>Helsinki is a clean, beautiful, peaceful city, filled with beautiful buildings, wonderful parks, wide streets and cozy places. Given the opportunity I would move there in a heartbeat.<br><br>Everyone we met was kind, helpful and trusting.<br><br>I was surprised how quiet everybody is. Compared to my hometown everyone seems to be whispering all the time.<br><br>Coming from a country where people are quite distrusting and a bit paranoid, the trusting nature of the people in Helsinki felt odd. There was nobody to check our ferry tickets, no badge or person to check if we were conference attendees, no check to see if we were hotel guests when we went for breakfast. Croats often joke that we are the world’s best nation in circumventing rules and finding loopholes. In contrast, everyone in Helsinki seems to trusts others to be decent human beings.<br><br>I can’t wait to visit the home of the <a href="https://en.wikipedia.org/wiki/Moomins">Moomins</a> again.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzMwNj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--42d00ae6cf0b94d1ec3ea5cf2601a99f3563502f" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBaklCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--cb883b68be5b128f1b2e48cdae8df8269604a3b2/A445D99B-C5D7-43D7-8095-1111C2C4F2DF.jpeg" filename="A445D99B-C5D7-43D7-8095-1111C2C4F2DF.jpeg" filesize="5387350" width="2859" height="3902" previewable="true" caption="A white notebook with a gold embossed Moomin">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzA2LCJwdXIiOiJibG9iX2lkIn19--cfba961672026806543c5f82929bd098e21dca7f/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/A445D99B-C5D7-43D7-8095-1111C2C4F2DF.jpeg">
<figcaption class="attachment__caption">
A white notebook with a gold embossed Moomin
</figcaption>
</figure></action-text-attachment>
</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/222022-10-29T12:00:00Z2023-11-14T16:28:05ZWhat is a professional tool anyway?<div class="trix-content">
<div>One weekend in 2017 I was using my Macbook Pro when all of a sudden the screen turned black and it started spewing white smoke like it had just elected the new Pope.<br><br>Luckily, it didn’t catch fire. Later that day I opened up the laptop to see what happened.<br><br>To my surprise it wasn’t the battery, but it was the hard drive - the only replaceable component in a 2013 Macbook Pro.<br><br>I ordered a replacement, and a week later the laptop was working again. I even upgraded it’s storage from 512GB to 1TB.<br><br>But this problem got me thinking - was the MacBook Pro really a professional tool?<br><br>I always think of a DSLR camera when I think of a professional tool.<br><br>Its parts are mostly replaceable, and the parts that aren’t have redundancies.<br><br>You can switch out lenses, flashes, microphones. You can look through the view finder or the LCD to see what you are shooting. All settings can be changed using the buttons on the device or through the on-screen menu. The lenses, flashes and microphones are interoperable with newer and older camera bodies.<br><br>All of these functions give the tool a long service life and make it reliable and dependable. Therefore a higher price is justified and still a good investment for a professional user.<br><br>Yet there still exist cheaper point-and-shoots which offer similar image quality but without the modularity for less money.<br><br>As a developer, my most used tool is a computer. And for a long time my go to computer was a laptop.<br><br>In the laptop space modularity and repairability have been dead and buried long ago.<br><br>The ability to add more memory or space to your machine, to repair parts that break, has been sacrificed in the name of “progress” and “design”.<br><br>The question is the progress of what? And what design?<br><br>The hardware was never smaller and more reliable than today. Machines that were thin, slick, and modular yesteryear are just thin and slick today. <a href="https://www.ifixit.com/products/macbook-air-original-mid-2009-replacement-battery?variant=39371742150759">The original MacBook Air was a few millimetres thicker than today’s and had a replaceable battery.</a><br><br>The only progress I see here is the progress of companies maximizing profits.<br><br>The simple truth is that it’s cheaper to make hardware this way. And it forces people to buy the more expensive model up front out of fear that they might outgrow the cheaper one.<br><br>It used to be normal to buy a machine with moderate specs and upgrade it later if you wanted more memory or space. Hell, I upgraded my old 2007 iMac with more memory and an SSD back in 2009 - these machines used to be user-upgradeable.<br><br>Today it’s becoming harder and harder to find a computer that allows you to fix or upgrade anything.<br><br>Some companies even go out of their way to make repairs and upgrades difficult. Probably to up-sell you a more expensive model or to sell you a new device when the first one fails due to a banal problem.<br><br>I wouldn’t call these devices professional.<br><br>Sure, they have a lot of features, and they have cutting edge specs. But just like with the point-and-shoot camera, you get what you get and once something breaks the whole machine becomes a paperweight.<br><br>In the last few years a few companies that offer upgradeable and repairable laptops did appear, and I wish them the best of luck. We could use more companies like these, and more devices that aren’t designed to fail, that are repairable and upgradable.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/232022-11-04T12:00:00Z2023-11-14T16:28:05ZFight perfection<div class="trix-content">
<div>Seeking perfection is a fool’s errand.<br><br>Perfection is a fading spark that can only be achieved for oneself - never when working with others - and as soon as it’s gone the search for the new perfect begins.<br><br>By constantly searching for perfection you end up living in the fantastic future - in the “what could be” - instead of living in the here and now - in reality. It’s an escape from facing what you should do now.<br><br>Perfection is the siren song that will steer you into a vortex.<br><br>So fight it, seek reality, seek better, seek good enough.<br><br>In my childhood I was encouraged to seek perfection - to get straight A grades, to write the best assignments, to know every subject matter inside out at any moment - and that thought me that one can only achieve perfection for oneself.<br><br>When I wrote an assignment that I though was perfect - one that I was happy with in every regard - most often I got a B for it. Why? Because it wasn’t perfect for my teacher. Even when it was grammatically perfect, my teacher thought that I could improve flow, or phrasing, or something else. This theme continued in high school, college and at work. Every time I was satisfied - though something was perfect - at least one other person didn’t. There was always something to add, adjust or remove to transform my perfect into their perfect. And sometimes I came to adopt their perfect as my perfect, but sometimes I didn’t.<br><br>This taught me that perfection is in the eye of the beholder. Every person will have a different opinion about what is perfect. Sometimes people can agree that something is perfect, other times they can’t. Even when a collective perfect is achieved it doesn’t last because every new person that comes along will have a new notion about what could be improved.<br><br>Searching for perfection among many people can be an impossible task. So don’t do it. Fight this notion. Don’t seek a consensus about what is perfect, do what is good enough for everybody, do that which is better than what you have now.<br><br>Later on, when I started working, I took part in meetings where we discussed that the perfect solution to a project would be.<br><br>This usually resulted in some sort of a shared notion of a perfect solution that was presented to the client, but no clear way how to get from where we are now to that perfect solution.<br><br>Since we didn’t want to inconvenience or disappoint the client we clang to that notion of perfect, this lead to many inconveniences and disappointments for us and the client.<br><br>Because we had such a rigid goal we never reconsidered if what we were doing made sense. So every now and then we ran into an insurmountable problem or built out nonsensical features that would eventually make sense after something else was built. We were going straight to perfect, regardless of what it took.<br><br>By thinking about the future we forgot about the road that we have to follow to get there. So don’t focus on the future where everything is perfect, focus on reality, focus on the here and now.<br><br>We could have presented the perfect solution as an eventual goal that may or may not come true, and commit to build out a project that brings us closer to it, and then commit to another project, and another, and another.<br><br>That way there isn’t a commitment to a grand goal, so both we and the client are free to change course at any time to avoid problems as they arise. We might not end up at perfect that way, but we would certainly get close to it. And maybe the definition of perfect would change along the way, to one which we could reach.<br><br>The worst kinds of projects at work were ones that never left the drawing board. On these kinds of projects the client was so obsessed about figuring out the perfect solution that they ran out of money before the product was even close to being done.<br><br>They spent weeks in meetings seeking the perfect solution, and always came up with something better than what they had before - they found a new perfect and moved the goal post there. That is the nature of being human - we always find a way to make things better than they are now, we can always dream bigger.<br><br>But you won’t ever make anything better if all you ever do is think about how you can make something better. Don’t obsess about doing something perfectly, just do it better, or good enough.<br><br>Seeking perfection can give you a vision of the future, but remember that it’s only a vision. If you follow it blindly it can lead you down a twisty and thorny road to nowhere. Use it as a compass to make things better than they are, or else fight its siren song tooth and nail for it can ruin everything it touches.<br><br>P.S. This topic has been on my mind for a long time. I tried to write it up a couple of times before but always fell short of the goal I set for my self - I didn’t notice that I was chasing perfection. And that’s something that I have struggled with as an adult. It’s only a few years back that I noticed how corrosive this chase for perfection can be. And it’s only a few weeks back that I learned that Tolkien shared the same notion. He said that Sauron started out as a perfectionist which desired order, and in his effort to make things perfect he became the embodiment of evil.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/242022-11-13T12:00:00Z2023-11-14T16:28:05ZTech debt for non-techies<div class="trix-content">
<div>Tech dept doesn’t occur by accident. You go into it intentionally - it is a loan like any other, but instead of loaning money you loan time.<br><br>When you want to “save time” by implementing a hack (something that goes against the grain of the software’s architecture) you are making an intentional bad decision - you don’t have the time to come up with a better architecture so you defer the problem to future you (or, with any luck, someone else). Essentially, you are borrowing time you don’t have now from the future.<br><br>Like with a regular loan, you have to pay that time back. But without a bank to remind you that your installment is due, it’s easy to forget about this loan. Forgetting the loan sometimes works out, and other times it doesn’t.<br><br>Tech debt won’t cause you problems if everybody on the team understands what it is and what its trade-offs are - how time is traded for functionality, and how to manage this kind of loan. Technical team members are painfully aware of tech debt because they are the ones that have to manage it and pay it back if necessary, but non-technical team members are usually oblivious to it.<br><br>And I don’t blame them, it’s a concept that can be hard to fully grasp. Sure, most people will think they understand it from reading the loan analogy from before. But they won’t be able to see it even when it’s happening in front of their eyes.<br><br>But I found a way to illustrate the problems and processes that cause tech debt through Factorio.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzM0Nj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--451961ea2b52d7589fcf75c4cb7a05ce86664711" content-type="video/webm" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBbG9CIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--b170644cf3140b1e8a185b892b1067536315ae99/factorio_blogpost_intro-2400.webm" filename="factorio_blogpost_intro-2400.webm" filesize="23668808" previewable="true" caption="Montage that demonstrates the basics of Factorio - gathering resources, fighting aliens, building factories and finally launching the rocket">
<figure class="attachment attachment--preview attachment--webm">
<video controls="controls" autoplay="autoplay" loop="loop" muted="muted" loading="lazy" poster="/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzQ2LCJwdXIiOiJibG9iX2lkIn19--64164352f36192bc184ef73be19e321d127ee684/eyJfcmFpbHMiOnsiZGF0YSI6eyJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--21cb38a28c52d6c3cf41ca159781d10834673b37/factorio_blogpost_intro-2400.webm" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzQ2LCJwdXIiOiJibG9iX2lkIn19--64164352f36192bc184ef73be19e321d127ee684/factorio_blogpost_intro-2400.webm"></video>
<figcaption class="attachment__caption">
Montage that demonstrates the basics of Factorio - gathering resources, fighting aliens, building factories and finally launching the rocket
</figcaption>
</figure></action-text-attachment>Factorio is a game in which you play as an engineer stranded on an alien planet. Surrounded by nothing but trees, rocks, minerals, and a hoard of aggressive aliens. Your goal is to make shelter, fight or co-exist with the aliens, and get off the planet by building and launching a rocket.<br><br>The problem is obvious - how do you make a rocket and rocket fuel from sticks and stones? The answer is simple - step by step.<br><br>In Factorio, more complex devices can be made out of simpler ones or out of raw materials like rock, iron, copper, coal, oil and uranium.<br><br>In the beginning you can only gather and build things. So you do exactly that. You gather rocks to make a furnace, then you gather Iron ore to smelt it into plates.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzM0Nz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--a8f557d2a1c93e40ff62f1e8bf142797849062a1" content-type="video/webm" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBbHNCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--73235682a8989e7c67ef5c6657da891ee9b93c8d/factorio_blogpost_crafting.webm" filename="factorio_blogpost_crafting.webm" filesize="15056051" previewable="true" caption="Gathering of 5 stone to craft a furnace, and then smelting an iron plate in it by feeding it iron ore and coal">
<figure class="attachment attachment--preview attachment--webm">
<video controls="controls" autoplay="autoplay" loop="loop" muted="muted" loading="lazy" poster="/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzQ3LCJwdXIiOiJibG9iX2lkIn19--2bd5f8a1045737c344a3b4a4736bddc30f597802/eyJfcmFpbHMiOnsiZGF0YSI6eyJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--21cb38a28c52d6c3cf41ca159781d10834673b37/factorio_blogpost_crafting.webm" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzQ3LCJwdXIiOiJibG9iX2lkIn19--2bd5f8a1045737c344a3b4a4736bddc30f597802/factorio_blogpost_crafting.webm"></video>
<figcaption class="attachment__caption">
Gathering of 5 stone to craft a furnace, and then smelting an iron plate in it by feeding it iron ore and coal
</figcaption>
</figure></action-text-attachment>Iron plates enable you to build drills and conveyor belts. Now you don’t have to dig and walk so much anymore.<br><br>Soon you can build robotic arms that can pick up things from the conveyor belts and feed them to other machines. Now you can automate whole processes like smelting iron and copper.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzM0OD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--9843ae9ce3bc997b4d317487935fef871693e525" content-type="video/webm" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBbHdCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--cb77a367b5e5e482755a6ed9ee7428f1a873ff1d/factorio_blogpost_automation.webm" filename="factorio_blogpost_automation.webm" filesize="3974033" previewable="true" caption="A basic automated coal and iron mining setup">
<figure class="attachment attachment--preview attachment--webm">
<video controls="controls" autoplay="autoplay" loop="loop" muted="muted" loading="lazy" poster="/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzQ4LCJwdXIiOiJibG9iX2lkIn19--fd4c44c325fa1af218e1d27ce7e0b01c088f15c9/eyJfcmFpbHMiOnsiZGF0YSI6eyJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--21cb38a28c52d6c3cf41ca159781d10834673b37/factorio_blogpost_automation.webm" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzQ4LCJwdXIiOiJibG9iX2lkIn19--fd4c44c325fa1af218e1d27ce7e0b01c088f15c9/factorio_blogpost_automation.webm"></video>
<figcaption class="attachment__caption">
A basic automated coal and iron mining setup
</figcaption>
</figure></action-text-attachment>And that is the basic loop of the game - gather, build, automate. But it gets complicated. Over time you will build a rocket building factory. There will be machines making iron, turning it into electronics, building other machines that are parts of other machines that are part of your rocket, and all of them will be connected with conveyor belts and robotic arms.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzM0OT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--399eebd031dbf2c7c1902555cab7267717365400" content-type="video/webm" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBbDBCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--57882c0c3d4f4908cee5d00c7398999a4bf088e9/untitled.webm" filename="untitled.webm" filesize="6250547" previewable="true" caption="Running through one part of a bigger factory">
<figure class="attachment attachment--preview attachment--webm">
<video controls="controls" autoplay="autoplay" loop="loop" muted="muted" loading="lazy" poster="/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzQ5LCJwdXIiOiJibG9iX2lkIn19--8bac7e9fab7e3af2477efc87869a6e4aeb271106/eyJfcmFpbHMiOnsiZGF0YSI6eyJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--21cb38a28c52d6c3cf41ca159781d10834673b37/untitled.webm" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzQ5LCJwdXIiOiJibG9iX2lkIn19--8bac7e9fab7e3af2477efc87869a6e4aeb271106/untitled.webm"></video>
<figcaption class="attachment__caption">
Running through one part of a bigger factory
</figcaption>
</figure></action-text-attachment>Building such a factory is no easy task if you can’t manage tech debt. And you will run into tech debt pretty quickly.<br><br>Your first encounter with tech debt will be through conveyor belts. In the beginning you will place parts of your factory wherever they are easiest to fit. Probably close to the resources required to build the parts<br><br>You will build conveyors close to where you smelt iron. That’s only logical since conveyor belts are expensive and you just spent a lot of resources on an assembler to make the gears and conveyors.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzM1MD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--c3511b31a909bb020e78aa5f49b0b80abef439db" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBbDRCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--c41eb8aedbbef55646255474f6be1a9ecf00c28c/untitled.png" filename="untitled.png" filesize="1112404" width="893" height="489" previewable="true" caption="A furnace and two assemblers connected with conveyors, building new conveyors">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzUwLCJwdXIiOiJibG9iX2lkIn19--c757a52a5b793e08fcad08ab3191e91f7324e40f/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/untitled.png">
<figcaption class="attachment__caption">
A furnace and two assemblers connected with conveyors, building new conveyors
</figcaption>
</figure></action-text-attachment>This kind of thinking and behavior is known as “greedy behavior” in software. Greedy behavior is usually short-sighted, highly rewarding short-term, but it can be much worse than other approaches long-term.<br><br>And that is exactly what will happen here. If you now want to create an assembler that builds robotic arms, you will run into an issue - the gears that you need to make them are in a box right between the two assemblers and are fully surrounded by conveyor belts. There is no space left to add a robotic arm to extract them from the box and put them on another conveyor.<br><br>Now you have two choices - tear everything down and rebuild it better, or squeeze in a few conveyors.<br><br>Rebuilding is the better choice now that you know that you want to automate the assembly of robotic arms. But some people will choose to squeeze in conveyors because of the sunk cost fallacy - they can’t let go of the time and effort they have already invested - or because they don’t have the time to rebuild the factory - in which case they are entering tech debt by intentionally making a bad decision to save some time now.<br><br>Over time, squeezing in more and more conveyor belts will result in conveyor belt spaghetti. The factory will get so densely covered with conveyor belts that it will become difficult to keep track of which resources are going where. And even worse, it will become hard to add new conveyors and grow the factory - you will have to duplicate parts of your factory to be able to cart the resources to another part of the factory.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzM1MT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--eb1b528004d83d0d2c362124f6b69b21ff3ddeea" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBbDhCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--91e9d038cd01b9a7d8b9d3449a82b7b83761b1e7/untitled_1.png" filename="untitled_1.png" filesize="1173119" width="699" height="644" previewable="true" caption="What it looks like when you squeeze in conveyors and duplicate parts of the factory. There are two assemblers making gears, and two furnaces making iron plates. And everything is covered in conveyor belts.">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzUxLCJwdXIiOiJibG9iX2lkIn19--26aa8e783de6e1febd2c44b853b6eec7af472902/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/untitled_1.png">
<figcaption class="attachment__caption">
What it looks like when you squeeze in conveyors and duplicate parts of the factory. There are two assemblers making gears, and two furnaces making iron plates. And everything is covered in conveyor belts.
</figcaption>
</figure></action-text-attachment>Short-term, squeezing in conveyors will solve your problem. Long-term, it will cause you bigger problems. But there is a third solution - squeeze in the conveyor belts today, and reorganize the factory tomorrow. That is, enter tech debt today and pay it tomorrow.<br><br>To long term players this is obvious. They would build the factory with expansion in mind from the start. But to new players this isn’t obvious at all.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzM1Mj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--c0037115efbcbe11fad7f6c4dfa7ef24fd5535a2" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBbUFCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--0ed6fec3009c920a9e775cdfc86c62c3fb04e72c/untitled_2.png" filename="untitled_2.png" filesize="2465820" width="985" height="1041" previewable="true" caption="The same factory as before, but built with expansion in mind. Production of raw resources and parts is centralized and then the resources are carted around the factory on conveyor belts. There is plenty room to add more machines or expand existing ones.">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MzUyLCJwdXIiOiJibG9iX2lkIn19--8422039c8613ab332391147ab82391a9386d3ffe/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/untitled_2.png">
<figcaption class="attachment__caption">
The same factory as before, but built with expansion in mind. Production of raw resources and parts is centralized and then the resources are carted around the factory on conveyor belts. There is plenty room to add more machines or expand existing ones.
</figcaption>
</figure></action-text-attachment>That’s why developers complain and moan about hacks and tech debt, while managers usually struggle to understand the problem. Developers know that, eventually, the code they write today will become the foundation of a new feature tomorrow and they want to prepare for it. While managers are concerned with delivering something today.<br><br>That is also why tech debt rarely gets addressed in some organizations. When a manager - the person deciding what the developer or designer will build - has to choose between working on a feature or maintenance they will always choose the feature. It is a feature today, they understand that. They don’t understand that maintenance will be a feature tomorrow.<br><br>In their eyes one is fact, the other is a hypothetical. While in the builder’s eyes both are fact.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/252022-11-20T21:00:00Z2023-11-14T16:28:04ZEmail<div class="trix-content">
<div>For me, instant messaging is overwhelming. Keeping up with <a href="https://slack.com/">Slack</a> or <a href="https://discord.com/">Discord</a> messages often feels like drinking from a fire hose of information. And some features of these apps bring out the worst in people. I wish these apps were more like email, because email got a lot of things right.<br><br>My biggest complaint with instant messaging is how distracting it is. Notifications pop up to grab your attention for every message, most of which aren’t intended for me.<br><br>This is a necessary byproduct of organizing communication into channels. On the surface this is a great idea to focus the discussion on a topic, but in practice it heavily hinges on the netiquette of the participants.<br><br>Like every group chat ever, a channel will spawn off-topic conversations that will ping everyone in the channel.<br><br>Eventually, somebody will start discussing sandwiches in a channel for handling outages. Or someone will start discussing a baseball game in the general channel. Both of these are real examples.<br><br>I feel like the only real option in such cases is to mute or leave the channel, but that’s not always an option. I can’t leave the outage channel, and I can’t mute it. So what now?<br><br>Topics (or sub-chats) are a great way to combat this. But again, they hinge on the netiquette of the participants. If nobody uses threads they are useless. If too many are created, they become hard to follow.<br><br>Then there is the instant part of instant messaging that entices some people to write down their stream of consciousness for others to interpret, instead of taking the time to organize their thoughts into a sentence.<br><br>You<br><br>Know<br><br>The<br><br>Type<br><br>Of<br><br>People<br><br>Who<br><br>Send<br><br>Messages<br><br>Like<br><br>This<br><br>Just<br><br>So<br><br>They<br><br>Can<br><br>Write<br><br>Something<br><br>Instantly<br><br>Then there is the other kind of instant in instant messaging - the instant response - to which some people feel entitlement.<br><br>I’ve had people ping me with “I saw you replied in the channel 15 min ago, but not to me”. Or people ping me with a question and 3 min later sending another one saying that my lack of response will be considered a no. And I had people ping me with “Are you there?” every 5 min.<br><br>I don’t know why instant messaging brings out the worst in people. But I know that a lot of people choose to always appear offline to avoid these problems.<br><br>Which brings me to email. It never had an online status to begin with, every conversation is a thread, there is no expectation of an instant response, and people usually write full sentence responses.<br><br>Some of these “features” were technological or practical limitations of the time, and some evolved out of the way people used email.<br><br>But email evolved into a great, distraction free, way to communicate by using these friction points to its advantage.<br><br>It has its problems, like not having a mechanism to leave a thread, or having some way to track branching threads. But apps like <a href="https://www.hey.com/">Hey</a> have shown that it’s possible to resolve these problems. And showed that the federated design of email allows anyone to innovate and improve <a href="https://en.wikipedia.org/wiki/History_of_email">this 50-year-old protocol</a>.<br><br>There are so many wonderful solutions on top of email out there. Apps like <a href="https://mailbrew.com/">Mailbrew</a> - which combines RSS, Reddit, Twitter even weather forecasts into a single daily email.<br><br>Anyone can send you an email. Which is a blessing and a curse thanks to spammers. But I’ve had wonderful conversations over the years that have cold-mailed me after stumbling upon one of my projects or articles. And, again, apps like Hey add ways to screen incoming emails before they end up in your inbox.<br><br>I recently tried <a href="https://twistapp.com/">Twist</a> and found that by mimicking email it became a better instant messenger - if you can call it that. It only shows that there is still a lot of room left to improve instant messengers. At least the work-focused ones like Slack.<br><br>Email is still the best communication method for me.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/262022-11-27T21:00:00Z2023-11-14T16:28:03ZCutting through the noise<div class="trix-content">
<div>I use writing as a tool to organize my thoughts, but lately I have started to question if that’s the right order to do things.<br><br>Writing is thinking. Text is just someone’s thoughts on paper. It’s obvious that thinking has to come first, but that’s not always the case.<br><br>When I sit down to write something I often just have a notion about what I want to write, and only through the process of writing do I form a full thought. This process can take days to produce an article. But as I started writing more consistently, the time required to write an article went down to a few hours.<br><br>This week I heard that DHH writes articles in 15min and I was taken aback. I enjoy his articles, he has a way of explaining things in a clear and forward way - something I am trying to improve, and which I envy.<br><br>The only way I see that this is possible is by sitting down with a fully formed thought before writing a single word. He’s able to cut through the noise and distill a thought without the aid of writing.<br><br>Have I been putting the carriage before the horse all this time?<br><br>Well, yes and no. To write that way I have to trust my self more - something I see I have been avoiding by using writing as a crutch to verify my thoughts, double and triple check them. I’m just avoiding criticism, but that means that my written thoughts are a watered down vanilla version of the real thing. I have to become bolder in expressing my opinion. Then again, there is nothing wrong with using writing as a tool from time to time.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/272022-12-04T20:00:00Z2023-11-14T16:28:04ZElden Ring<div class="trix-content">
<div>This week I finished Elden Ring, and the moment I put the controller down I felt a void as if a good friend moved to another country and I wouldn’t see them for a long time. This was the best game I have played in years.<br><br>But why? On the surface it seems just like any other open-world ARPG, yet Elden Ring is bigger than the sum of its parts. This game is a master class in the <a href="https://en.wikipedia.org/wiki/Omakase"><em>omakase</em></a> design approach and that’s what makes it so amazing.<br><br>The game makes two key decisions for you - the difficulty level and the story exposition.<br><br>There is no way to change the difficulty in this game. You find a boss difficult? Tough luck, you will have to get better. Everybody has to face the same challenging fights, enemies, quest and puzzles. This was the first <a href="https://en.wikipedia.org/wiki/Soulslike">Souls game by From Software</a> I have played and this design decision felt hostile to me, sometimes I just want to relax with a game and enjoy the story.<br><br>But after I beat the first boss I felt a sense of accomplishment and then it clicked - the difficulty is what makes each win feel so rewarding. The combat is punishing, but it’s also fair. It’s like a dance to which you have to figure out the steps or get trampled (or get so overpowered that you can steamroll through it). There are a few bullshit attacks that are hard to dodge, but everything else is clearly telegraphed - if you make a mistake you know it was on you. The dance-like nature of the combat is also why getting attacked by more than one enemy at a time is infinitely more difficult - you have to keep track of two dance partners.<br><br>But the thing that makes the combat of Elden Ring so great is its open world. When I got stuck, bored or annoyed I could always just go somewhere else and have fun. There is no better example of this than the first mini-boss. I exited the tutorial level to find myself in a lush grassy field with a giant golden knight in the distance. As the naive spring chicken that I was, I walked up to the shiny knight only to see a large “YOU DIED” message a second later. Then I tried to fight the knight, and no matter what I did the knight would demolish me before I got him to half-health. At that moment I understood that the game was trying to teach me that if I’m not having fun that I should look elsewhere. Nothing was forcing me to fight that kings, I could just go around him and come back later.<br><br>There are a few traps in the game that take away your ability to travel and only then does it become obvious how crucial the open-world is to the combat experience of this game. I got eaten by the Abductor Virgin (Iron Maiden) in Raya Lucaria Academy and got teleported to a lava lake in Volcano Manor, under-geared and under-leveled, only to spend and hour in agony before I gave up and looked up how to cheese an escape.<br><br>The game does use its combat difficulty as a level-gate to steer you round the world. But because of this the world always feels challenging and keeps you on your feet. I was decimating archers and knight, fighting my way to a demigod boss only to emerge in another region of the world after the fight where a stray dog and a guy with a torch could mess my day up.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzIyOD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--919e17d86a58ad879209ae3e688f8dae3ac0fccf" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZVE9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--8d9446cc9bd29deaf11e4133430aacc907a3c0d5/meme.png" filename="meme.png" filesize="720668" width="1155" height="963" previewable="true" caption="Strong Doge meme. The strong one tanks hits by demigods, the weaker one is bullied by two dogs and a guy with a torch">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjI4LCJwdXIiOiJibG9iX2lkIn19--5aa96f25b1649ec0e274e3b765f6801acbe5061d/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/meme.png">
<figcaption class="attachment__caption">
Strong Doge meme. The strong one tanks hits by demigods, the weaker one is bullied by two dogs and a guy with a torch
</figcaption>
</figure></action-text-attachment>Exploration is not only important to get better gear and level up, but also for the story. The game doesn’t hold your hand when it comes to the main story, it expects that you will go out of your way to figure it out. This also seemed hostile at first.<br><br>There are a few expositions during key quest moments and demigod fights, but that’s not enough to figure out what is going on. The story through the cutscenes alone is a jumbled mess - you want to become Elden Lord, because that’s what one does, and there are demigods you have to kill to get shards of the Elden Ring, which shattered in the past, and only with those shards will you be able to become Elden Lord. What is this? Pokemon fan fiction? How many gym trainers do I have to beat to become a Pokemon master?<br><br>The details of the story are scattered in side-quests and item descriptions which can only be found by exploring. I was half-way through the game when I discovered that <a href="https://old.reddit.com/r/Eldenring/comments/vgvy7y/best_item_description/">item descriptions contain the lore of the world</a>. The other thing I discovered half-way through the game is that finishing side quests unlocks alternate endings. This made exploration more enjoyable and rewarding compared to other open-world ARPGs, I felt like Indiana Jones recovering parts of the Ark of the Covenant.<br><br>These two decisions - a single difficulty level for all and a vague story - complement each other in a way that makes the world feel like a sandbox where I can go explore and adventure. It doesn’t get old and it’s not a gimmick like in other ARPGs. It’s so good that this is the first game where, after beating it, I immediately started a second play-through to experience what I have missed.<br><br>This is only possible because of omakase. The team at From Software made a tailored experience, an experience that takes existing elements of ARPGs, sacrifices some quality-of-life features, but arranges what’s left in such a way that they become better than in all other ARPGs.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/282023-01-21T16:00:00Z2023-11-14T16:28:03ZCOVID hangover<div class="trix-content">
<div>With a lot of tech companies laying people off in the last month I got to see how insane the firing practices in the US are and how greed rules supreme in the tech industry. But this is only a harbinger of what’s to come.<br><br>Through friends and from what I’ve seen first hand, the standard procedure to fire someone in the US is to invite them to a meeting, tell them that they have been let go, and immediately suspend all their company accounts. <a href="https://blog.google/inside-google/message-ceo/january-update/">Or email them to tell them they have been let go (effective immediately)</a>, or worse yet <a href="https://www.npr.org/2022/11/17/1137265843/elon-musk-fires-employee-by-tweet">tell them they have been let go via a tweet</a>.<br><br>There is zero humanity in laying people off this way. These people dedicated months or years of their lives building these companies and then one morning they get kicked out the door <a href="https://www.forbes.com/advisor/personal-finance/what-is-the-warn-act/">with a 60 day severance package - which is the legal minimum</a> - and aren’t even given the decency to say good bye to their former coworkers.<br><br>These layoffs weren’t triggered by financial losses but by speculations of a looming recession. Companies are afraid that money won’t be cheap and profits will be low in the coming years and they are cutting their losses while they still can in an effort to keep investors and shareholders happy. In other words, they are cutting headcount to cut expenses in an effort to prop up profits. Which is extremely greedy after a year of record profits and shows no regard for the humans that made this growth possible in the first place.<br><br>I believe that these layoffs are not an indication of the tech bubble bursting, but are just the hangover after the COVID lock-downs when tech seemed like the only unaffected industry. This caused every company in the industry to over-hire and expand in the pursue of even higher profits. But expecting 20-30% returns year-after-year and infinite growth is unrealistic - that can only be a bubble. After a couple of good years a bad one is bound to happen. And if there is going to be a recession then profits should be low - don’t turn a bad day for you into a tragedy for others. Sit it out and don’t be greedy.<br><br>If the industry weathers this recession relatively unscathed people will see tech as an even better investment vehicle which will in turn lead to more investment, more hype, and the bubble will inflate even more. This seems to be the outcome that most big companies are banking on since even after the layoffs most of them still have more employees than they had two years ago, indicating that they want to use the extra manpower to expand.<br><br>I don’t know when or how this bubble will burst, but it will be one hell of a pop when it does.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/292023-01-24T14:00:00Z2023-11-14T16:28:04ZVanilla Rails view components with partials<div class="trix-content">
<div>Many projects I work on have some kind of view component that is repeated multiple times in the same view, or is present in multiple different views. These view components can be anything that has a specific styling, JavaScript specific attributes (like Stimulus controllers), rendering logic, and HTML element structure that always has to be the same for the component to render properly - like cards, containers, content boxes, modals, lists, tables, and so on.<br><br>There are three ways that I have seen people work with view components in Rails - by copy-pasting them around, by using one of the <a href="https://github.com/ViewComponent/view_component">view</a> <a href="https://github.com/trailblazer/cells">component</a> gems, and partials.<br><br>Copy-pasting, even when using <a href="https://getbem.com/introduction/">CSS conventions like BEM</a>, has the downside of being laborious to update should the component ever change. And I find the gems to be unnecessary since vanilla Rails already has all the features to render and manage view components through <a href="https://guides.rubyonrails.org/layouts_and_rendering.html#using-partials">partials</a>.<br><br>This is an example of three different view components combined together to render a table-like list and a map within a content box with a title using just partials.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzIyOT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--45f42c4e84bd7c0e0e37049c1429999d9be820d6" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZVU9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--d264ba289f6bb195ff1548beb6b68914a181f556/Untitled.png" filename="Untitled.png" filesize="601174" width="2512" height="752" previewable="true" caption="Code and browser view of multiple vanilla Rails view components composed together to show location details and a map side-by-side within a titled content box.">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6MjI5LCJwdXIiOiJibG9iX2lkIn19--f6b735658e573b7fceaf7a1a1201cc782f7f7397/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/Untitled.png">
<figcaption class="attachment__caption">
Code and browser view of multiple vanilla Rails view components composed together to show location details and a map side-by-side within a titled content box.
</figcaption>
</figure></action-text-attachment>Code and browser view of multiple vanilla Rails view components composed together to show location details and a map side-by-side within a titled content box.<br><br>The <strong>component</strong> method might seem like there is a lot of magic under the hood, but there really isn’t. I added this method through <a href="https://api.rubyonrails.org/classes/ActionController/Helpers.html">a view helper</a> and it does just two things - it calls <strong>render</strong> with <strong>"components/#{component_name}"</strong> so that I don’t have to write the same incantation all over the place, and I’ll explain the the second purpose later.<br><br>For now, this is what the helper looks like</div><pre>#!/usr/bin/ruby
module ComponentHelper
def component(path, options = {}, &block)
full_path = Pathname.new("components") / path
render(full_path.to_s, options)
end
end</pre><div>So <strong>component("map", longitude: 15.9765701, latitude: 45.8130054)</strong> is the same thing as <strong>render("components/map", longitude: 15.9765701, latitude: 45.8130054)</strong>.<br><br>Partials allow you to render a snippet of a view wherever you like. For example, instead of having to repeat all the HTML elements and Ruby code that renders two tables:</div><pre><!-- app/views/matches/show.html.erb -->
<h2>Winners</h2>
<table>
<thead>
<tr>
<th>Rank</th>
<th>Player</th>
<th>Score</th>
</tr>
</thead>
<tbody>
<%= @match.winners.each do |player| %>
<tr>
<td><%= player.rank %></td>
<td><%= player.name %></td>
<td><%= player.score %></td>
</tr>
<% end %>
</tbody>
</table>
<h2>Losers</h2>
<table>
<thead>
<tr>
<th>Rank</th>
<th>Player</th>
<th>Score</th>
</tr>
</thead>
<tbody>
<%= @match.losers.each do |player| %>
<tr>
<td><%= player.rank %></td>
<td><%= player.name %></td>
<td><%= player.score %></td>
</tr>
<% end %>
</tbody>
</table></pre><div>With partials you can just render the partial once for each table and pass it a local with the content it should render:</div><pre><!-- app/views/matches/show.html.erb -->
<h2>Winners</h2>
<!-- we are passing `@match.winners` as the local `players` to the partial -->
<%= render "players_table", players: @match.winners %>
<h2>Losers</h2>
<%= render "players_table", players: @match.losers %>
<!-- app/views/matches/_players_table.html.erb -->
<table>
<thead>
<tr>
<th>Rank</th>
<th>Player</th>
<th>Score</th>
</tr>
</thead>
<tbody>
<!-- the `players` local shows up in this partial as the `players` variable -->
<%= render collection: players, partial: "matches/player" %>
<!-- the above ^ is a shorthand for:
```
<% players.each do |player| %>
<%= render "matches/player", player: player %>
<% end %>
```
-->
</tbody>
</table>
<!-- app/views/matches/_player.html.erb -->
<tr>
<td><%= player.rank %></td>
<td><%= player.name %></td>
<td><%= player.score %></td>
</tr></pre><div>Using partials like this works for components that have predetermined content (like tables, maps and lists), but it’s a pain for components that can have anything inside them (like containers, content boxes and modals).<br><br>You could extract the content of each component into its own partial and pass the the name of the partial you want to render.</div><pre><!-- app/views/teams/show.html.erb -->
<%= render "content_box", content_partial: "weekly_metrics", content_options: { team: @team } %>
<%= render "content_box", content_partial: "monthly_metrics", content_options: { team: @team } %>
<!-- app/views/teams/_content_box.html.erb -->
<section>
<%= render content_partial, content_options %>
</section>
<!-- app/views/teams/_weekly_metrics.html.erb -->
<p>Matches won: <%= team.matches.where(created_at: (1.week.ago...)).won.count %></p>
<p>Matches lost: <%= team.matches.where(created_at: (1.week.ago...)).lost.count %></p>
<p>Upcoming matches: <%= team.matches.where(created_at: (Time.current...)).count %></p>
<!-- app/views/teams/_monthly_metrics.html.erb -->
<p>Matches won: <%= team.matches.where(created_at: (1.month.ago...)).won.count %></p>
<p>Matches lost: <%= team.matches.where(created_at: (1.month.ago...)).lost.count %></p></pre><div>This can be avoided by passing a block to the partial and yielding to it.</div><pre><!-- app/views/teams/show.html.erb -->
<%= render "content_box" do %>
<p>Matches won: <%= @team.matches.where(created_at: (1.week.ago...)).won.count %></p>
<p>Matches lost: <%= @team.matches.where(created_at: (1.week.ago...)).lost.count %></p>
<p>Upcoming matches: <%= @team.matches.where(created_at: (Time.current...)).count %></p>
<% end %>
<%= render "content_box" do %>
<p>Matches won: <%= @team.matches.where(created_at: (1.month.ago...)).won.count %></p>
<p>Matches lost: <%= @team.matches.where(created_at: (1.month.ago...)).lost.count %></p>
<% end %>
<!-- app/views/teams/_content_box.html.erb -->
<section>
<%= yield %>
<!-- `yield` will be replaced by whatever is passed in the block -->
</section></pre><div>That’s how <strong>component("content_box", title: "Location") do</strong> works.</div><pre><!-- app/views/properties/show.html.erb -->
<% component("content_box", title: "Location") do %>
<h1>Hello World!</h1>
<% end %>
<!-- app/views/components/_content_box.html.erb -->
<section class="rounded shadow bg-white">
<!-- `local_assigns` is a hash that contains all the locals passed to a partial -->
<!-- we can check if a local is present and access it's value through it -->
<% if local_assigns.key?(:title) %>
<h4 class="text-gray-900 text-lg">
<%= local_assigns[:title] %>
<!-- I could have used `title` instead of `local_assigns[:title]` -->
</h4>
<% end %>
<div>
<%= yield %>
</div>
</section></pre><div>This brings me back to the second purpose of the <strong>component</strong> method - fixing localization.<br><br>If I would use full localization keys for everything then there wouldn’t be a problem with just using <strong>render</strong>
</div><pre><!-- app/views/properties/show.html.erb -->
<% render("components/content_box", title: t("properties.show.location")) do %>
<h1><%= t("properties.show.hello_world") %></h1>
<% end %>
<!-- config/locales/en.yml -->
en:
properties:
show:
location: "Location"
hello_world: "Hello World!"
<!-- rendered HTML -->
<section class="rounded shadow bg-white">
<h4 class="text-gray-900 text-lg">Location</h4>
<div>
<h1>Hello World!</h1>
</div>
</section></pre><div>But I’m lazy and like to use relative localization keys everywhere, which causes a problem.</div><pre><!-- app/views/properties/show.html.erb -->
<% render("components/content_box", title: t(".location")) do %>
<h1><%= t(".hello_world") %></h1>
<% end %>
<!-- config/locales/en.yml -->
en:
properties:
show:
location: "Location"
hello_world: "Hello World!"
<!-- rendered HTML -->
<section class="rounded shadow bg-white">
<h4 class="text-gray-900 text-lg">Location</h4>
<div>
<h1>
<!-- This is what Rails render when a translation is missing -->
<!-- Notice that it looked for the translation in `en.components.content_box.hello_world` -->
<!-- instead of in `en.properties.show.hello_world` -->
<span class="translation_missing" title="translation missing: en.components.content_box.hello_world">
Hello World
</span>
</h1>
</div>
</section></pre><div>With relative localization keys, Rails prefixed the key with the path of the component instead of the parent.<br><br>When a view yields the passed block is rendered in the context of that view, not in the context of the view where the block was created. Since the partial yields the relative localization key prefix is <strong>components.content_box</strong>.<br><br>To solve this I can either use full localization keys within components, which is annoying; or I can add the translation to the component’s localization, which means that I would have to think about possible key collisions when using relative localization keys within components.<br><br>Thankfully there is a way to capture a part of a view within the context of the current view and store it as a variable. With the <strong>capture</strong> <a href="https://api.rubyonrails.org/classes/ActionView/Helpers/CaptureHelper.html#method-i-capture">method</a> I can yield the block passed to the component in the context of the current view, store the result to a variable, and pass that variable in a block to <strong>render</strong>.<br><br>Here is the full <strong>ComponentHelper</strong> code with the relative locale keys fix.</div><pre>#!/usr/bin/ruby
module ComponentHelper
def component(path, locals = {}, &block)
full_path = Pathname.new("components") / path
if block
# Render the passed block within the current context and
# store it in the `content` variable
content = capture do
block.call
end
# Call render but pass it a new block that yields just
# the contents of the `content` variable
render(full_path.to_s, locals) { content }
else
render(full_path.to_s, locals)
end
end
end</pre><div>And that’s all there is to it - <strong>render</strong>, <strong>local_assigns</strong>, <strong>yield</strong> and <strong>capture</strong>.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/302023-04-05T08:00:00Z2023-11-14T16:28:04ZFrom Markdown to ActionText<div class="trix-content">
<div>I started this iteration of my blog because I grew dissatisfied with Medium. Like anyone migrating from any one platform to another I requested an export of my blog posts from them and got back a ZIP file containing HTML files. <br><br>The first thing that stood out was a lack of any media files - there were no images or gifs in the export. Opening the HTML files revealed that they contained the exact same HTML as was served by Medium.com. The source of the images and gifs pointed to Medium's servers. And the HTML itself had a lot of formatting and hidden elements specific to Medium and SEO.<br><br>I spent a whole day downloading images and cleaning up the files into something that could be imported somewhere else. I decided to convert my blog posts into Markdown - the pendulum had swung too far in the opposite direction. That way I could easily import them anywhere later. I fantasied about writing blog posts directly in my text editor with no distraction like headings, italics, bold, fonts - just my thoughts and text. And it was fun to develop a markdown rendering pipeline for the blog!<br><br>After a few months I gave up on the fantasy of writing in a text editor. While markdown is great, having a few simple options for editing text and being able to see the result right away is much more pleasing than knowing that # will eventually turn into a h1 element.<br><br>I started writing and editing articles in Notion, exporting them to markdown, adjusting them for my blog, and then uploading them here. This was unnecessarily complicated, but it gave me the ability to write wherever I went (I like to go for a walk, think, then sit down on a bench when I figure something out and write it down).<br><br>Now I had a different problem - editing an article after it was published. With this process I had to first update the version in Notion, export it, adjust it again, and upload it again. This was annoying when I started publishing more often and therefore catching less typos in editing.<br><br>This convoluted process eventual led me to stop publishing articles here. Then a few weeks ago I had the urge to write and publish something. But the thought of having to go through this publishing process made me put it off. This week I decided to correct this pendulum swing and migrate to ActionText to make things as simple as possible.<br><br>ActionText is a framework within Rails that enabled rich text editing and presentation. Adding it took no time at all, but migrating data to it took a whole day. There are many opinionated decisions in the framework with which I butted heads in the process.<br><br><strong>The first "problem"</strong> I ran into were attachments. ActionText uses Rails' own ActiveStorage framework for uploading, processing and serving attachments, while the blog used Shrine (because I like Shrine). But the way I used Shrine was very reminiscent of how ActiveStorage works, so instead of reinventing the wheel further I decided to migrate over.<br><br>This wasn't an issue and was as simple as iterating through all <strong>Article::Attachment</strong> records, taking their attachment, calling <strong>to_io</strong> on it and then passing that to <strong>ActiveStorage::Blob.create_and_upload!</strong>
</div><pre>#!/usr/bin/ruby
blob = ActiveStorage::Blob.create_and_upload!(
io: attachment.attachment.to_io,
filename: attachment.attachment_data&.dig("metadata", "filename"),
content_type: attachment.attachment_data&.dig("metadata", "mime_type"),
identify: false
)</pre><div>
<strong>The second "problem"</strong> I ran into was paragraph support. ActionText doesn't use paragraphs (p elements) for text, instead it puts all text into a single div element and breaks it into "paragraphs" using line breaks (br elements). This was a problem because I wanted to use <a href="https://tailwindcss.com/docs/typography-plugin">Tailwind's Typography plugin</a> and its <strong>prose</strong> class - which expects paragraphs as p elements. After fighting with the framework to make it work with p elements I ran across <a href="https://github.com/basecamp/trix/issues/202#issuecomment-461166895">a GitHub issue where the reasoning behind using a div was explained</a> which made me realized that it was far easier to write a bit of custom CSS than to make paragraphs work with ActionText in all browsers.<br><br>Without the explanation from the GH issue the decision to use divs seemed completely arbitrary and I wanted it to work with Tailwind. It would have been nice to have this explanation in the Readme.<br><br><strong>The third problem</strong> was converting my previously rendered markdown into ActionText's HTML format (using a div and br elements) and embedding images into it. Though this turned out to be easier than I initially thought. A simple script and some Nokogiri magic and everything work like a charm.</div><pre>#!/usr/bin/ruby
Nokogiri.HTML(article.rendered_markdown).tap do |doc|
# Remove the article's title
doc.css("h1").first&.remove
# Convert all paragraphs into a DIV with BRs
content = doc.css("p").first
content.name = "div"
node = content
while (node = node.next_sibling)
content.inner_html += "<br>#{node.inner_html}"
node.remove
end
# Convert images to attachments
doc.css("img").each do |node|
node.name = "action-text-attachment"
end
article.update!(content: doc.to_s)
end</pre><div>
<strong>The fourth problem</strong> was the <a href="https://github.com/basecamp/trix/issues/539">lack of support for tables</a>. But I decided to use Judo here and just <a href="https://github.com/basecamp/trix#inserting-a-content-attachment">embed the tables as content attachments</a>, then take screenshots of them and embed them as images. This isn't ideal, but it works and it's what I used to do on Medium. I intend to add table support at some point, but currently only two articles use tables and spending a few days on this wasn't worth it.<br><br><strong>The fifth "problem"</strong> was code block highlighting. At first I though this would be a significant undertaking that would require me to hook into ActionText, but in the end I used the Judo approach again and simply created a helper that changes the HTML before rendering.</div><pre>#!/usr/bin/ruby
module RichTextHelper
def transform_rich_text(rich_text, transforms: nil)
if transforms.blank?
transforms = methods
.select { |name| name.to_s.ends_with?("_rich_text_transform") }
.map { |name| name.to_s.gsub(/_rich_text_transform$/, "") }
end
document = Nokogiri.HTML(rich_text.to_s)
transforms.reduce(document) do |doc, transform|
public_send("#{transform}_rich_text_transform", doc)
end.to_s
end
# Makes code blocks look prettier / more readable
def highlight_code_blocks_rich_text_transform(document)
document.css("pre").each do |code_block|
code = code_block.text
formatter = Rouge::Formatters::HTML.new
lexer = Rouge::Lexer.guess({ source: code })
code_block.inner_html = formatter.format(lexer.lex(code))
code_block["class"] = [code_block["class"], "highlight"].select(&:present?).join(" ")
end
document
end
end
# then in some view
<%= transform_rich_text(@article.content) %></pre><div>Though my initial idea of hooking into ActionText led me to read it's source code, because of which I'm now much more familiar with the framework. So this detour wasn't a waste time.<br><br><strong>The final "problem"</strong> were link previews. That's the feature I love the most in this iteration of the blog and wanted to have it in action text. At first I through that I'd have to extend the helper from before but then it dawned on me that I could inject everything needed for link previews through Stimulus. I liked that approach in this case because the previews are a "frontend" feature and they don't work if someone disabled JS or blocks it. But I'm still unsure it that was a good idea and I might move the code to setup previews into the helper.<br><br>All in all, I love using ActionText and I'm glad I fixed my over-correction. Having an editor baked into the blog makes it so much easier to edit (at home and on the go). And since the HTML format is well documented and simple I can still convert it to some other format if need be.<br><br>ActionText was easy to integrate, and with hindsight I can say that it was also easy to migrate to it.<br><br><em>P.S. while reading through ActionText's source and issues </em><a href="https://github.com/basecamp/trix/issues/626#issuecomment-800338265"><em>I ran across an interesting quote form the author of markdown</em></a><em>:</em>
</div><blockquote>I have no idea why there are now apps that use Markdown as their back end storage format but only show styled text without the Markdown source code visible. Hey World, for example, gets this right: they just do simple WYSIWYG editing where bold is bold, italic is italic, and links look like links and the linked URL is edited in a popup. If you want WYSIWYG, do WYSIWYG. If you want Markdown, show the Markdown. Trust me, it’s meant to be shown.<br><br>- <a href="https://en.wikipedia.org/wiki/John_Gruber">John Gruber</a>
</blockquote>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/312023-04-26T06:01:00Z2023-11-14T16:28:05ZCorporate newspeak<div class="trix-content">
<blockquote>If thought corrupts language, language can also corrupt thought<br><br>- George Orwell</blockquote><div>One thing that annoyed me anywhere I worked was that people limit themselves to a narrow subset of English. A subset that's full of buzzwords and phrases that sound impressive but lack meaning. It's a subset designed to limit your ability to say anything that could be misinterpreted, and anything that expresses any emotion beyond a notion of happiness or dissatisfaction. This language couldn't mimic George Orwell's Newspeak more even if it tried, and therefore I call it "Corporate newspeak".<br><br>What annoys me is that speaking with someone in corporate newspeak feels like we are speaking in memes. We aren't saying what we think, we aren't being honest with one another, instead we are communicating in vague concepts and ideas. We are obscuring our though instead of clarifying them.<br><br>Take "We don't have the bandwidth" as an example. What does that sentence mean? It could be "The network switches in our server farm can't handle any more network traffic", or it could be "There isn't anyone available to work on this project in the next month, we would have to pause another project", or "Your project isn't something I think we should invest time in right now". There is no way to know without asking for clarification. But that will probably yield an answer with more newspeak and more buzzwords. Miscommunication is the goal of newspeak. If no one knows exactly what was said then there is a lot of room to avoid any kind of conflict.<br><br>Another thing that annoys me with newspeak is that it makes everything sound insincere, hollow, and boring because the vocabulary is made up of cliches. Take "We have an exciting announcement. From next quarter our Widget 3000 will come in two new colors - blue and green" as an example. Is that really an exciting announcement? I mean, it could be. But the excitement is undermined by every other announcement that was made that day that was also "exciting". If everything is exciting then nothing is exciting - it rings hollow.<br><br>Some find newspeak to be sincere, but I don't and maybe that's a culture difference. I find only language that I would use every-day sincere. And I don't know a single person that uses newspeak to talk to a friend, or a cashier, or a random person in the street. Maybe that's because of where I grew up? In Croatian the only difference between how you would talk to your friend and a business client is that you'd us the formal "You" instead of the regular "you" and you'd use "Greetings" instead of "Hi" when addressing them.<br><br>Communicating in memes slowly becomes thinking in memes. The cliches drown out any authenticity. Thoughts become blurry and hard to explain to others. That's a very high price for conflict avoidance.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/322023-07-31T12:00:00Z2023-11-14T16:28:03ZRent in Zagreb is hell<div class="trix-content">
<div>Last month my landlord called me and told me that I had to move out because he was going to sell the apartment I live in.<br><br>In a way, this came as a blessing. I wanted to move out of that place by the end of the year either way. But at the time I had no clue what hellscape finding an apartment in Zagreb had become since I last moved. This is a cautionary tale of what to look out for and what to avoid when renting in Zagreb.<br><br>I immediately started looking for a new place on Njuskalo (a local classified ads site).<br><br>The first thing I noticed was that rent has gone up by about 25%. I read about that in the news, so that didn’t surprise me too much. But what the news articles didn’t say is that for 3 bedroom apartments the price went up by 100% or more, which sucks because I always go for a 3 bedroom apartment so that I can have an office. Having the ability to clearly separate my work and my life environment is important to me.<br><br>After some searching I found a few reasonably priced apartments and arranged to go see them.</div><h2>Chapter 1: Paradigms & Marketing</h2><div>Seems like in the years since my last move most landlords took a philosophy course and decided to tackle the age old question of <em>“What is the ground floor of a building anyway?”</em>.<br><br>The first apartment I went to see was supposed to be on the ground floor of a small two story building. But upon entering the building - from the street - the real estate agent took me down a stairwell to a floor with a single door, opened it, and we stepped into the apartment.<br><br>“Very weird ground floor” I told him, “It’s almost 3 meters underground”. The real estate agent responded without hesitation with a happy but somewhat hostile sounding tone “Depends on what you consider the ground floor to be. This apartment is on the ground, and it has windows to the outside which are at waist height, it even has an exit to the garden. By all means this is the ground floor.”<br><br>I wasn’t in the mood to discuss paradigms like Physics professors defending the foundation of their lives work at some conference, so I sad my goodbyes and left. <br><br>Little did I know that this would just be the first in a series of apartments that were on the ground floor while at the same time at least one stairwell below the entrance and 1+ meters underground.<br><br>There were two camps of people; one believes that the ground floor is what's at the main entrance and everything above or below that is either + or - 1 floor, while the other camp believes that the ground floor is everything that touches the ground.<br><br>I am a firm believer in the ground floor is at the main entrance, but I was thrown into some bizarre world out of Kafka's fever dreams and just didn't know it yet.<br><br>How could a fundamental fact like this be subject to relative interpretation? I tried to see the the other camp's perspective - to understand their paradigm - but no matter how I sliced it this was just complete madness. <br><br>If everything that touches the ground is the ground floor then some buildings would have two ground floors or more - one on the street level and another in the basement - which made no sense at all. How would you explain to someone where you live? <br><br>How would someone know by looking from the street how many stories the building had, or even if they entered it? There would have to be a label on the building that warned visitors that there were more stories below ground just for them to be able to find the right floor without a map. <br><br>I already imagined a conversation like this: “See George, if you are one of the street-entrance-is-the-ground-floor folk, then I live on the 2nd floor; but if you are one of the-lowest-apartment-on-the-ground-is-the-ground-floor folk then I must tell you that there are two stories below street level in this building so I live on the 4th floor; but, on the other hand, if you are the there-are-multiple-ground-floors kind of guy then I', on the 2nd floor counting from the street entrance”.<br><br>Of course this wasn’t madness, this was just marketing. <br><br>Dugouts, and basement apartments aren’t hip in Zagreb anymore. Nobody in their right mind would pay 800 Euro or more to live in a damp basement, have to listen to 3 families living above their head, and battle mold all day long. When for 850 euro you can live on the 1st or 2nd floor of the same building, or buy your own apartment (while interest rates were low).<br><br>That’s a problem for real estate agencies as they are paid by commission, and for landlords as they want to get the most out of their property. But Zagreb has a real estate problem - price per square meter is comparable to Berlin, yet over half the buildings are 70 to 130 years old, and most of them are in a bad shape from being neglected for years due to co-owners not being able to agree on anything.<br><br>(seriously though, in my last building the co-owner's counsel couldn't agree on the color of the front gate for 2 months and then they just gave up and let the gate rust)<br><br>So realtors and landlords shifted the paradigm and the basement suddenly became the ground floor. In a way this is just a white lie. If you really like the apartment and the price is right for you, all they did was bend reality a bit to get you to see the apartment. But in the process they are gas-lighting everybody and inflating housing prices.</div><h2>Chapter 2: Lies, lies, lies!</h2><div>No matter which apartment I went to see all of them were built after 1963. This was odd for Zagreb - a city known for it’s extremely old buildings. <br><br>For apartments in Zagreb, and Croatia in general, knowing that a building was built after 1963 is very important due to earthquake code. <br><br><a href="https://en.wikipedia.org/wiki/1963_Skopje_earthquake">Before 1963 the code didn’t cover earthquakes of magnitude 6 or 7</a>. And with a recent 5.4 magnitude earthquake in 2020 that severely damaged a lot of buildings, people don’t want to pay as much for a building that could collapse on their head at any moment. <br><br>So landlords started lying, and over night all buildings got 50 years younger. A case so curious that Benjamin Button would be surprised. <br><br>Some didn’t outright lie and just said that they don’t know when the building was built. This is very odd since they invested several hundred thousand euros into that apartment, one would expect them to do due diligence. <br><br>But some outright lie. One apartment claimed to be build in 1965, but on Google Street View it was clearly visible that the facade had a stamp with the name of the architect that designed the building and the build year - which was 1898. <br><br>In my opinion this should be a criminal offense, as this lie could get someone killed when they thought that they were perfectly safe (and that they were paying for that safety).<br><br>This was the worst lie I encountered, but it wasn’t the only one. <br><br>People lied about the building’s construction. For example, wooden beam floors and ceilings become concrete floors and ceilings. One is paper thin and through it you can hear your upstairs neighbor like they are your roommate, while the other blocks all sound but direct impacts on the floor above. <br><br>Wooden windows become PVC windows. Two single pane windows become double pane windows. And stuff like that.<br><br>The funniest lie I saw was a "Modern *rustic* apartment in the center" which was actually an apartment in which somebody's grandmother lived until she passed away (recently) and they decided to rent it out with her furniture. This is usually OK, but the furniture was from the 60's, they charged for it like the place was freshly renovated, and to explain that they just slapped on "rustic" in the name of the ad like it would magically explain that the cabinets were made before nuclear reactors were invented and are starting to fall apart.</div><h2>Chapter 3: *Upkeep not included</h2><div>For a number of years there has been a growing number of landlords that don’t take on the upkeep but instead expect the tenant to pay it. They claim that upkeep is a utility like water or electricity so they exclude it from the rent. <br><br>Upkeep can be 30 to 150 euro or more, and it can change at any time. So an 850 euro apartment plus upkeep can be as expensive, if not more, as a 1000 euro apartment with upkeep included.<br><br>But the real problem is that upkeep isn’t like a utility. I, as a tenant, can control which utilities I use and how much I use them. And I <strong>consume</strong> the utility that I pay for.<br><br>But I can’t consume the upkeep and I can’t influence how the upkeep is spent. Only the owner of the apartment can do that.<br><br>This is like taxation without representation. The upkeep can be 30 euro today, and my landlord lobbies for a new facade for the building which then raises the upkeep to 150 euro and I'm stuck with the bill for it. I don’t get to keep the facade when I move out, and I don’t get to vote on if the building gets a new facade or not. So why should I pay for the upkeep?<br><br>This trend is just another way for landlords to reduce their risk and offload it to the tenant. It ensures that the landlord gets a constant sum of money every month, regardless of what the other co-owners of the building decide to do or if something in the building breaks. It also enables them to improve their property on the tenant's expense without having to renegotiate rent.</div><h2>Chapter 4: Parking in my apartment </h2><div>A few apartments seemed quite small for their advertised size. For example one 60 square meter apartment seemed more like a 40 square meter apartment to me. <br><br>So I asked the real estate agent what was going on, and he explained that the two out door parking spaces (8 square meters each) count towards the total area of the apartment.<br><br>Turns out that this is completely normal, and has been like this since forever. But to me this is utterly insane. <br><br>A parking space isn’t the same as a room in an apartment or in a garage. And I can’t believe that this requires explaining. <br><br>For one you can’t leave stuff unattended in a parking space. Just imagine leaving your laptop in a parking space and then complaining to the police that it was stolen while you weren't there. Or imagine one of your neighbors grilling on their parking space, or building a shed, or something like that...<br><br>But it seems I’m the crazy one here. Just keep this in mind when comparing two apartments by their listed size.</div><h2>Chapter 5: Contracts and Xenophobia </h2><div>I was down to a few apartments and started negotiating lease contracts when I learned that another major thing has changed in the years since my last move - contracts.<br><br>Before, it was normal to have a lease contract that basically said that you are leasing the place, that you have to give one month’s rent in advance as insurance, that you will return the place in the state you found it, and that by the 15th of each month you have to pay rent and utilities to the landlord.<br><br>Today, you have to sign a contract which gives the landlord the right to seize money from your bank account (or anything you own in case there isn't enough) in case you miss payment or owe them anything. This is in addition to a one month’s rent as insurance.<br><br>When I consulted a lawyer they warned me that the “in case you miss payment” is arbitrary and that the landlord can actually seize any amount whenever they want as long as the contract is active and they have semi-valid evidence to do so. <br><br>A government body does the seize in the name of the contract, they would give me 7 days to produce evidence that the seize isn’t valid after which they would just deduct money from my account and send me a processing fee for it. The problem here is that producing evidence is fairly easy for both sides. Just think about it, how would you prove that you are missing money? Or that it wasn’t paid in cash? Or deducted for a repair or other? So most cases end up in favor of the landlord and the tenant has to sue the landlord to reclaim the money, which can take years.<br><br>This if of course completely insane. Who in their right mind would allow a random person they don’t know unrestricted access to everything they own for the privileged to live in an apartment that they happen own?<br><br>When I asked why this was needed I got just one reply from all landlords “We had bad experiences with foreigners“. <br><br>So… xenophobia and lies. <br><br>Obviously I’m not a foreigner yet I’m still required to sign such a contract. But even if I was, how would this be any insurance? The Croatian government has only jurisdiction in Croatia - if a Canadian signs the contract, moves in, skip a months rent, then moves back to Canada, you’d still have to sue them and drag me to court to get your money.<br><br>So this is again just another new way of shifting risk from the landlord to the tenant.<br><br>Things can happen, people lose jobs, people get sick, people die, but the landlord doesn’t care about any of this; they just want a sure way to get their rent and this contract ensures it.<br><br>Another interesting addition to some contracts was a non-termination clause. In other words the contract can’t be terminated by the tenant, if they decide to move away before the lease expires they have to pay the lease in full for the duration of the contract. And, of course, leases are non transferable meaning that you can’t find them another tenant to replace you, and neither can you sub-lease your apartment.<br><br>This is slavery, and should be illegal. I can’t know going into a lease what will happen 9 months down the line. And I can’t know if the landlord lied to me before I move into the apartment, or what the neighbors are like. Yet I am expected to lock in for at least a year of rent, give unlimited access to everything I own, based on a 15min tour of the apartment… insanity… <br><br>This isn't normal, and this was never normal. Avoid such contracts at all cost. If you really like the place offer up two month's rent as insurance and see if they go for it.</div><h2>Chapter 6: Living in a forest</h2><div>My dream is to one day live in a house in a forest, or at least very close to one.<br><br>After a month of searching for an apartment I found one that’s in a forest, in a new building, made out of concrete, has a balcony, has an acceptable price, has a bog standard lease, has upkeep included, and is in a quiet neighborhood! <br><br>But it only has two bedrooms, and it’s 40min by foot to the closes store (a very European problem since until now I didn't have a reason to own a car).<br><br>I decided to go for it, and I’m so glad I did.<br><br>Now I enjoy my morning tea looking at the forest and the fog that emanates. The cold night air carries scents of leaves and lulls me to sleep. It's much easier to recharge and take a break than it was in a city apartment.<br><br><action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQyOT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--ef118b877e9024608e3683891e1a458eb2d32fac" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBcTBCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--110ac084205b627a093b83eb29b3e6fa5a30706a/IMG_4052.jpg" filename="IMG_4052.jpg" filesize="258483" width="1200" height="1600" previewable="true" presentation="gallery" caption="View from the balcony at night">
<figure class="attachment attachment--preview attachment--jpg">
<img loading="lazy" style="aspect-ratio:1200 / 1600;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDI5LCJwdXIiOiJibG9iX2lkIn19--494e652dfa96c15d08f9036d0d2f0d1bc0fe8eeb/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--0ab9331364f3a49361e292e4e525962adf38ddb2/IMG_4052.jpg">
<figcaption class="attachment__caption">
View from the balcony at night
</figcaption>
</figure></action-text-attachment>
</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/332023-08-28T05:00:00Z2023-11-14T16:28:04ZHold your own Poison Ivy<div class="trix-content">
<div>After two years, this week I finally caved and changed my current employment title on LinkedIn from Software Engineer to Software Architect.<br><br>This isn’t a position where I'm am supposed to do much programming - the thing I love most, and the thing I'm good at - but rather make high-level technical decision and manage people to achieve them. My problem with that, and any position like that, is best captured in the Croatian proverb <em>"It's easy to hold poison Ivy in somebody else's hands"</em> (that's the SFW Americanized version, <a href="https://translate.google.com/?sl=hr&tl=en&text=lako%20je%20tu%C4%91im%20kurcem%20po%20koprivama%20mlatiti&op=translate">here's the NSFW original</a>).<br><br>That proverb describes a decision made by someone who isn't involved in the execution of the decision, and doesn't suffer its consequences.<br><br>I don't want to be the kind of person who just lobs some solution over the wall and lets other people deal with the fallout.<br><br>I’ve had "managers" like that in the past, and it was hell working with them. Being told to do something that you know will cause problems, then bringing that up for discussion, explaining all the implications and new problems we will face, only to be shutdown with some hand-wavy explanation, and then having to deal with the consequences of that decision months or even years later is soul crushing. And if they can’t admit to their mistakes it can become burnout inducing.<br><br>And I know that engineering isn't really about <em>solving problems</em> as much as it’s about <em>displacing them</em>. We are constantly trading one problem for one or more other problems. But we try to do so conscientiously, weight the pros and the cons, and pick a solution that produces problems that we can live with.<br><br>The problem with Software Architects, or any purely managerial position, is that you aren't eating your own dog food, and therefore you don't live with the consequences of your decisions. So you produce <em>misfit solutions</em>.</div><blockquote>The problem of displacement is compounded by the existence of designers—special people whose job it is to solve problems, in advance, for other people. Designers, like landlords, seldom if ever experience the consequences of their actions. In consequence, designers continually produce misfits. A misfit is a solution that produces a mismatch with the human beings who have to live with the solution. Some mismatches are downright dangerous.<br><br>— <strong>Donald C. Gause</strong>, <strong>Gerald M. Weinberg</strong> in <a href="https://www.amazon.com/Are-Your-Lights-Figure-Problem/dp/0932633161">Are Your Lights On?: How to Figure Out What the Problem Really Is</a>
</blockquote><div>After two years, I’m confident that I’ve managed to avoid becoming that kind of “manager” by holding my own poison Ivy, and by helping others make decisions for themselves.<br><br>When I'm presented with a problem, I try to implement a proof of concept solution my self. Not a demo, held up with glue and duct tape; but a proper minimal implementation. Then I present the solution to my coworkers and gather feedback. That way I can feel the consequences of my solutions.<br><br>When someone comes to me with a problem, we go over all the possible solutions that we can come up with, then we discuss the pros and cons, and that's it. I let person decide for themselves which solution they will go with. All I ask of them is to tell me which one they chose. The point is to shine some light on problems that the person maybe couldn't see on their own, and point out some bigger-picture things that they might not know about. That way they can make an informed decision and choose the solution with which they can live with - so there aren't any misfit solutions.<br><br>This isn't a silver bullet by any means. Bad decisions still get made, I fuck up, my coworkers fuck up. But we learn, fix stuff, build trust, and improve.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/342023-10-10T09:45:00Z2023-11-14T16:28:05ZRails World<div class="trix-content">
<div>It was Wednesday morning and I got up even earlier than I usually do because I had to be on a plain to Amsterdam in a few hours. I haven't been to Amsterdam since 2019 when I briefly meandered through it's streets on my way to EuRuKo in Rotterdam. This time I was going there for Rails World which was special to me since I've never been to Rails-specific conference before.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQzMj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--6c58d48731fbb7a084c66a222667a54634fa0490" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBckFCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--520bd03d5edae837a9d6bda589ff787b825a8259/railsworld_banner.jpeg" filename="railsworld_banner.jpeg" filesize="1379733" width="2385" height="2746" previewable="true" presentation="gallery" caption="Rails World banner">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" style="aspect-ratio:2385 / 2746;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDMyLCJwdXIiOiJibG9iX2lkIn19--a6648eb141e8dc2d318aba91ed371b897cef4e9b/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/railsworld_banner.jpeg">
<figcaption class="attachment__caption">
Rails World banner
</figcaption>
</figure></action-text-attachment>Two brief flights and a train ride and I was at Amsterdam Centraal train station. It was a cloudy and windy day. I made my way to Beurs Van Berlage - a beautiful, old, ornate, red brick building with a tall clock tower, right in the heart of the city - to register for the conference.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQzMz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--b814c3e7e5ab8981c1eecc2604668aa350d116c2" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBckVCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--16e00255343ca7b39066dd235051ae6b8b4fcdfd/railsworld_beurs_van_berlage.jpeg" filename="railsworld_beurs_van_berlage.jpeg" filesize="3896779" width="3024" height="4032" previewable="true" presentation="gallery" caption="Beurs van Berlage">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" style="aspect-ratio:3024 / 4032;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDMzLCJwdXIiOiJibG9iX2lkIn19--6487c9cf4e62b255a9bfb21b4dff6a1599fde67a/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/railsworld_beurs_van_berlage.jpeg">
<figcaption class="attachment__caption">
Beurs van Berlage
</figcaption>
</figure></action-text-attachment>Then I made my way to the Zoku hotel where I would be staying until Sunday. The hotel looked modern from the Booking.com listing, but the pictures don't do it justice. To check-in I had to take an elevator all the way up to the 6th floor, which turned out to be a rooftop terrace with small ponds, a glassed-in walkway to the lobby, and a garden on each side of the walkway. The view was amazing, the clouds had cleared a bit, and the sun was just starting to set over the city.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQzND9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--c41732fea168ec511471cca325a29383ae08aa44" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBcklCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--f1d1d91660153c83e45a338261c437a14015e4c1/zoku_sunset.jpeg" filename="zoku_sunset.jpeg" filesize="2384375" width="2268" height="4032" previewable="true" presentation="gallery" caption="The sunset over Amsterdam as seen from the rooftop lobby of the Zoku hotel">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" style="aspect-ratio:2268 / 4032;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDM0LCJwdXIiOiJibG9iX2lkIn19--179866348c74d2434ff629899d65a805d70283b2/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/zoku_sunset.jpeg">
<figcaption class="attachment__caption">
The sunset over Amsterdam as seen from the rooftop lobby of the Zoku hotel
</figcaption>
</figure></action-text-attachment>The room was very stylish. It had everything a regular apartment would have - a stove top, a kitchen sink, a dishwasher, a dining table, a TV, a speaker, a couch, an office desk, shelves full of books, a bed, plus a bathroom - but it was arranged in a very compact way which made it fun to just explore the room. For example, the bed is above the shelves and to get into it you have to pull out a set of stairs out of the shelves to be able to climb up. The office desk is in a cupboard under the bed. In fact everything under the bed is storage space with various doors to it that act like closets. And there were a lot of quirky design touches like the office desk chair being made of concrete, the light switch in the bathroom being a plunger on a red string hanging from the ceiling, or the athletic rings hanging from the living room ceiling.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQzNT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--e32b1f05dc8304475be6745e558427b2051a0b71" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBck1CIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--747fabf5293d92747a843af819bb47fd1e960358/zoku_room.jpeg" filename="zoku_room.jpeg" filesize="1710227" width="3024" height="4032" previewable="true" presentation="gallery" caption="My room in the Zoku Hotel Amsterdam">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" style="aspect-ratio:3024 / 4032;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDM1LCJwdXIiOiJibG9iX2lkIn19--0f0cae69d1875a0f463a6d6703dd6f53c64e41ac/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/zoku_room.jpeg">
<figcaption class="attachment__caption">
My room in the Zoku Hotel Amsterdam
</figcaption>
</figure></action-text-attachment>The next morning, at 8:30 am, I made my way back to Beurs Van Berlage. There weren't many people out an about, which was strange to me. In Zagreb, there would be a flood of people in the streets at this time of day.<br><br>The venue was even more beautiful from the inside than from the outside. The red brickwork from the outside continued on the inside. There were four rooms, each of which was an atrium lit by sunlight coming through the colored glass ceiling held up by iron beams. Each room was three stories tall, and three of them had brick walkways on the first and second floor held up by arches and pillars.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQzNj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--fc8b4485b3f92756b1808dcae1edd9dc63ae8269" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBclFCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--ae23915971d0a443b1caf3b1303ed87761ae8d8d/beurs_van_berlage_ceiling.jpeg" filename="beurs_van_berlage_ceiling.jpeg" filesize="2851329" width="3024" height="4032" previewable="true" presentation="gallery" caption="The ceiling of the Beurs van Berlage">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" style="aspect-ratio:3024 / 4032;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDM2LCJwdXIiOiJibG9iX2lkIn19--c65b804651597856640807ad3989605c6e4f75b9/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/beurs_van_berlage_ceiling.jpeg">
<figcaption class="attachment__caption">
The ceiling of the Beurs van Berlage
</figcaption>
</figure></action-text-attachment>The opening keynote was about to start, so I made my way to the Track 1 room to find a chair. The room had moody lighting and a stage as wide as the whole room. <br><br>Amanda Perino, the executive director of the Rails Foundation, walked up on stage and give a captivating introduction. <a href="https://rubyonrails.org/foundation">She explained what the role of the Rails Foundation is</a>. Then she showed everywhere where people came from, the person that traveled the longest distance, and the person that traveled the shortest distance. People from all over the world came to Rails World. Finally she introduced the MCs, which in term introduced DHH.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQzNz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--55d943e328638f3ff39cb39587a2b00a26700c54" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBclVCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--4f328c60bab5417cf0d79adbb0268729df133654/railsworld_stage.jpeg" filename="railsworld_stage.jpeg" filesize="2307337" width="3024" height="4032" previewable="true" presentation="gallery" caption="The stage on Track 1 of Rails World">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" style="aspect-ratio:3024 / 4032;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDM3LCJwdXIiOiJibG9iX2lkIn19--d19e103ecaa93d3dd75f2c5f6da54fbdf7eb191b/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/railsworld_stage.jpeg">
<figcaption class="attachment__caption">
The stage on Track 1 of Rails World
</figcaption>
</figure></action-text-attachment>His keynote was a banger. It started off by introducing all the new features coming to Rails - solid_cache, solid_queue, Strada, Kamal, Propshaft, and Turbo 8 - and then switched to the concept of the “Renaissance Developer”, an analog to the Renaissance man or Polymath. A developer knowledgeable in many subjects - the backend, the frontend, deployments, and mobile apps - that can act as a one-person team. And Rails is exactly what enables such a developer. With money being tight as of late, these are the people we need to make great software.<br><br>The idea of a Renaissance Developer deeply resonated with me. I always like to broaden my knowledge, learn new thing, learn how things work, how people solved some problems and what were their reasons behind making certain decisions, and then dive deep into topics that interest me most. But I was always met with advice like “<em>specialize in one thing, because that’s financially smarter</em>” or “<em>don’t be a jack of all trades and a master of none</em>”. This never stopped me from perusing knowledge, but it wore me down to hear that I was going in “<em>the wrong direction</em>” every other day. Hearing DHH, and later people in the hallways, talk about the return of the Renaissance Man - the Renaissance Developer - was a breath of fresh air, inspiration, and motivation. I felt like I wasn't the crazy one after all.<br><br>I went to the talk about Strada by Jay Ohms next. He gave an introduction into using Strada to build iOS and Android apps. Explained how it interacts with Turbo native and Rails/HTML, the design decisions behind it and trade-offs. Like, it being designed to easily deploy new versions of the app without going through AppStore reviews, and having backwards and forwards compatibility in mind. It seems to be a really solid way to build mobile apps. It doesn’t yet fully cover offline, but it seems that there is a plan how to expand into offline territory with local storage and IndexedDB. All in all a great talk.<br><br>I learned how to hide UI elements using CSS and move them to native controls instead. You can do this because Strada updates a data attribute on the HTML element with a list of components it supports, so you can check if a component is enabled or not using square brackets and =~ selectors. Basically you implement a native method that adds a navbar button, for example, with which elements it should have and what it should send back when a button is pressed, from there you click the corresponding hidden UI element with JS and that’s it. It’s simple and that’s what I love about it. Everything we know and use in the web daily is translated over to mobile with a thin wrapper to control native UI elements.<br><br>Next, I went to Jorge Manrubia’s talk about Making a difference with Turbo. His talk was great. He explained all the new features in Turbo 8 with a focus on Turbo Morph. Basically, they are building a calendar for Hey (if you watched closely during DHH’s keynote you could notice a calendar button in the top center of his Imbox) and rendering it with turbo streams would have been difficult so they looked for simpler solutions and developed Turbo Morph in the process.<br><br>Generally, the easiest thing to do is to refresh a page when something updates - that's the old-school HTTP request-response way. But a full refresh is a jarring experience - the page flickers, you lose your scroll position, opened menus close again, CSS animations reset. <br><br>The next easiest solution are turbo frames, which allow you to update the page in sections as you interact with that section. This is similar to a full page refresh though without most of the jarring side effects, but you have to keep track of when you want to break out of the frame and that each page you can navigate to from the frame has a frame of the same name in it. <br><br>Finally, the hardest solution are turbo streams which are basically raw HTML manipulation events from the server. You have to know in which context you are sending them so you manipulate the HTML correctly. But they are the least jarring and feel like a native app.<br><br>Now with <a href="https://github.com/hotwired/turbo/pull/1019">Turbo Morph</a> you have the same native look and feel as turbo streams with the ease of use of a full page refresh. Morph basically does a full page refresh, takes the new HTML and morphs the current HTML into the new one. This preserves CSS animations, scroll state, open menus (with a flag), and doesn’t flicker the screen. All of this is done using a library called <a href="https://github.com/bigskysoftware/idiomorph">idiomorph</a>. They were inspired by Phoenix’s LiveView does the same thing just using a different library.<br><br>I’m hyped for the release of Turbo 8. Morph will make my life so much easier. I can basically delete most turbo stream responses at work and simplify the code a lot.<br><br>Then came the lunch break. The food was amazing! The catering service deserves a lot of praise. There were many different options and all of them were delicious.<br><br>After lunch, I went to Marco Roth’s talk about the future of Rails as a full-stack framework powered by Hotwire. His talk was also great. He focused more on the ecosystem around Hotwired, and how he came to Hotwired. He added a lot of nice user-friendly errors to Stimulus, like when you have a typo on a controller name or action it will tell you that instead of doing nothing. He also made an LSP for stimulus which looks amazing. It can suggest you completions specific to your app, like the exact controller, action, class, value or target name. He is also starting a weekly newsletter about Hotwired. And he started <a href="https://hotwire.io">hotwired.io</a> to aggregate and share all the cool thing in the ecosystem and community.<br><br>Then I went to Alivija Rojas’ talk about an offline experience with a Rails powered PWA. It was a very insightful talk about how to build a progressive web app with Rails, that caches certain pages locally and stores any form submissions made with no Internet to be synced later when Internet service is available. This can be done with a <a href="https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API/Using_Service_Workers">service worker</a> with which you cache the index and new pages. Then you add a Stimulus controller on the form that checks if Internet service is available before submission and if it isn’t then you store the form data in <a href="https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API">Indexed DB</a>. On the index page you check what you have in your local storage and render that. A neat idea.<br><br>Next, I went to Xavier Noria’s talk about Zeitwerk’s internals. I liked his presentation style a lot and enjoyed the talk. I did read through <a href="https://github.com/fxn/zeitwerk">Zeitwerk</a> some time ago so it was nice to hear the explanation of each concept in the codebase and how some tricky situations are resolved when loading in classes and modules.<br><br>For the last regular talk of the day I went to Breno Gazzola’s talk about <a href="https://github.com/rails/propshaft">Propshaft</a>. The goal of propshaft is to simplify asset serving a lot. It intentionally doesn't to anything besides generating digests and serving pre-processed files. This is in response to Sprockets, which do everything from integrating with Gulp to compiling coffeescript.<br><br>Finally, the closing keynote by Eileen Uchitelle. It was the same keynote she gave at <a href="https://m.youtube.com/watch?v=TKulocPqV38">RailsConf this year</a>. A very inspiring talk, that motivated me to contribute to the framework more when I could. This talk also reminded me that I had an open PR to Rails that I forgot about.<br><br>That was a wrap for day one so I walked back to my hotel room. Now there was a sea of people in the streets, seems like Amsterdam comes alive in the afternoon. <br><br>There are a few things I’ve heard that I didn’t go to a talk to or that didn’t have a talk about them. First, solid_queue is essentially, or at least is inspired by, <a href="https://github.com/bensheldon/good_job">good_job</a> but will be database agnostic. And it's coming by the end of the year. Second, <a href="https://github.com/rails/solid_cache">solid_cache</a> is a database backed cache. The speed of retrieving data is worse than e.g. caching in Redis, but we are talking about 1.2ms instead of 0.8ms, nobody is going to notice that difference. The benefit of using the database as a cache is that your cache can live for much longer because SSDs are larger and cheaper for storage than RAM. Having longer lived caches means that you have to compute results less often which can dramatically decrease response times.<br><br>The next day came around and I made my way back to the conference for an AMA with the Rails Core Team (10 out of 12 members were present). The AMA was lead by the creator of <a href="https://ohmyz.sh/">OhMyZSH</a>, Robby Russle. It was both inspiring and funny. Inspiring because the core team had nothing but words of encouragement towards the public, they want to help people contribute to Rails and know that some things can be daunring to beguinners which they could and are improving. And funny because of the answers, haggling with the host - especially between Aaron Patterson and Robby - and the bloopers. There was a wonderful moment when Robby asked all the contributors to rails to stand up, and about 10% of the audience stood up. Then he asked that anybody that has ever commented on a PR or similar to stand up, and another 20% of the audience stood up. Around 30% of the audience contributed to Rails in some way, which is a lot.<br><br>Next were two talks that were both very interesting and which I wanted to attend but they were happening at the same time - that was kind of the theme of the second day.<br><br>I went to the Rails and Ruby Garbage Collector talk by Peter Zhu first. I met the guy at EuRuKo last year and he seemed excited about this topic then so I figured this would be a great talk, and it was. He started off by introducing how GC works in Ruby. I wasn’t there for the rest of the talk but I heard that he created <a href="https://github.com/Shopify/autotuner">a gem called autotuner</a> to help you tune your Rails app’s GC.<br><br>Midway through Peter Zhu’s talk I switched to the Powerful Rails features you might not know talk by Chris Oliver. This was a fun talk to listen to, Chris is an excellent presenter and speaker. He showed many interesting tips and tricks that you can use in Rails. So many that they’d warrant a post of their own to cover.<br><br>I can’t wait for the videos to come so that I can watch both talks in full. <br><br>I didn’t go to any of the next two talks that were next because I met a few people I knew from other conferences and some that I met yesterday so we chatted for a bit. This was also perfect timing since lunch was up next and we could get a head start on the food before queues form.<br><br>After lunch, I went to Ryan Singer’s talk about Applying <a href="https://basecamp.com/shapeup">Shape Up</a> in the Real World. <a href="https://www.feltpresence.com/srl/">He sells a course</a> that seems to be a much more through version of this talk. Shape Up formalized most intuitions I already had about project management and taught me very helpful tools and processes for project management and development that are so simple and obvious that they are just beautiful, so I was looking forward to this talk. The talk was amazing, in my opinion it could be a keynote. <br><br>He started off with introducing all the problems of modern project management and development and drew analogs from that. <br><br>Like, “<em>working on tickets is like putting your project plan through a paper shredder</em>”. I can’t state enough how strongly I agree with this analogy. I’m amazed that people don’t notice that most problems during a project occur because nobody sees the big picture and decisions are made basically in the dark.<br><br>Next he had an analogy about cooperation which deeply resonated with me. “<em>Imagine renovating a room and spending moths laying out the new interior and particularly picking a lamp that will hang from the wall above the couch. Only to show the plan to the electrician and for him to tell you that there is no wire in that wall and that he’ll have to rip up the wall if you want to put the lamp there</em>”. I’ve been so many times in a spot where somebody just threw something extremely hard to implement over the wall for me to solve. This was always made worse by working on tickets since often times I didn't notice a problem until I was in the middle of the project. Just asking the team how hard something is to implement is a huge improvement in cooperation between project management and development.<br><br>There were so many nuggets of wisdom in this talk, it’s well worth the watch when the video comes out.<br><br>Next, I went to Adam Wathan’s talk about <a href="https://tailwindcss.com">Tailwind</a>. It was an introduction to Tailwind with some tips and tricks along the side, all explained and shown through live coding examples. Respect for pulling that off, it was an excellent talk.<br><br>Then I went to Irina Nazarova’s talk about making profit on OSS. This was a very interesting talk about which strategies you have if you want to make money on OSS and how you can bootstrap yourself. Among many other things, I learned that you can ask for sponsorship when a company asks for a feature on your project.<br><br>For the last regular talk on Friday I went to Don’t call it a Comeback by Jason Charnes. It was a very inspiring talk about the comeback of Rails with a reminder that we have to work to keep it that way and not take it for granted. We have to help people learn Rails, make the guides better, improve documentation, make videos, record podcasts and more to keep it relevant.<br><br>The closing keynote was Future of Developer Acceleration with Rails by Aaron Patterson. The talk started out with a lot of jokes poking fun at the opening keynote. <br><br>Then he turned to his aggravated onion, a personification of layers of things he’s mad about. It was fun to hear him rant for a bit. He mostly focused on how rewriting things to other languages is way more complicated than improving the algorithm something uses and it doesn’t actually solve the fundamental issue - a slow sort is slow in Rust and in Ruby. That people think that interop between Rust and Ruby is safe when in fact it comes with new problems, like memory leaks. And concluded that we all should just enable YJIT since it gives a great performance boost for no extra cost.<br><br>After that he turned to profiling. With profiling you can figure out which parts of your app are slow and should be optimized. He went over how a sampling profiler works and how to read a flame-graph and a call tree. Then he introduced <a href="https://github.com/jhawthorn/vernier">Vernier</a>, a sampling profile made by John Hawthorn and himself. Vernier is easy to integrate into Rails and comes with a nice UI to help you figure out what’s going on in your app.<br><br>Finally, he turned to language servers. He created a language server for Rails that shows you model attributes, routes and other useful information right in your editor. This is now part of <a href="https://github.com/Shopify/ruby-lsp-rails">ruby-lsp-rails</a> which is in term an addon for for <a href="https://github.com/Shopify/ruby-lsp">ruby-lsp</a>. His goal is for Rails to have one standard, built-in, LSP so that developers have the best experience possible out-of-the-box.<br><br>Except for the aggravated onion, these are things that he covered on his <a href="https://www.youtube.com/@TenderlovesCoolStuff/streams">live streams</a> in the past few months.<br><br>And that was it. Just the 20th anniversary celebration was left. The lights in the rooms were dimmed, and there were celebratory cupcakes given out. I was so tired that I decided to go to the hotel room after 15min, but I heard that the party was a blast.<br><br>I met a lot of new people, and a few old friends. <br><br>I got to chat with Chris Oliver who’s podcast - <a href="https://remoteruby.com">RemoteRuby</a> - is my go to Ruby podcast. It’s a blast to listen to him, Jason and Andrew. I even got a hand crafted, artisan, limited edition podcast sticker.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQzOD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--f3f25399a76ef454c6bddbe8231c905df7b70b23" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBcllCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--5b48d68add0ee2ae7d87766e798033b3ff19123d/remoteruby_sticker.jpeg" filename="remoteruby_sticker.jpeg" filesize="3270178" width="4032" height="3024" previewable="true" presentation="gallery" caption="Remote Ruby podcast sticker">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" style="aspect-ratio:4032 / 3024;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDM4LCJwdXIiOiJibG9iX2lkIn19--c709295c123d3d374ffa814cb3c905ab7626d07e/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/remoteruby_sticker.jpeg">
<figcaption class="attachment__caption">
Remote Ruby podcast sticker
</figcaption>
</figure></action-text-attachment>I also met DHH and Rafael Franca. Though the only thing I mustered to say to them was “<em>Thank you both, Rails really changed my life</em>”. I wanted to ask and say so much more but I was too tired to think at that point. And decided to stop there and go rest.<br><br>Meeting these people really humanized them. Until then they almost seemed super human, but talking to them, meeting them in person, and just seeing them in the crowd really showed me that they are just regular people which is inspiring to me.<br><br>I think Rails World was a huge success. The talks were so amazing that it was hard to pick which one to go to. Everything was well organized. The people were extremely nice. The volunteers/employees were super helpful. It wasn’t exorbitantly expensive (except for the hotels, but that's not their fault) and the location was great. Overall, just amazing. Amanda Perino did an outstanding job. I’m glad that I could attend it, and be there for Rails’ 20th anniversary - 20 years and still going strong!<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQzOT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--c97ae3b0f1e43313531f7021eada19c58558e4f2" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBcmNCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--7f96d8c3bc309cfcabd577f555128567e4933cad/rails_20th_anniversary_pin.jpeg" filename="rails_20th_anniversary_pin.jpeg" filesize="1572782" width="2483" height="2669" previewable="true" presentation="gallery" caption="Rails' 20th anniversary pin">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" style="aspect-ratio:2483 / 2669;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDM5LCJwdXIiOiJibG9iX2lkIn19--648f62f75e1002f5b141e3096e82f3bbc0caee56/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/rails_20th_anniversary_pin.jpeg">
<figcaption class="attachment__caption">
Rails' 20th anniversary pin
</figcaption>
</figure></action-text-attachment>The next morning was a busy day. It was my last day before going back so I wanted to tour the city and enjoy the museums. The Rijksmuseum is my favorite museum, I’ve been to it three times already, but I wasn’t visiting it this time. This time I bought tickets for the Van Gogh museum since I’ve never been there before.<br><br>The museum was huge. There are three floors full of his and his contemporaries' works. The first floor goes over his early life and how he got into painting. He used to be an art dealer, and at the age of 27 decided to become a painter.<br><br>The second floor covers his life as a painter, his struggles, and how he searched for his style. I learned that he was influenced by Japanese art, which he and his brother collected and admired. He traveled around but found happiness in the countryside and the simple life of peasants. On this floor are the sunflowers, one of his most famous works.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQ0MD9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--f785a373ceae22a1a50f7d8dda1c38e00b3cbd1c" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBcmdCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--2641030a0473b87b8031f0d8c0227372e59666f5/karla_with_the_sunflowers.jpeg" filename="karla_with_the_sunflowers.jpeg" filesize="2786919" width="3024" height="4032" previewable="true" presentation="gallery" caption="My girlfriend in front of the sunflowers by Van Gogh">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" style="aspect-ratio:3024 / 4032;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDQwLCJwdXIiOiJibG9iX2lkIn19--50b5a67c132def2ab18b93214ddca715de4ae526/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/karla_with_the_sunflowers.jpeg">
<figcaption class="attachment__caption">
My girlfriend in front of the sunflowers by Van Gogh
</figcaption>
</figure></action-text-attachment>On the last floor are the works immediately before his death, and the work of authors that he inspired. His painting career was relatively short lived with him committing suicide 10 years after becoming a painter. On this floor there were a number of his most famous paintings like the almond tree blossoms and the wheat fields.<br><br>I was surprised to see that most paintings were covered by glass. This really sucks because Vincent put on very heavy brush strokes which gave his pictures a texture, but through the glass this is barely noticeable due to the glare. I mean, it makes sense to protect these paintings when people throw tomato soup on priceless works of art, but it still sucks.<br><br>Something that I never noticed in prints of his work is how good he was at capturing depth. His paintings draw you in, there is a 3D effect going on that just isn’t visible in a photograph of the same painting. And the colors on the paintings seem much more vivid in person.<br><br>There was a crossover exhibition going on too. Since Vincent was inspired by Japanese art, the Pokemon illustrators made a few Pokemon cards and paintings in his style. Needless to say, the gift shop was raided for anything and all Pokemon.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQ0MT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--f62e235b1c5bb83bd1f9c148d7d23ec46fd10afc" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBcmtCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--415e6bbd95dda4be8ad318453af75205aa49e52d/van_gogh_pikatchu.jpg" filename="van_gogh_pikatchu.jpg" filesize="2109815" width="2860" height="3705" previewable="true" presentation="gallery" caption="Pikachu with a hat, drawn in the style of Van Gogh">
<figure class="attachment attachment--preview attachment--jpg">
<img loading="lazy" style="aspect-ratio:2860 / 3705;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDQxLCJwdXIiOiJibG9iX2lkIn19--898e02fa1e9fb379df4b64b7732c99ce1e0e435b/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--0ab9331364f3a49361e292e4e525962adf38ddb2/van_gogh_pikatchu.jpg">
<figcaption class="attachment__caption">
Pikachu with a hat, drawn in the style of Van Gogh
</figcaption>
</figure></action-text-attachment>Next up was the the De Hortis, Amsterdam’s botanical garden. It’s one of the best botanical gardens I’ve been to. The garden has 7 glass houses, an outdoor park, and bee hives. There is a tropical, sub-tropical, and desert glass house. The desert plans look like they came from an alien planet. While the tropical glass house had gigantic plants and was full of small brown singing frogs.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQ0Mj9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--613bbd5a41b125be237377c5132e2705dc26cc5a" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBcm9CIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--dd9c0a5a0353719faafce4030512f8962adde451/giant_liliypads.jpeg" filename="giant_liliypads.jpeg" filesize="3675699" width="3024" height="4032" previewable="true" presentation="gallery" caption="Gian lily pads from the tropical glass house at the De Hortis">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" style="aspect-ratio:3024 / 4032;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDQyLCJwdXIiOiJibG9iX2lkIn19--eda6a86fc2cc6dbe34442c6b981d19f79bcc74ae/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/giant_liliypads.jpeg">
<figcaption class="attachment__caption">
Gian lily pads from the tropical glass house at the De Hortis
</figcaption>
</figure></action-text-attachment>There was a very beautiful glass house with all sorts of plants.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQ0Mz9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--b32af0113c6b1c0ee9ea3f7e52e68321dbc3a038" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBcnNCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--ff2ccc1cd8a5f088398b9852b340cb2633eb07a3/glass_house.jpeg" filename="glass_house.jpeg" filesize="4969197" width="3024" height="4032" previewable="true" presentation="gallery" caption="A glass house at the De Hortis">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" style="aspect-ratio:3024 / 4032;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDQzLCJwdXIiOiJibG9iX2lkIn19--626a831e11b760fc19860fd5fb13d81576c0d142/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/glass_house.jpeg">
<figcaption class="attachment__caption">
A glass house at the De Hortis
</figcaption>
</figure></action-text-attachment>And a glass house that served as a nursery for butterflies with dozens of them flying about pollinating plants.<br><br>By now it was already late in the afternoon so I went back to the hotel to pack up.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQ0ND9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--e31f641162bec4722e68bc4099a01e4de26c7ba9" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBcndCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--98f9d10bbbffe3d0926319404b9231c881b4cc36/zoku_walkway.jpeg" filename="zoku_walkway.jpeg" filesize="2055083" width="2268" height="4032" previewable="true" presentation="gallery" caption="Walkway from the elevator to the lobby of the Zoku hotel">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" style="aspect-ratio:2268 / 4032;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDQ0LCJwdXIiOiJibG9iX2lkIn19--e8c6489e346a70f25f987b40e62feb653cbf147d/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/zoku_walkway.jpeg">
<figcaption class="attachment__caption">
Walkway from the elevator to the lobby of the Zoku hotel
</figcaption>
</figure></action-text-attachment>The next morning I made my way to Schiphol airport and then back to Zagreb.<action-text-attachment sgid="BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vYmxvZy9BY3RpdmVTdG9yYWdlOjpCbG9iLzQ0NT9leHBpcmVzX2luBjsAVEkiDHB1cnBvc2UGOwBUSSIPYXR0YWNoYWJsZQY7AFRJIg9leHBpcmVzX2F0BjsAVDA=--d44c0fb5d32ef0cd73a37ba1baadcff8464d46ac" content-type="image/jpeg" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBcjBCIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--69d0e2ad6d3eca122b26dfa393f5d8087e370352/shiphol_terminal_b36.jpeg" filename="shiphol_terminal_b36.jpeg" filesize="2729773" width="3024" height="4032" previewable="true" presentation="gallery" caption="Terminal B36 at Schiphol Airport">
<figure class="attachment attachment--preview attachment--jpeg">
<img loading="lazy" style="aspect-ratio:3024 / 4032;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDQ1LCJwdXIiOiJibG9iX2lkIn19--3533716cee2f09d4cc1a9f05737c3038f3c56f1a/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJqcGVnIiwicmVzaXplX3RvX2xpbWl0IjpbMTAyNCw3NjhdfSwicHVyIjoidmFyaWF0aW9uIn19--26576d22b8977d18c8e7297a2965dea96336b076/shiphol_terminal_b36.jpeg">
<figcaption class="attachment__caption">
Terminal B36 at Schiphol Airport
</figcaption>
</figure></action-text-attachment>
</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/352023-11-14T05:17:00Z2023-11-14T16:28:03ZTracking online presence with ActionCable<div class="trix-content">
<div>This month I worked on tracking when a device connected and disconnected to a WebSocket backed with <a href="https://guides.rubyonrails.org/action_cable_overview.html">ActionCable</a>. At first, this seemed like a simple problem to solve, but it turned out to be much more complicated.<br><br><a href="https://github.com/monorkin/actioncable-presence-detection-demo/tree/b86c5b02b4e411c5f43cd006629a8549dbbabd65">I started with a simple solution</a>, which was to mark a Device as being online in the #subsribed method, and marking it as offline in the #unsubscribed method. These methods are called when a client subscribes or unsubscribes to a channel. <a href="https://guides.rubyonrails.org/action_cable_overview.html#example-1-user-appearances">This is also the solution from the ActionCable guides</a>.</div><pre><em>#!/usr/bin/ruby
</em>
class EventsChannel < ApplicationCable::Channel
def subscribed
Current.device.came_online
stream_from Current.device
end
def unsubscribed
Current.device.went_offline
end
end</pre><div>#came_online and #went_offline internally just update an online boolean column to either true or false, and also create a log record that the device came online or went offline.</div><pre><em>#!/usr/bin/ruby
</em>
class Device < ApplicationRecord
has_many :online_status_changes, dependent: :destroy
scope :online, -> { where(online: true) }
scope :offline, -> { where(online: false) }
def came_online
update!(online: true)
online_status_changes.create!(status: :online)
end
def went_offline
update!(online: false)
online_status_changes.create!(status: :offline)
end
end</pre><div>And that works pretty well. If I connect as a device I can see a green orb appear next to to the device's name, and if I disconnect the orb turns red.<action-text-attachment sgid="eyJfcmFpbHMiOnsiZGF0YSI6ImdpZDovL2Jsb2cvQWN0aXZlU3RvcmFnZTo6QmxvYi80OTI_ZXhwaXJlc19pbiIsInB1ciI6ImF0dGFjaGFibGUifX0=--0765f58931d0578a87473313894bdc29e80035b1" content-type="video/webm" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDkyLCJwdXIiOiJibG9iX2lkIn19--9e238d7f25351c8d1a9c1fe01f674baa24040c3f/presence_app_demo.webm" filename="presence_app_demo.webm" filesize="347915" width="1920.0" height="1012.027972027972" previewable="true" caption="Demonstration of the device going online and offline within a simple presence tracking app">
<figure class="attachment attachment--preview attachment--webm">
<video controls="controls" autoplay="autoplay" loop="loop" muted="muted" loading="lazy" poster="/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDkyLCJwdXIiOiJibG9iX2lkIn19--9e238d7f25351c8d1a9c1fe01f674baa24040c3f/eyJfcmFpbHMiOnsiZGF0YSI6eyJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--21cb38a28c52d6c3cf41ca159781d10834673b37/presence_app_demo.webm" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDkyLCJwdXIiOiJibG9iX2lkIn19--9e238d7f25351c8d1a9c1fe01f674baa24040c3f/presence_app_demo.webm"></video>
<figcaption class="attachment__caption">
Demonstration of the device going online and offline within a simple presence tracking app
</figcaption>
</figure></action-text-attachment>I soon discovered a problem with that solution. If I open the page on my phone, connect as a device, and then turn airplane mode on, the app will still think that the device is online even through the phone isn't connected to WiFi anymore.<action-text-attachment sgid="eyJfcmFpbHMiOnsiZGF0YSI6ImdpZDovL2Jsb2cvQWN0aXZlU3RvcmFnZTo6QmxvYi80OTc_ZXhwaXJlc19pbiIsInB1ciI6ImF0dGFjaGFibGUifX0=--f214a5056a9583150d9be6ddad38aa7e5789e526" content-type="video/webm" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDk3LCJwdXIiOiJibG9iX2lkIn19--cf03897002eae451e81c03b024eccd239e3458a2/presence_app_offline_demo_av1.webm" filename="presence_app_offline_demo_av1.webm" filesize="3013833" width="1918.0" height="1080.0" previewable="true" caption="Demonstration of the device staying online when the device is unplugged from the Internet">
<figure class="attachment attachment--preview attachment--webm">
<video controls="controls" autoplay="autoplay" loop="loop" muted="muted" loading="lazy" poster="/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDk3LCJwdXIiOiJibG9iX2lkIn19--cf03897002eae451e81c03b024eccd239e3458a2/eyJfcmFpbHMiOnsiZGF0YSI6eyJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--21cb38a28c52d6c3cf41ca159781d10834673b37/presence_app_offline_demo_av1.webm" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDk3LCJwdXIiOiJibG9iX2lkIn19--cf03897002eae451e81c03b024eccd239e3458a2/presence_app_offline_demo_av1.webm"></video>
<figcaption class="attachment__caption">
Demonstration of the device staying online when the device is unplugged from the Internet
</figcaption>
</figure></action-text-attachment>What is going on?<br><br>Well, most operating systems keep a connection "alive" and buffer messages for it if the other end suddenly disconnects (without closing the TCP connection). This is so that small hiccups in the network don't cause you problems, errors or warning messages. This connection state is known as <a href="https://en.wikipedia.org/wiki/TCP_half-open">half-open</a>.<br><br>Depending on the OS, a connection can be considered held half-open in this way anywhere from a few minutes to nearly half an hour, or until the buffer overflows which can be dozens to hundreds of megabytes of messages.</div><div>Since I want to know when a device disconnects as soon as possible this buffer and the delay it introduces is a problem.<br><br>This behavior surprised me a bit since I know that ActionCable keeps a heartbeat which is supposed to detect sudden disconnects and prevent half-open connections from lingering.<br><br>Why didn't the heartbeat save me this trouble? I had to dive deep into ActionCable to figure out.<br><br>A heartbeat is a simple message, sent back and forth between the client and server every few seconds. It generates traffic, so the connection doesn't become stale, and if one side doesn't receive the heartbeat in time it knows that it has lost connection to the other side. The heartbeat messages are commonly called "ping" and "pong". The WebSocket protocol itself has a special ping & pong message, but most WebSocket libraries implement their own application-level ping & pong for various reasons I won't get into.<br><br>The ping message can be sent from the server to the client - this is known as a server-initiated ping - or from the client to the server - which is known as a client-initiated ping. Usually, whoever gets the ping responds back with a pong message. If one side doesn't receive a ping or pong back within a given time frame they assume that the other side has disconnected.<br><br>Turns out, in ActionCable, the heartbeat is server-initiated and one-directional. This means that the server periodically sends a ping to all clients, but it doesn't expect a pong back.<br><br>That's a bit curious, but it makes sense. The point is to help the client discover that it has lost connection so that it can reconnect. The downside of such a heartbeat is that the server can't know that the client has disconnected abruptly, so it can't close the connection until either the OS buffer overflows or times out.<br><br>In my case, just adding a pong response to the client, and the ability for the server to close any connection that didn't receive a pong within two heartbeats would allow me to detect a disconnect within 9 sec which is a great improvement over the 30 min from before.<br><br><a href="https://github.com/monorkin/actioncable-presence-detection-demo/tree/7d239de992654ec944784a02c55a3dd69823aa75">So I monkey patched that in</a>, and it worked pretty well.<action-text-attachment sgid="eyJfcmFpbHMiOnsiZGF0YSI6ImdpZDovL2Jsb2cvQWN0aXZlU3RvcmFnZTo6QmxvYi80OTg_ZXhwaXJlc19pbiIsInB1ciI6ImF0dGFjaGFibGUifX0=--15855d18c99820f8ddb89dea38323fee64632ef2" content-type="video/webm" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDk4LCJwdXIiOiJibG9iX2lkIn19--367f1b5629ffeb12090bbf2289bad97c8088ebbd/presence_app_pong_monkeypatch.webm" filename="presence_app_pong_monkeypatch.webm" filesize="2728875" width="1920.0" height="1080.0" previewable="true" caption="Demonstration of the device going offline when unplugged when PONG messages are added">
<figure class="attachment attachment--preview attachment--webm">
<video controls="controls" autoplay="autoplay" loop="loop" muted="muted" loading="lazy" poster="/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDk4LCJwdXIiOiJibG9iX2lkIn19--367f1b5629ffeb12090bbf2289bad97c8088ebbd/eyJfcmFpbHMiOnsiZGF0YSI6eyJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--21cb38a28c52d6c3cf41ca159781d10834673b37/presence_app_pong_monkeypatch.webm" src="/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NDk4LCJwdXIiOiJibG9iX2lkIn19--367f1b5629ffeb12090bbf2289bad97c8088ebbd/presence_app_pong_monkeypatch.webm"></video>
<figcaption class="attachment__caption">
Demonstration of the device going offline when unplugged when PONG messages are added
</figcaption>
</figure></action-text-attachment>In my opinion this is a generally useful feature so I went to open up a PR to ActionCable and checked for any preexisting PRs or issues related to ping and pong messages. There were two closed issues (<a href="https://github.com/rails/rails/issues/24908">#24908</a> & <a href="https://github.com/rails/rails/issues/29307">#29307</a>) and <a href="https://github.com/rails/rails/issues/45112">another open one</a>.<br><br>While slightly different all of them seem to focus on the same underlying issue of handling sudden disconnects. But the proposed solution for them isn't a pong response, instead it's a switch to client-initiated heartbeats.<br><br>I was a bit surprised by the discussion leaning towards client-initiated heartbeats. Pong messages are a much simpler and less invasive solution which solves the same issue. So I read through all the discussions and did some research to understand what else client-initiated heartbeats bring to the table.<br><br>The issues highlight the following as advantages of client initiated heartbeats:</div><ul>
<li>Ability to change the heartbeat on the client-side, this would allow some clients to do heartbeats more frequently to detect if the connection broke more quickly</li>
<li>The client could measure latency - the time it takes for a message to do a round trip - to the server</li>
</ul><div>There were two other features mentioned:</div><ul>
<li>Dropped message detection</li>
<li>Reporting of the last-received message</li>
</ul><div>These are technically doable by adding a <a href="https://en.wikipedia.org/wiki/Lamport_timestamp">Lamport timestamp</a> to each message, but they raise a new question "<em>What do we do when we detect a dropped message?</em>" which is application-level stuff and a major expansion of the framework, so I won't delve into these features.<br><br>I did <a href="https://github.com/monorkin/actioncable-presence-detection-demo/tree/82a851f0de0f8db414e3c8055fa28be4031ffd90">a proof of concept implementation of client-initiated heartbeats</a>, before I found out <a href="https://socket.io/">socket.io</a> actually changed from client-initiated to server-initiated heartbeats. <br><br>Reading through the <a href="https://github.com/socketio/socket.io/issues/2769#issuecomment-639300952">GitHub issue</a> where the switch was decided, I learned that <a href="https://developer.chrome.com/blog/timer-throttling-in-chrome-88/">Chrome has started throttling setTimeout and setInterval for tabs that aren't in the foreground</a> in 2021 to lower power consumption (extend battery life). When your app is in a tab that gets throttled the fastest that the heartbeat timer could run is once a minute. When that happens the client would send out heartbeats so slow that the server would think that it disconnected and it would close the connection, that would cause the client to reconnect, only to get disconnected again before the next heartbeat. Basically, the client would reconnect loop.<br><br>Because of this, I abandoned client-initiated heartbeats and returned to the original pong message idea.<br><br>There were a few things to iron out in the pong monkey patch. The most important one was forwards and backwards compatibility. The original implementation waited for the client to respond with a pong to switched the connection to expect pong messages. This allows clients that don't support pong responses to still function, while clients that know about pong responses get improved presence detection.<br><br>This is forwards-compatible (<em>client that don't send pong messages can connect to servers that expect them</em>), and somewhat backwards-compatible (<em>clients that do send pong messages can connect to servers that don't expect them</em>) if you don't mind the error log messages that this will cause (<em>it won't crash the WebSocket</em>).<br><br>But there is also a <a href="https://en.wikipedia.org/wiki/Race_condition">race condition</a> in this upgrade process, when a client connects and then immediately disconnects before they respond to the first heartbeat. When that happens the server is stuck thinking that the client doesn't support pongs even though they do, and will wait for the OS to close the connection and improved presence detection is lost. That’s exactly what I don’t want.<br><br>The more elegant and robust solution would be for the client to signal to the server that it supports pongs when it’s opening the WebSocket. And for the server to respond back that pong messages have to be used, so that the client knows it should send them. That way I can mark the connection as expecting pongs right away, and the client can connect to servers that don't support pong messages.<br><br>Luckly, WebSockets provide a mechanism for exactly that - it's called sub-protocols. When a client makes a HTTP request to open a WebSocket connection it can send a list of sub-protocols to the server in <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Protocol_upgrade_mechanism#sec-websocket-protocol">a Sec-WebSocket-Protocol header</a>. The sub-protocols have to be ordered by preference (the one you'd like the most first). The server chooses the first protocol from the list that it supports, and adds a Sec-WebSocket-Protocol header to it's response with just the chosen protocol. With that both the server and the client know exactly how to communicate over the WebSocket.<br><br>ActionCable already uses Sec-WebSocket-Protocol both on the server and in the official client. The name of its protocol is actioncable-v1-json. Since the pong message is a change to the protocol, I created a new protocol revision - actioncable-v1.1-json - which expects a pong response to the server's heartbeat ping, and <a href="https://github.com/monorkin/actioncable-presence-detection-demo/tree/9da4ce9df09de2b51e681f24df090aafe5949016">updated the monkey patches</a>.<br><br>The only thing left to do now was to open a PR to Rails.<br><br>The first step towards that was to expose the latency metric to the application somehow. Up until now I only measured the latency and logged it in the connection object itself. It would be nice if I could somehow hook into that metric so that I can process it and store it on the device model somehow. For this I chose <a href="https://api.rubyonrails.org/classes/ActiveSupport/Notifications.html">ActiveSupport Notifications</a>. I've added <a href="https://github.com/monorkin/actioncable-presence-detection-demo/commit/e52488751c25dc4c2707835d5a6b60ff2d792c10">a new connection.latency notification</a>, through which I can monitor, and respond to, the latency of any ActionCable connection.<br><br>The second step was to improve the naming of methods. As I developed the monkey patches I changed terminology a few times so the names are all over the place. I settled on calling such connections <a href="https://github.com/monorkin/actioncable-presence-detection-demo/commit/a3e4e682f93cdadf59fc532d19f1b30f5bd192da">half-open instead of dead, stale or expired</a>. That seems to be the most apt description.<br><br>The third step was to <a href="https://github.com/monorkin/actioncable-presence-detection-demo/commit/e6d090199db396a9fcd5ef70d633068216692d2c">treat the connection as open if any message was received within the PONG timeout</a>. After all, you can't receive messages if the connection is half-open, and if the server is processing a lot of incoming messages it might take too long to process a PONG message from some client and incorrectly think that the connection is half-open. There is also <a href="https://github.com/rails/rails/pull/49168">an open PR to add the same behavior to the official JS client</a>.<br><br><a href="https://github.com/monorkin/rails/tree/add-pong-response-to-heartbeat-ping-messages">The fourth step was to fork Rails, create a new branch from main, apply all the changes from the monkey patches to it, and add tests</a>.<br><br>The tests were a bit odd compared to other core Rails gems. In all other gems you only have to run bin/test to run the whole test suite, but in ActionCable you also have to run yarn pretest, bin/test and yarn test. This isn't well documented, I stumbled upon this by accident when I intentionally wrote a failing test.<br><br>And the final step was to double check that I did everything in the <a href="https://guides.rubyonrails.org/contributing_to_ruby_on_rails.html">contributing to Rails guide</a>, and <a href="https://github.com/rails/rails/pull/50039">opening a PR</a>.<br><br>With that out of the way, there is one more edge case that I've found. A device can suddenly lose connection, notice that, regain Internet service, and reconnect to the server, before the server closes the leftover half-open connection. When the server finally closes the half-open connection the device will be marked as offline in the database, even through it isn't.<br><br>To solve that all I had to do is change the online boolean column to a connection_count integer column. Every time the device connects that column is incremented, and every time it disconnects it's decremented. If the count is zero, the device is offline, if it's positive the device is online. The only thing that I had to do in addition to that is to lock the row before updating the count so that a race condition doesn't change the count incorrectly.</div><pre><em>#!/usr/bin/ruby</em>
class Device < ApplicationRecord
has_many :online_status_changes, dependent: :destroy
scope :online, -> { where(connection_count: (1...)) }
scope :offline, -> { where(connection_count: 0) }
def came_online
with_lock do
old_count = reload.connection_count
update!(connection_count: old_count + 1)
end
online_status_changes.create!(status: :online) if old_count.zero?
end
def went_offline
with_lock do
old_count = reload.connection_count
update!(connection_count: old_count - 1)
end
online_status_changes.create!(status: :offline) if connection_count.zero?
end
end
<br><br></pre><div>And that's it.</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/362024-02-05T13:00:00Z2024-02-26T07:52:25ZDeconstructing Action Cable<div class="trix-content">
<div>
<action-text-attachment sgid="eyJfcmFpbHMiOnsiZGF0YSI6ImdpZDovL2Jsb2cvQWN0aXZlU3RvcmFnZTo6QmxvYi81MDM_ZXhwaXJlc19pbiIsInB1ciI6ImF0dGFjaGFibGUifX0=--aa1cc49ba66fd6279e8b54d764475a2f846291fc" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NTAzLCJwdXIiOiJibG9iX2lkIn19--b922449e26a0d70fc8cd220e3404325e9c43fe75/00011-2829697632.png" filename="00011-2829697632.png" filesize="614577" width="904" height="512" previewable="true" presentation="gallery" caption="Deconstructing Action Cable">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" style="aspect-ratio:904 / 512;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NTAzLCJwdXIiOiJibG9iX2lkIn19--b922449e26a0d70fc8cd220e3404325e9c43fe75/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/00011-2829697632.png">
<figcaption class="attachment__caption">
Deconstructing Action Cable
</figcaption>
</figure></action-text-attachment>I knew what Action Cable was about and roughly how it worked since it was released. But lately I've developed a much deeper understanding of it because I had to extend it and explain its internals to others.<br><br>
</div><div>The following is how I explained Action Cable to myself. I took a top-down approach, starting at what I could observe Action Cable doing in my browser and following the code until I found the implementation, which I then deconstructed to understand why it does what it does.<br><br>
</div><div>This was a along journey down a deep rabbit hole, but it was worth it. At the end I now understand how and why Action Cable does what it does, and appreciate how simple and elegant it actually is.</div><h2>From HTTP to WebSocket</h2><div>Action Cable is a framework that allows your server to push messages to the browser, or any other client.<br><br>
</div><div>Usually your server can only respond to requests that a client, like your browser, makes to it. But Action Cable opens a WebSocket between the client and the server through which the server can push messages to the client at any time.</div><div>
<br>The client can also push messages to the server through that WebSocket. But that's in my opinion less interesting as we could do that without Action Cable.<br><br>
</div><div>
<em>How does the client connect to Action Cable's WebSocket?<br></em><br>
</div><div>When anyone or anything wants to connect to your Rails app via Action Cable they first have to make a HTTP request to "/cable" and after a bit of back and forth out pops a WebSocket.<br><br>
</div><div>
<em>But how does a HTTP request become a Websocket?<br></em><br>
</div><div>Back in 1997 HTTP version 1.1 was released. It's very similar to HTTP 1.0 with a few additions, the one that's important for WebSockets is <a href="https://datatracker.ietf.org/doc/html/rfc2616#section-10.1.2">the addition of status 101 aka. Switching protocols and the <em>"Upgrade"</em> header</a>. The intent of status 101 was to enable browsers to gracefully transition from HTTP 1.x to an eventual HTTP 2 or some other protocol that would be incompatible with HTTP 1.<br><br>
</div><div>And that's exactly what a browser does when it connects to a WebSocket. It first makes a request with the <em>"Upgrade"</em> header set to "websocket" which tells the server "hey, I want to switch to using WebSockets for this request". If the server supports WebSockets it responds with status 101 indicating that the upgrade was accepted and that, from now on, it will communicate using the WebSocket protocol on that request's connection.<action-text-attachment sgid="eyJfcmFpbHMiOnsiZGF0YSI6ImdpZDovL2Jsb2cvQWN0aXZlU3RvcmFnZTo6QmxvYi81MDQ_ZXhwaXJlc19pbiIsInB1ciI6ImF0dGFjaGFibGUifX0=--5a68beb829d7cb9c21f7e6cf2922b1476bda48d0" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NTA0LCJwdXIiOiJibG9iX2lkIn19--f6ae760fe0f0ebfe84c7e2b5b2f45ee5400f67f3/Pasted%20image%2020240124151038.png" filename="Pasted image 20240124151038.png" filesize="196098" width="771" height="756" previewable="true" presentation="gallery" caption="A WebSocket upgrade request">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" style="aspect-ratio:771 / 756;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NTA0LCJwdXIiOiJibG9iX2lkIn19--f6ae760fe0f0ebfe84c7e2b5b2f45ee5400f67f3/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/Pasted%20image%2020240124151038.png">
<figcaption class="attachment__caption">
A WebSocket upgrade request
</figcaption>
</figure></action-text-attachment><em>But why do we need the HTTP request in the first place? Can't we just connect directly via the WebSocket protocol since we already know that the server supports it?<br></em><br>
</div><div>The genius of WebSockets lies in the initial HTTP request. Because of that initial HTTP request we have access to all the browser's cookies that are sent over, so we can use them to figure out who wants to open a WebSocket, if they are allowed to do that, and we can remember who the socket is for.<br><br>
</div><div>
<em>So Action Cable converts HTTP requests to WebSockets?<br></em><br>
</div><div><strong>Yes! It's a WebSocket server, but it also does so much more.</strong></div><h2>Connections</h2><div>
<em>So someone's connected to our WebSocket, but how do we know who we are talking to and how can we send messages to them?<br></em><br>
</div><div>In Action Cable, each open WebSocket has a corresponding Connection object. The connection object is responsible for keeping track of who the WebSocket is for and figuring out if the client is allowed to open a WebSocket.<br><br>
</div><div>When a client makes the initial HTTP request to <em>"/cable"</em>, Action Cable will take the request's headers, cookies, URL and create a Connection object from it.<br><br>
</div><div>If it exists, the Connection object will be an instance of <em>"ApplicationCable::Connection"</em>. If that class doesn't exist it will be an instance of <a href="https://github.com/rails/rails/blob/cc633481602daf004eaab845c3780c6d885f882c/actioncable/lib/action_cable/connection/base.rb"><em>"ActionCable::Connection::Base"</em></a>. You can change the class of the connection by setting <em>"config.action_cable.connection_class = ->{ WhateverClassIWant }"</em>.<br><br>
</div><div>When a connection is established the <a href="https://github.com/rails/rails/blob/cc633481602daf004eaab845c3780c6d885f882c/actioncable/lib/action_cable/connection/base.rb#L186"><em>"#connect"</em></a> method on our <em>"ApplicationCable::Connection"</em> object will get called. In that method you'd usually want to figure out who opened the connection, and reject it if they aren't allowed to open it.</div><pre>#!/usr/bin/ruby
module ApplicationCable
class Connection < ActionCable::Connection::Base
attr_accessor :current_person
def connect
Rails.logger.debug "Someone wants to connect via WebSocket!"
set_current_person
unless current_person
Rails.logger.debug "I don't know who this is. Closing the WebSocket."
reject_unauthorized_connection
end
Rails.logger.debug "It's, #{current_person.name}"
end
private
def set_current_person
person =
# We can access the param of the HTTP request
Person.find_by(token: request.params[:token]) ||
# We can access the session of the HTTP request
Person.find_by(token: request.session[:person_id])
unless person
# We can access the cookies of the HTTP request
session = Session.find_by(id: cookies.encrypted[:session_id])
person = session&.person
end
self.current_person = person
end
end
end</pre><div>There is also a <a href="https://github.com/rails/rails/blob/cc633481602daf004eaab845c3780c6d885f882c/actioncable/lib/action_cable/connection/base.rb#L204"><em>"#disconnect"</em></a> method that gets called when a WebSocket is closed. It's useful if you want to set something up in <em>"#connect"</em> and then tear it down when the WebSocket closes. E.g. if you'd want to count the number of open connections that a person has, you could do something like this.</div><pre>#!/usr/bin/ruby
module ApplicationCable
class Connection < ActionCable::Connection::Base
attr_accessor :current_person
def connect
set_current_person
reject_unauthorized_connection unless current_person
current_person.increment!(:connection_count)
end
def disconnect
current_person&.decrement!(:connection_count)
end
private
# ...
end
end</pre><div>Now you know who the WebSocket connection is for, but Rails doesn't. To tell Rails which attr_accessor(s) we'll use for identification we have to change our <em>"attr_accessor"</em> to <a href="https://github.com/rails/rails/blob/cc633481602daf004eaab845c3780c6d885f882c/actioncable/lib/action_cable/connection/identification.rb#L20"><em>"identified_by"</em></a>, which is a class method that Action Cable provides.</div><pre>#!/usr/bin/ruby
module ApplicationCable
class Connection < ActionCable::Connection::Base
#attr_accessor :current_person # CHANGE THIS
identified_by :current_person # TO THIS
def connect
set_current_person
reject_unauthorized_connection unless current_person
end
private
# ...
end
end</pre><div>
<em>Ok, now we've got a connection and we know who it's for, how do we send a message?</em> Do we have to send some kind of JSON or XML or whatnot?</div><h2>The Wild WebSocket West</h2><div>We are used to HTTP in our Rails apps. With it we can transfer text, images, audio, video and more from our server to the browser. But WebSockets are the wild west of the Web.<br><br>
</div><div>As the "socket" part implies, a WebSocket is similar to a Unix, TCP or UDP socket in that you can transfer whatever you like over it and in any fashion you'd like. It's up to you to implement a protocol between your server and the browser on top of your WebSocket.<br><br>
</div><div>Luckily we don't have to implement anything as <strong>Action Cable is not only a WebSocket server but also a protocol.<br></strong><br>
</div><div>When you send a message to a client - or vice versa - the the object you are sending will first be turned into a JSON-compatible object like a String, a Number, a Hash, or an Array. Then it will be wrapped inside a Hash under a key called "message" together with some additional things I'll explain in a bit. Then that Hash will be dumped as JSON and sent as an UTF-8 encoded String, byte for byte through the WebSocket. On the other end the message is assembled into a String again and parsed into an object.<action-text-attachment sgid="eyJfcmFpbHMiOnsiZGF0YSI6ImdpZDovL2Jsb2cvQWN0aXZlU3RvcmFnZTo6QmxvYi81MDU_ZXhwaXJlc19pbiIsInB1ciI6ImF0dGFjaGFibGUifX0=--aa3e6b1f4a5ef77604f38cde33c898adbd9b154c" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NTA1LCJwdXIiOiJibG9iX2lkIn19--87d43f71fc7bff0b010a4a630da56a2ebb7a32b3/Pasted%20image%2020240124152110.png" filename="Pasted image 20240124152110.png" filesize="41574" width="775" height="232" previewable="true" presentation="gallery" caption="Raw message sent via ActionCable to the browser">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" style="aspect-ratio:775 / 232;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NTA1LCJwdXIiOiJibG9iX2lkIn19--87d43f71fc7bff0b010a4a630da56a2ebb7a32b3/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/Pasted%20image%2020240124152110.png">
<figcaption class="attachment__caption">
Raw message sent via ActionCable to the browser
</figcaption>
</figure></action-text-attachment><em>Why do we need the wrapper Hash?<br></em><br>
</div><div>The wrapper Hash allows us to have different types of messages. The protocol has three types of messages that a client can send to the server:</div><ul>
<li>subscribe</li>
<li>unsubscribe</li>
<li>message</li>
</ul><div>And six kinds of messages that the server can send to the client:</div><ul>
<li>welcome</li>
<li>disconnect</li>
<li>ping</li>
<li>confirm_subscription</li>
<li>reject_subscription</li>
<li>message</li>
</ul><div>
<a href="https://github.com/rails/rails/blob/cc633481602daf004eaab845c3780c6d885f882c/actioncable/lib/action_cable/connection/base.rb#L211"><strong>The "welcome" message</strong></a> is the simplest of them all. It's just a Hash with a key "type" that holds the value "welcome". All messages sent from the server to a client have a "type" field. This is always the first message that a server sends to any client that connects to it. It's used to indicate to the client that the WebSocket is functional.</div><pre>{
"type": "welcome"
}</pre><div>
<a href="https://github.com/rails/rails/blob/cc633481602daf004eaab845c3780c6d885f882c/actioncable/lib/action_cable/connection/base.rb#L135"><strong>The "ping" message</strong></a> is used to keep a heartbeat. It's sent to all connected clients every 3 seconds so that the connection doesn't become stale and therefore terminated by some load balance in the middle. And it enables clients to detect when they have disconnected from the server. If the client doesn't receive a heartbeat message within 6 seconds - that's 2 heartbeats - it will try to reconnect. In addition to the "type" field, a ping message will also have a "message" field which holds the Unix timestamp of the moment the message was sent. This timestamp isn't used in the official client implementation.</div><pre>{
"type": "ping",
"message": 1705848059
}</pre><div>
<a href="https://github.com/rails/rails/blob/cc633481602daf004eaab845c3780c6d885f882c/actioncable/lib/action_cable/connection/base.rb#L110"><strong>The "disconnect" message</strong></a> instructs a client to disconnect from the server. In addition to the "type" field it has a "reconnect" field which holds a Boolean. If the value of this field is true, the client should immediately try to reconnect after it disconnects. This is useful in case an error occurs and the connection has to be restarted. And in addition to the "reconnect" field there is a "reason" field which holds a String that gives you a brief explanation why your client is being disconnected.</div><div>There are four possible disconnect reasons:</div><ul>
<li>
<em>unauthorized</em> - sent when "reject_unauthorized_connection" is called</li>
<li>
<em>invalid_request</em> - sent when the initial HTTP request was malformed</li>
<li>
<em>server_restart</em> - sent when the server is about to restart</li>
<li>
<em>remote</em> - send when the client is kicked for whatever reason</li>
</ul><pre>{
"type": "disconnect",
"reconnect": true,
"reason": "server_restart"
}</pre><div>To explain the <em>"message"</em>, <em>"subscribe"</em>, <em>"confirm_subscription"</em>, <em>"reject_subscription"</em> & <em>"unsubscribe"</em> messages I first have to explain Channels.</div><h2>Channels</h2><div>
<em>What are Channels?<br></em><br>
</div><div>Channels are Action Cable's controllers.<br><br>
</div><div>In a regular HTTP request you'd specify which method you want to call on which Controller by setting the request's path and method. E.g. if you'd want to call the <em>"create"</em> method of the <em>"ArticlesController"</em> you'd make a POST request to "/articles" and pass any parameters you'd like the new Article object to have.</div><pre>#!/usr/bin/ruby
class ArticlesController
# POST /articles
def create
if Article.create(params.require(:article).permit(:title, :content))
redirect_to action: :show, status: :see_other
else
render :new, status: :unprocesssable_entity
end
end
end</pre><div>In Action Cable you specify which method you want to call on which Channel. To do that, you first have to subscribe to to a Channel and then you have to send a message to it.</div><pre>#!/usr/bin/ruby
class ChatChannel < ApplicationCable::Channel
def post_message(data)
Chat::Message.create(
content: data[:content],
poster: current_person
)
end
end</pre><div>(Note that we get access to <em>"current_person"</em> because we used <em>"identified_by"</em> in our Connection object)<br><br>
</div><div>Now if you want to invoke <em>"ChatChannel#post"</em> you first have to subscribe to the <em>"ChatChannel"</em> with a <em>"subscribe"</em> message.<br><br>
</div><div>
<a href="https://github.com/rails/rails/blob/c7551d08fc572655105da38394d8672b8259c22f/actioncable/app/javascript/action_cable/subscriptions.js#L88"><strong>The "subscribe" message</strong></a> has a <em>"command"</em> field instead of a <em>"type"</em> field. All messages sent from the client to the server have a <em>"command"</em> field. In addition to that it also has an <em>"identifier"</em> field which holds a Hash. The identifier Hash has at least a <em>"channel"</em> field which is the class name of the channel. But it can also have other, user defined, values which will become available in the Channel object through the <em>"params"</em> method.</div><pre>{
"command": "subscribe",
"identifier": {
"channel": "ChatChannel",
"room_name": "Ruby Zagreb"
}
}</pre><div>These params can be used however you'd like. E.g.</div><pre>#!/usr/bin/ruby
class ChatChannel < ApplicationCable::Channel
def post_message(data)
current_person
.chat_rooms
.find_by(name: params[:room_name]) # PARAMS FROM THE SUBSCIBE MESSAGE
&.messages
&.create(content: data[:content])
end
end</pre><div>When someone subscribes to a Channel the <em>"subscribe"</em> method on the channel will be called. In it you can do various things, one of which is to authorize the subscription. Let's say that you want to reject subscriptions for chat rooms that a person isn't a member of, you could do something like this</div><pre>#!/usr/bin/ruby
class ChatChannel < ApplicationCable::Channel
def subscribed
@chat_room = current_person
.chat_rooms
.find_by(name: params[:room_name])
reject unless @chat_room
end
def post_message(data)
@chat_room.messages.create(content: data[:content])
end
end</pre><div>If the <em>"subscribe"</em> method doesn't call <em>"reject"</em> in some cases the server will respond with a <em>"confirm_subscription"</em> message.<br><br>
</div><div>
<a href="https://github.com/rails/rails/blob/cc633481602daf004eaab845c3780c6d885f882c/actioncable/lib/action_cable/channel/base.rb#L301"><strong>The </strong><strong><em>"confirm_subscription"</em></strong><strong> message</strong></a> has a <em>"type"</em>, and an <em>"identifier"</em> field. The identifier holds the same value that was sent in the original <em>"subscribe"</em> message.</div><pre>{
"type": "confirm_subscription",
"identifier": {
"channel": "ChatChannel",
"room_name": "Ruby Zagreb"
}
}</pre><div>If <em>"reject"</em> was called then the server will respond with a <em>"reject_subscription"</em> message.<br><br>
</div><div>
<a href="https://github.com/rails/rails/blob/main/actioncable/lib/action_cable/channel/base.rb#L316"><strong>The "reject_subscription" message</strong></a> has the same format as the "confirm_subscription" message.</div><pre>{
"type": "reject_subscription",
"identifier": {
"channel": "ChatChannel",
"room_name": "Ruby Zagreb"
}
}</pre><div>Now that the client is subscribed they can invoke an action on the Channel using the regular message type.<br><br>
</div><div>
<strong>The "message" type messages</strong> has two variants - one for messages sent from the server to the client, and another for messages sent from the client to the server.<br><br>
</div><div>
<a href="https://github.com/rails/rails/blob/c7551d08fc572655105da38394d8672b8259c22f/actioncable/app/javascript/action_cable/subscription.js#L83"><strong>If the client is sending the message</strong></a>, then the message will contain a <em>"command"</em> field with the value "message", an <em>"identifier"</em> and a <em>"data"</em> field. Again, the identifier holds the same value that was sent in the original <em>"subscribe"</em> message. While data can be anything depending on what the client has sent.</div><pre>{
"command": "message",
"identifier": {
"channel": "ChatChannel",
"room_name": "Ruby Zagreb"
},
"data": "Hello, Zagreb!"
}</pre><div>
<a href="https://github.com/rails/rails/blob/cc633481602daf004eaab845c3780c6d885f882c/actioncable/lib/action_cable/channel/base.rb#L167C11-L167C25">When the channel receives a message</a>, it will check if it implements a <em>"receive"</em> method and if it does it will invoke it and pass the message's data to it.<br><br>
</div><div>Though, if the client sent over a Hash, then the Channel will check if the data Hash contains the key "action" with a String value. If it does, and the Channel implements a method with the same name, then it will be invoked to process the message's data.</div><pre>{
"command": "message",
"identifier": {
"channel": "ChatChannel",
"room_name": "Ruby Zagreb"
},
"data": {
"action": "post_message",
"content": "Hello, Zagreb!"
}
}</pre><div>I'll explain the server-to-client message format in a moment.<br><br>
</div><div>
<em>Now we know how to send messages from the client to Action Cable. But how do we send messages from Action Cable to the client?<br></em><br>
</div><div>This is in my opinion the more interesting part. To send something we first have to create a stream to which we can publish messages to.<br><br>
</div><div>A stream is a PubSub channel. I <em>guess</em> it's called a stream in Action Cable because, semantically, this PubSub channel acts like a stream of messages for the client.</div><div>
<em>What is a PubSub channel?<br></em><br>
</div><div>
<a href="https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern">PubSub (short for Publish-subscribe)</a> is a common programming pattern for sending messages between objects. An object can subscribe to messages from, or publish messages to, a channel (sometimes called a topic). When one object publishes a message to a channel, all objects subscribed to that channel receive it. The message can be anything - a String, a Hash, a Number, or another Object.</div><div>
<br>In Action Cable, our Channel object is the subscriber and our application (controllers, models, jobs, ...) is the publisher.<action-text-attachment sgid="eyJfcmFpbHMiOnsiZGF0YSI6ImdpZDovL2Jsb2cvQWN0aXZlU3RvcmFnZTo6QmxvYi81MDY_ZXhwaXJlc19pbiIsInB1ciI6ImF0dGFjaGFibGUifX0=--6cf2b66f2101c96bc30b45ad50dd62dbc7bfa635" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NTA2LCJwdXIiOiJibG9iX2lkIn19--be1246b9ba0d759098a571da395185b64312dc89/Pasted%20image%2020240128081709.png" filename="Pasted image 20240128081709.png" filesize="53140" width="979" height="274" previewable="true" presentation="gallery" caption="Illustration of how PubSub works">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" style="aspect-ratio:979 / 274;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NTA2LCJwdXIiOiJibG9iX2lkIn19--be1246b9ba0d759098a571da395185b64312dc89/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/Pasted%20image%2020240128081709.png">
<figcaption class="attachment__caption">
Illustration of how PubSub works
</figcaption>
</figure></action-text-attachment>To create a stream we can use either <a href="https://github.com/rails/rails/blob/cc633481602daf004eaab845c3780c6d885f882c/actioncable/lib/action_cable/channel/streams.rb#L78"><em>"stream_from"</em></a> which creates a PubSub channel using a String identifier that we pass to it <em>(e.g. "chat:1337")</em>. Or <a href="https://github.com/rails/rails/blob/cc633481602daf004eaab845c3780c6d885f882c/actioncable/lib/action_cable/channel/streams.rb#L103"><em>“stream_for”</em></a> which accepts an object <em>(e.g. "Chat.find(1337)")</em>. <em>"stream_for"</em> internally generates a String identifier for the given object and calls <em>"stream_from"</em> with it.</div><pre>#!/usr/bin/ruby
class ChatChannel < ApplicationCable::Channel
def subscribed
@chat_room = current_person
.chat_rooms
.find_by(name: params[:room_name])
reject unless @chat_room
# creates a PubSub channel to which we can publish messages to from anywhere
stream_for @chat_room
# the above is basicaly the same as
# stream_from "chat_room:#{@chat_room.id}"
end
end</pre><div>Remember how I earlier said that the server will confirm a subscription only in some cases? <strong>Well, it will only confirm subscriptions which open a stream.</strong> If a subscription doesn't open a stream the server just won't respond to a subscribe message.<br><br>
</div><div>Now that we have a stream we can publish messages to it from anywhere using <em>"broadcast_to"</em> which accepts two arguments - the object we are broadcasting to, and the messages to broadcast.</div><pre>#!/usr/bin/ruby
ChatChannel.broacast_to(
ChatRoom.find(params[:id]), # The same record that we gave to stream_for
"Hello, Zagreb!" # The message I want to send
)</pre><div>If you used <em>"stream_from"</em> to create your stream, then you have to use <em>"ActionCable.server.broadcast"</em> instead of <em>"broadcast_to"</em>. It also expects two arguments - the stream we are broadcasting to, and the message we are broadcasting.</div><pre>#!/usr/bin/ruby
ActionCable.server.broadcast("chat_room:123", "Hello, Zagreb!")</pre><div>You can also send a message directly from the Channel object using the <a href="https://github.com/rails/rails/blob/3d132b855541c35236e93b3a676a3ee80d02a38a/actioncable/lib/action_cable/channel/base.rb#L214"><em>"transmit"</em> method</a>.</div><pre>#!/usr/bin/ruby
class ChatChannel < ApplicationCable::Channel
def subscribed
@chat_room = current_person
.chat_rooms
.find_by(name: params[:room_name])
reject unless @chat_room
transmit "Welcome back, #{current_person.first_name}!"
end
end</pre><div>
<em>Ok, but what happens when we broadcast or transmit a message?<br></em><br>
</div><div>Well, <a href="https://github.com/rails/rails/blob/cc633481602daf004eaab845c3780c6d885f882c/actioncable/lib/action_cable/channel/base.rb#L223">it's wrapped in a hash containing two keys</a> - <em>"identifier"</em> and <em>"message"</em>. The identifier holds the same value that was sent in the original <em>"subscribe"</em> message. While the message holds whatever you are sending to the client.</div><pre>{
"identifier": {
"channel": "ChatChannel",
"room_name": "Ruby Zagreb"
},
"message": "Hello, Zagreb!"
}</pre><div>Then that Hash is turned into JSON and sent via the WebSocket.<br><br>
</div><div>When the client gets that message, it can figure out for which of its subscriptions it's for based on the <em>"identifier"</em>, and then it can process the <em>"message"</em> however it likes.</div><h2>In the browser</h2><div>
<em>Now we have sent a message from the server, but how can we respond to it in a browser?<br></em><br>
</div><div>
<a href="https://www.npmjs.com/package/@rails/actioncable">Action Cable ships with an official JavaScript client</a> that enables you to connect to a server via Action Cable, subscribe to any channel, and receive messages. Just like its server counterpart, it hides the protocol from you and allows you to focus only on the messages.<br><br>
</div><div>To connect to a server you have to create a <em>consumer</em>, which is a wrapper around a WebSocket connection to our server.<br><br>
</div><div>You can create a consumer that will connect to "/cable" without any extra params just by calling <em>"createConsumer"</em>.</div><pre>#!/usr/bin/node
import { createConsumer } from "@rails/actioncable"
const consumer = createConsumer()</pre><div>If you want to pass extra params to the server's Connection object, e.g. for authentication, you'll have to pass the exact URL with params and all to <em>"createConsumer"</em>.</div><pre>#!/usr/bin/node
import { createConsumer } from "@rails/actioncable"
// Fetches the user's auth token from a meta tag in the DOM
const token = document.querySelector("meta[name=authe_token]")?.content
// Takes the current URL,
// changes it's path to "/ws",
// and adds "?token=#{token}" as the query params.
// The result looks something like "https://example.com/ws?token=123"
const webSocketURL = new URL(window.location.href)
webSocketURL.pathname = "/ws" // usually it's "/cable"
webSocketURL.search = `?token=${token}`
const consumer = createConsumer(webSocketURL.toString())</pre><div>Now that you have a consumer, you can subscribe with it to any channel you'd like. To subscribe, you have to create a subscription with an identifier.<br><br>
</div><div>If you remember from before, an identifier has to have a <em>"channel"</em> field, but can also have any additional fields you want and you'll have access to these fields as <em>"params"</em> in your Channel object.</div><pre>#!/usr/bin/node
import { createConsumer } from "@rails/actioncable"
const consumer = createConsumer()
// Subscribe to the ChatChannel
// and pass { room_name: "Ruby Zagreb" } as params
consumer.subscriptions.create(
{ channel: "ChatChannel", room_name: "Ruby Zagreb" }
)
<br></pre><div>
<em>How do I process an incoming message with this?<br></em><br>
</div><div>Well, <a href="https://github.com/rails/rails/blob/c7551d08fc572655105da38394d8672b8259c22f/actioncable/app/javascript/action_cable/connection.js#L154">when your subscriptions receives a message it will try to call a <em>"received"</em> method of the subscription object</a> to process the message.<br><br>
</div><div>You can create a received method on the subscription yourself, like so</div><pre>#!/usr/bin/node
import { createConsumer } from "@rails/actioncable"
const consumer = createConsumer()
// Subscribe to the ChatChannel
// and pass { room_name: "Ruby Zagreb" } as params
const subscription = consumer.subscriptions.create(
{ channel: "ChatChannel", room_name: "Ruby Zagreb" }
)
subscription.received = function(message) {
document
.querySelector("#messages")
?.insertAdjacentHTML("beforeend", `<div>${message}</div>`)
}</pre><div>Though this approach has a problem - if you get a message immediately as you subscribe, but before you create your received method, you'll miss that message.<br><br>
</div><div>Action Cable allows you to extend the Subscription object as you create it to avoid this problem. You can pass a second argument while creating a subscription. That argument has to be an object with which the newly created Subscription object will be extended. Extending an object in JavaScript is similar to including a mixin in a class in Ruby.</div><pre>#!/usr/bin/node
import { createConsumer } from "@rails/actioncable"
const consumer = createConsumer()
// Subscribe to the ChatChannel
// and pass { room_name: "Ruby Zagreb" } as params
const subscription = consumer.subscriptions.create(
{ channel: "ChatChannel", room_name: "Ruby Zagreb" },
{
received(message) {
this.appendMessage(message)
}
appendMessage(message) {
this.messageContainer()?.insertAdjacentHTML(
"beforeend",
`<div>${message}</div>`
)
}
messageContainer() {
document.querySelector("#messages")
}
}
)</pre><div>But there are other events that you can process in your Subscription object besides receiving a message.<br><br>
</div><div>You can process <a href="https://github.com/rails/rails/blob/c7551d08fc572655105da38394d8672b8259c22f/actioncable/app/javascript/action_cable/subscriptions.js#L34">initialization events</a>, <a href="https://github.com/rails/rails/blob/c7551d08fc572655105da38394d8672b8259c22f/actioncable/app/javascript/action_cable/connection.js#L147">connect</a> and <a href="https://github.com/rails/rails/blob/c7551d08fc572655105da38394d8672b8259c22f/actioncable/app/javascript/action_cable/connection.js#L172">disconnect</a> events, as well as subscription <a href="https://github.com/rails/rails/blob/c7551d08fc572655105da38394d8672b8259c22f/actioncable/app/javascript/action_cable/subscriptions.js#L50">rejections</a>. Each event requires you to define a method to process it. Initialization requires an <em>"initialized"</em> method, rejection a <em>"rejected"</em> method, connect and disconnect a <em>"connected"</em> and <em>"disconnected"</em> method.</div><pre>#!/usr/bin/node
import { createConsumer } from "@rails/actioncable"
const consumer = createConsumer()
const subscription = consumer.subscriptions.create(
{ channel: "ChatChannel", room_name: "Ruby Zagreb" },
{
// Called right after the Subscription object is created
initialized() {
console.log(`A subscription to ${this.identifier} was created`)
}
// Called if the subscription was rejected by the server
rejected() {
console.log(`The server rejected the subscription to ${this.identifier}`)
}
// Called when the subscription gets confirmed by the server
connected(data) {
// The data object has a single property `reconnected`
// which indicates if this was a resubscribe after a disconnect
console.log(`The server confirmed the subscription to ${this.identifier}! Reconnected: ${data.reconnected}`)
}
// Called when the WebSocket closes
disconnected(data) {
// The data object has a single propert `willAttemptReconnect`
// which indicates if a reconnect attempt will be made or not
console.log(`WebSocket to ${this.consumer.url} closed! Will attempt reconnect: ${data.willAttemptReconnect}`)
}
// Called when a message is sent from the server
received(message) {
console.log(`Received message: ${message}`)
}
}
)</pre><div>There are also two actions that you can perform from your subscription - send messages to the server, and unsubscribe.<br><br>
</div><div>To unsubscribe just call <a href="https://github.com/rails/rails/blob/c7551d08fc572655105da38394d8672b8259c22f/actioncable/app/javascript/action_cable/subscription.js#L86"><em>"unsubscribe"</em></a>.</div><pre>#!/usr/bin/node
import { createConsumer } from "@rails/actioncable"
const consumer = createConsumer()
const subscription = consumer.subscriptions.create(
{ channel: "ChatChannel", room_name: "Ruby Zagreb" },
{
received(message) {
console.log(`Received message: ${message}`)
if (message.toLowerCase() === "avada kedavra") this.unsubscribe()
}
}
)
<br></pre><div>And to send a message call the <a href="https://github.com/rails/rails/blob/c7551d08fc572655105da38394d8672b8259c22f/actioncable/app/javascript/action_cable/subscription.js#L82"><em>"send"</em> method</a> with the message you want to send.</div><pre>#!/usr/bin/node
import { createConsumer } from "@rails/actioncable"
const consumer = createConsumer()
const subscription = consumer.subscriptions.create(
{ channel: "ChatChannel", room_name: "Ruby Zagreb" }
)
subscription.send({ action: "post_message", content: "Hello, Zagreb!" })
<br></pre><h2>Thousands of connections, one server</h2><div>Now you know how to connect to Action Cable and send messages back and forth. But if you have ever tuned a Rails application you probably know that there is a <a href="https://en.wikipedia.org/wiki/Thread_pool">thread pool</a> that determines how many simulations requests your server can process.<br><br>
</div><div>
<em>Does that mean that there is a maximum number of connections that Action Cable can handle?<br></em><br>
</div><div>Well, no. As long as your server has memory available it will be able to accept and serve more Action Cable connections. Granted, it will process them slower and slower, but it will work.<br><br>
</div><div>
<em>How does that work?<br></em><br>
</div><div>
<a href="https://github.com/puma/puma">Puma</a>, currently the most popular Ruby/Rack application server, is a multi-threaded server. This means that when you make a request to it, your request will be assigned to one thread that will process it and generate a response.</div><div>But, you don't want to have too many threads. Each thread running on your server gets a very short slice of time to run its code on the processor. The more threads you have, the more any one thread has to wait to get to run its code on the processor.<br><br>
</div><div>So the smarter thing to do is to have a preset number of threads - a thread pool - that can process requests. If a request comes in, and all threads in the pool are busy, the request goes into a queue. It will wait there until a thread becomes available. That's why you want to take as little time as possible to generate a response for a request. The more a request takes to process, the higher the chance that some requests will end up in the queue which means that your app feels slow.<br><br>
</div><div>
<em>But Action Cable turns requests into WebSockets, which can stay open for days, how don't we run out of threads in the pool?<br></em><br>
</div><div>That's the interesting part. <strong>Action Cable processes only the initial HTTP request in Puma's thread pool.</strong> After you get a 101 response from the server the request is "hijacked" and put into an event loop. <strong>So Puma's thread pool determines the maximum number of WebSocket connections that can be opened at once.<br></strong><br>
</div><div>
<em>Hijack? Event loop? What?<br></em><br>
</div><div>Rails doesn't integrate directly with an application server. Instead it implements Rack's protocol through which it gets requests from, and returns responses to, the application server. This enables applications and frameworks like Rails to work with any application server like Puma, <a href="https://github.com/ruby/webrick">WebBrick</a> or <a href="https://github.com/socketry/falcon">Falcon</a>.<br><br>
</div><div>Rack is great for request-response interactions like HTTP, but a WebSocket isn't request-response like. It's a bi-directional stream of data.</div><div>To support WebSockets and similar stream-like protocols, Rack offers a way to read and write bytes directly to a connection initiated by a request - it's called hijacking the socket.<br><br>
</div><div>When you hijack a Rack request you get it's underlying socket. You can read and write bytes to the sockets freely. But this means that it's your responsibility to implement whatever protocol you want over that socket.<br><br>
</div><div>
<a href="https://github.com/rails/rails/blob/main/actioncable/lib/action_cable/connection/stream.rb#L96">Action Cable hijacks the socket</a> and passes it to <a href="https://github.com/faye/websocket-driver-ruby/tree/main">Faye's WebSocket Driver</a> library which implements the WebSocket protocol and it puts that socket into an <a href="https://en.wikipedia.org/wiki/Event_loop">event loop</a>.<br><br>
</div><div>An event loop is a pattern for responding to events. You start an infinite loop in which you wait for some event, when it occurs you process it and the loop starts again.<br><br>
</div><div>In Action Cable's case it's waiting for bytes to read from the socket it hijacked. When they are available, it reads them and passes them on to the WebSocket driver.<br><br>
</div><div>That's easy enough for one socket. But to wait for any of an infinite number of sockets to have some bytes ready to be read would require an infinite number of threads - one for each socket.<br><br>
</div><div>To avoid spawning a thread for each WebSocket, Action Cable uses <a href="https://github.com/socketry/nio4r">nio4r</a> <a href="https://github.com/rails/rails/blob/main/actioncable/lib/action_cable/connection/stream_event_loop.rb#L95">which notifies it when a socket has some bytes ready to be read</a>, without spawning any threads. It does so using different functions of the server's operating system's kernel - such as <a href="https://ruby-doc.org/3.2.2/IO.html#method-c-select">select</a>, <a href="https://en.wikipedia.org/wiki/Epoll">epoll</a> and <a href="https://en.wikipedia.org/wiki/Kqueue">kqueue</a>.<br><br>
</div><div>Once it's notified that <a href="https://github.com/rails/rails/blob/3d132b855541c35236e93b3a676a3ee80d02a38a/actioncable/lib/action_cable/connection/stream_event_loop.rb#L95">a socket is ready</a>, <a href="https://github.com/rails/rails/blob/3d132b855541c35236e93b3a676a3ee80d02a38a/actioncable/lib/action_cable/connection/stream_event_loop.rb#L109">it reads its bytes</a>, parses them into a message, <a href="https://github.com/rails/rails/blob/3d132b855541c35236e93b3a676a3ee80d02a38a/actioncable/lib/action_cable/connection/stream_event_loop.rb#L116">and then passes the message to the Connection object</a>, <a href="https://github.com/rails/rails/blob/3d132b855541c35236e93b3a676a3ee80d02a38a/actioncable/lib/action_cable/connection/base.rb#L87">which then puts in in a thread pool to be processed</a> - <a href="https://github.com/rails/rails/blob/3d132b855541c35236e93b3a676a3ee80d02a38a/actioncable/lib/action_cable/server/worker.rb#L10">called the worker pool</a> - to process that message.<action-text-attachment sgid="eyJfcmFpbHMiOnsiZGF0YSI6ImdpZDovL2Jsb2cvQWN0aXZlU3RvcmFnZTo6QmxvYi81MDg_ZXhwaXJlc19pbiIsInB1ciI6ImF0dGFjaGFibGUifX0=--c9adff6975a06248a19728e17cdd4102dfe121b8" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NTA4LCJwdXIiOiJibG9iX2lkIn19--6a3d1d71c954fe9efc0e6d79bf9c4ee9aa432aba/Pasted%20image%2020240204081845.png" filename="Pasted image 20240204081845.png" filesize="51350" width="1107" height="256" previewable="true" presentation="gallery" caption="Illustration of all the pools that a request goes through">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" style="aspect-ratio:1107 / 256;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NTA4LCJwdXIiOiJibG9iX2lkIn19--6a3d1d71c954fe9efc0e6d79bf9c4ee9aa432aba/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/Pasted%20image%2020240204081845.png">
<figcaption class="attachment__caption">
Illustration of all the pools that a request goes through
</figcaption>
</figure></action-text-attachment><em>Another thread pool?<br></em><br>
</div><div>Yes. <strong>This worker pool allows Action Cable to process multiple incoming messages simultaneously.<br></strong><br>
</div><div>You can tweak the size of this worker pool by setting <em>"config.action_cable.worker_pool_size"</em> to the number of threads you'd like to have in that pool.<br><br>
</div><div>
<strong>The more threads in this pool the higher the number of incoming messages that you can process simultaneously</strong>. More threads also means that your latency will go up (as one threads will have to wait for longer to get access to the processor), your database connection count will go up (as more threads can access the database simultaneously), and your memory consumption will go up as more messages and objects are in memory at the same time.<br><br>
</div><div>But there is one more thread pool in Action Cable - the event loop thread pool.<br><br>
</div><div>This one is used to dispatch events like <a href="https://github.com/rails/rails/blob/3d132b855541c35236e93b3a676a3ee80d02a38a/actioncable/lib/action_cable/server/connections.rb#L29">sending heart beat messages</a>, <a href="https://github.com/rails/rails/blob/3d132b855541c35236e93b3a676a3ee80d02a38a/actioncable/lib/action_cable/connection/internal_channel.rb#L22">subscribing to</a> and <a href="https://github.com/rails/rails/blob/3d132b855541c35236e93b3a676a3ee80d02a38a/actioncable/lib/action_cable/connection/internal_channel.rb#L29">unsubscribing from</a> PubSub channels created by stream_from, <a href="https://github.com/rails/rails/blob/3d132b855541c35236e93b3a676a3ee80d02a38a/actioncable/lib/action_cable/connection/stream.rb#L104">attaching</a> and <a href="https://github.com/rails/rails/blob/3d132b855541c35236e93b3a676a3ee80d02a38a/actioncable/lib/action_cable/connection/stream.rb#L110">detaching</a> hijacked sockets, and <a href="https://github.com/rails/rails/blob/3d132b855541c35236e93b3a676a3ee80d02a38a/actioncable/lib/action_cable/channel/periodic_timers.rb#L67">triggering periodic timers</a> (to which I'll get to in just a bit).<br><br>
</div><div>This thread pool has a fixed size as it's intended to be an event dispatch of sorts where you just schedule a task which the pool only trigger, the execution of the task is mostly done in the worker pool.</div><h2>Things I wish I knew right away</h2><div>There are a few things that I learned about Action Cable that I wish I knew earlier.<br><br>
</div><div>
<strong>You can trigger actions and send messages periodically</strong>, like every few seconds. This is extremely useful if you have some state that you want to synchronize periodically, or some hose keeping you want to do.<br><br>
</div><div>In your Channel object you can call a <a href="https://github.com/rails/rails/blob/main/actioncable/lib/action_cable/channel/periodic_timers.rb#L31"><em>"periodically"</em> class method</a>, give it a method name or a block, and an interval. It will then run that method/block for you with whatever interval you specified.</div><pre>#!/usr/bin/ruby
class StockTickerChannel < ApplicationCable::Channel
# this will send a message to the client every 2 seconds
# as long as they are subscribed
periodically every: 2.seconds do
transmit value: @stock.value, timestamp: Time.now.to_i
end
def subscribed
@stock = Stock.find_by(symbol: params[:symbol])
reject unless @stock
end
end</pre><div>
<strong>Some error trackers won't catch errors from your Channel objects unless you explicitly send them.</strong> You can do that in your Connection using <em>"rescue_from"</em>, just like you would in a Controller.</div><pre>#!/usr/bin/ruby
module ApplicationCable
class Connection < ActionCable::Connection::Base
rescue_from Exception do |error|
MyErrorTracker.capture_exception(error)
end
# ...
end
end</pre><div>
<strong>There is a callback on the Connection object for when an action gets invoked on your Channel object.</strong> You can register such a callback using <em>"before_command"</em>, <em>"after_command"</em> and "around_command", which is extremely useful if you use <a href="https://api.rubyonrails.org/classes/ActiveSupport/CurrentAttributes.html">Current attributes</a>.</div><pre>#!/usr/bin/ruby
module ApplicationCable
class Connection < ActionCable::Connection::Base
around_command do
Current.set(person: current_person) { yield }
end
# ...
end
end</pre><div>
<strong>There are callbacks on the Channel object for when someone subscribes or unsubscribes.</strong> There is "<em>before_subscribe</em>", <em>"after_subscribe"</em>, <em>"around_subscribe"</em>, and "<em>before_unsubscribe</em>", <em>"after_unsubscribe"</em>, <em>"around_unsubscribe"</em>. These is useful for implementing common behavior through inheritance or mixins without having to call "super" form the <em>"subscribe"</em> or <em>"unsubscribe"</em> method.</div><pre>#!/usr/bin/ruby
module AppearanceTrackable
extend ActiveSupport::Concern
included do
after_subscribe unless: :subscription_rejected? do
Current.person.came_online!
end
after_unsubscribe do
Current.person.went_offline!
end
end
end
class ChatChannel < ApplicationCable::Channel
include AppearanceTrackable
def subscribed
@chat_room = current_person
.chat_rooms
.find_by(name: params[:room_name])
reject unless @chat_room
stream_for @chat_room
end
end</pre><div>
<strong>You probably want to bump Puma's worker timeout if you are debugging your Connection object.</strong> Puma will kill any worker that doesn't generate a response within 60 seconds.<br><br>
</div><div>This can be annoying if you are trying to debug something like how you authenticate Connections with <em>"debugger"</em>, <em>"binding.irb"</em> or <em>"binding.pry"</em>.<br><br>
</div><div>But you can raise that timeout using the <em>"worker_timeout"</em> method in <em>"puma.rb"</em>.</div><pre>#!/usr/bin/ruby
# config/puma.rb
require File.expand_path("../config/environment", File.dirname(__FILE__))
# ...
# Kill a worker thread if it didn't generate a responsw in 8 hours
worker_timeout 8 * 3600 if ENV.fetch("RAILS_ENV", "development") == "development"
<br></pre><div>
<strong>You have to tweak both Puma's thread and Action Cable's worker counts.</strong> Puma's thread count controls how many new WebSocket connections can be created simultaneously. Action Cable's worker pool size determines how many WebSocket messages you can process simultaneously. Both pools will create database connections!</div><pre>#!/usr/bin/ruby
# config/puma.rb
max_threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
min_threads_count = ENV.fetch("RAILS_MIN_THREADS") { max_threads_count }
threads min_threads_count, max_threads_count
# config/application.rb
config.action_cable.worker_pool_size = ENV.fetch("RAILS_MAX_THREADS") { 5 }</pre><div>
<strong>In development, sometimes the server can behave a bit wonky.</strong> I'm not a 100% sure what's going on but it seems like code reloading can cause rogue threads with stale code. If that happens just run "touch tmp/restart.txt" to quickly restart the server and reset everything.<br><br>
</div><div>
<strong>When a client looses it's Internet connection it will take Action Cable up to 20 min to notice that, close the connection, and trigger callbacks.</strong> I wrote about that in my <a href="https://stanko.io/tracking-online-presence-with-actioncable-ukEi6sUZ98Bz">previous article</a>. This can cause you headaches if you want to allow only a certain number of connections per client.<br><br>
</div><div>
<strong>You can use the official JS client outside of the browser, like in Node or Bun.</strong> The official client doesn't use <a href="https://developer.mozilla.org/en-US/docs/Web/API/WebSocket">the browser's WebSocket object</a> directly. It exports an <em>"adapters"</em> object which has a property called <em>"WebSocket"</em> which holds the object that will be used to establish WebSocket connections. So you can drop-in your own WebSocket object and use it.<br><br>
</div><div>
<strong>You can remotely disconnect any client.</strong> There is a <em>"ActionCable::RemoteConnections"</em> class that acts like an Active Record relation. You can query it using its <a href="https://github.com/rails/rails/blob/main/actioncable/lib/action_cable/remote_connections.rb#L36"><em>"where"</em> method</a> to get a connection. The where method searches for a connection by its <em>"identified_by"</em> fields. If it finds such a connection it gives you a proxy for that connection on which you can only call <a href="https://github.com/rails/rails/blob/main/actioncable/lib/action_cable/remote_connections.rb#L56"><em>"disconnect"</em></a> which then disconnects the client.</div><pre>#!/usr/bin/ruby
class Person < Applicationrecord
def ban!
update!(banned: true)
ActionCable::RemoteConnections
.where(current_person: self)
&.disconnect(reconnect: false)
end
end</pre><div>
<strong>There are multiple PubSub adapters available besides Redis.</strong> You can use Postgres instead of Redis as a backend (<a href="https://www.postgresql.org/docs/current/sql-notify.html">it has some limitations</a>). And you can provide your own if you need.</div><pre># config/cable.yml
production:
adapter: postgres</pre><h2>Final thought</h2><div>Action Cable is an amazing piece of software. It can seem complex, but that’s because it does a lot. When you look at each piece individually the complexity goes away.<action-text-attachment sgid="eyJfcmFpbHMiOnsiZGF0YSI6ImdpZDovL2Jsb2cvQWN0aXZlU3RvcmFnZTo6QmxvYi81MDk_ZXhwaXJlc19pbiIsInB1ciI6ImF0dGFjaGFibGUifX0=--cb39a5efd14a34a59a6ac8a168fc0e1e5d6445bc" content-type="image/png" url="https://stanko.io/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NTA5LCJwdXIiOiJibG9iX2lkIn19--c28b2eb369c4519bff9d60ec5630c91618771ac4/Pasted%20image%2020240205082517.png" filename="Pasted image 20240205082517.png" filesize="63541" width="537" height="955" previewable="true" presentation="gallery" caption="Block overview of everything discussed">
<figure class="attachment attachment--preview attachment--png">
<img loading="lazy" style="aspect-ratio:537 / 955;" src="https://stanko.io/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsiZGF0YSI6NTA5LCJwdXIiOiJibG9iX2lkIn19--c28b2eb369c4519bff9d60ec5630c91618771ac4/eyJfcmFpbHMiOnsiZGF0YSI6eyJmb3JtYXQiOiJwbmciLCJyZXNpemVfdG9fbGltaXQiOlsxMDI0LDc2OF19LCJwdXIiOiJ2YXJpYXRpb24ifX0=--567b63b27449fb73342219aac1215cfd3a1b572d/Pasted%20image%2020240205082517.png">
<figcaption class="attachment__caption">
Block overview of everything discussed
</figcaption>
</figure></action-text-attachment>
</div>
</div>
Stanko Krtalic Rusendichey@stanko.iotag:stanko.io,2005:Article/372024-02-06T07:10:00Z2024-02-06T12:01:16ZRunning Campfire behind traefik<div class="trix-content">
<div>I rent a bare metal server from <a href="https://www.scaleway.com/en/">Scaleway</a> where I host all my apps. For me, that's much more economical than running multiple small VPS boxes. 50€ a month buys me 2TB of disk space, 64GB of RAM and an 4 core / 8 thread CPU on which I can host two or three dozen apps. Compare that to AWS or DigitalOcean where for the same money I get about one quarter of those resources.<br><br>
</div><div>All my apps are deployed as Docker containers, most of them using <a href="https://kamal-deploy.org/">Kamal</a>, in front of which I put <a href="https://traefik.io/traefik/">traefik</a> as a reverse proxy to direct traffic and provide HTTPS via <a href="https://letsencrypt.org/">Let's Encrypt</a>.<br><br>
</div><div>Last weekend I wanted to setup a Campfire instance for Ruby Zagreb - my local Ruby user-group - on that machine.<br><br>
</div><div>So I ran the official installer and got an error.</div><pre><strong>#!/usr/bin/bash</strong>
sudo once setup 1111-1111-1111-1111
██████╗ ███╗ ██╗ ██████╗███████╗
██╔═══██╗████╗ ██║██╔════╝██╔════╝
██║ ██║██╔██╗ ██║██║ █████╗
██║ ██║██║╚██╗██║██║ ██╔══╝
╚██████╔╝██║ ╚████║╚██████╗███████╗
╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝
Failed to start the software
The software couldn't be started. Please try again. </pre><div>I expected the installer to fail since it can't know my setup, and it didn't ask me anything besides the domain of the instance.<br><br>
</div><div>Sadly, the email you get with your purchase doesn't explain what the installer does, and I can't check what it's doing since the installer is a binary blob. So I had to snoop around for a bit.<br><br>
</div><div>The first thing I found was the installer's error log which said that the container couldn't bind to port 443.</div><pre><strong>#!/usr/bin/bash</strong>
cat .once-error.log
2024/01/27 19:33:11 Error response from daemon: driver failed programming external connectivity on endpoint campfire (e32f1a334f8ac25b9f3df0c283adb3eb3e8a554074f0827d17f8b566d03940a5): Bind for 0.0.0.0:443 failed: port is already allocated</pre><div>Since traefik is bound to port 80 and 443 this makes perfect sense.<br><br>
</div><div>The error looks like it came from Docker so I checked my images, and sure enough there was a new <em>"registry.once.com/campfire:latest"</em> image and a stopped container which I promptly inspected.<br><br>
</div><div>Within the container runs a custom proxy called <a href="https://github.com/rails/rails/issues/50479">thruster</a> that terminates SSL and handles the <a href="https://docs.nginx.com/nginx/admin-guide/web-server/serving-static-content/#enabling-sendfile">sendfile protocol</a>. That's what's bound to port 443 and 80 of the container.<br><br>
</div><div>Inside the container is also a Redis instance, which implies that the installer sets at least <em>"sysctl vm.overcommit_memory=1"</em> else Redis wouldn't start.<br><br>
</div><div>I also saw that there were ENV variables injected into the container so I looked where those came from and found a config file in <em>"/root/.config/once/config.json"</em>.</div><pre><strong>#!/usr/bin/bash</strong>
sudo cat /root/.config/once/config.json | jq
{
"token": "1111-1111-1111-1111",
"product": "campfire",
"product_name": "Campfire",
"email_address": "phony.email@example.com",
"ssl_domain": "chat.rubyzg.org",
"validation_token": "PHONY_VALIDATION_KEY",
"secret_key_base": "PHONY_SECRET_KEY_BASE",
"vapid_private_key": "PHONY_PRIVATE_KEY",
"vapid_public_key": "PHONY_PUBLIC_KEY",
"storage_location": "/var/once/campfire",
"cron_hour": 2,
"once_binary_etag": ""
}</pre><div>Seems pretty straight forward. This is roughly what the installer does</div><pre><strong>#!/usr/bin/bash</strong>
sudo sysctl vm.overcommit_memory=1
docker run \
-d \
--name campfire \
--restart unless-stopped \
--env-file campfire/.env \
-p 443
-p 80
-v campfire/storage:/rails/storage \
registry.once.com/campfire:latest \
bin/boot;;
<br></pre><div>Where <em>"campfire/.env"</em> contains the values from <em>"/root/.config/once/config.json"</em> plus a few extras.</div><pre><strong># campfire/.env </strong>
SECRET_KEY_BASE=PHONY_SECRET_KEY_BASE
VAPID_PRIVATE_KEY=PHONY_PRIVATE_KEY
VAPID_PUBLIC_KEY=PHONY_PUBLIC_KEY
SSL_DOMAIN=chat.rubyzg.org
DISABLE_SSL=YES </pre><div><strong>To run Campfire without the installer, behind traefik, all I had to do is</strong></div><pre><strong>#!/usr/bin/bash</strong>
sudo sysctl vm.overcommit_memory=1
docker run \
-d \
--name campfire \
--network apps \
--restart unless-stopped \
--env-file campfire/.env \
--label-file campfire/labels \
-v campfire/Procfile:/rails/Procfile \
-v campfire/production.rb:/rails/config/environments/production.rb \
-v campfire/storage:/rails/storage \
registry.once.com/campfire:latest \
bin/boot;;</pre><div>Put this into <em>"campfire/labels"</em>
</div><pre># Enable traefik for the container
traefik.enable=true
# Convert HTTP traffic to HTTPS
traefik.http.routers.campfire.entrypoints=websecure
traefik.http.routers.campfire.tls.certresolver=letsencrypt
# Register the container to a domain - CHANGE THIS TO YOUR DOMAIN
traefik.http.routers.campfire.rule=Host(`chat.rubyzg.org`)
# Return a service not available screen if Campfire isn't running
traefik.http.middlewares.campfire.retry.attempts=10
traefik.http.middlewares.campfire.retry.initialinterval=1s
traefik.http.services.campfire.loadbalancer.healthcheck.path=/up
traefik.http.services.campfire.loadbalancer.server.port=80
traefik.http.services.campfire.loadbalancer.server.scheme=http </pre><div>Finally, I had to disable thruster and bind Puma to port 80 by updating the "web" entry in the Procfile (you get the source files in the purchase email)</div><pre><strong># campfire/Procfile </strong>
web: bin/rails server -b 0.0.0.0 -p 80
# ... </pre><div>This will boot the server, it will allow you to setup an admin account, create rooms, add a logo, <strong>but you won't be able to post messages because Action Cable can't connect to the server</strong>.<br><br>
</div><div>Traefik has <a href="https://github.com/traefik/traefik/issues/6076">a bug where it doesn't forward x-forwarded-for headers for WebSocket handshakes</a>. <br><br>To fix that I had to update <em>"production.rb"</em> with the exact URL that Action Cable would get requests from</div><pre><strong># production.rb</strong>
# ...
config.action_cable.url = "wss://#{ENV.fetch("SSL_DOMAIN")}/cable"
config.action_cable.allowed_request_origins = ["https://#{ENV.fetch("SSL_DOMAIN")}"]
# ...</pre><div>That's it.<br><br>
</div><div>It's isn't as straight forward as I'd like it to be, and it would be nice if Kamal and Campfire could play along nicely out of the box. I'd bet there is a large overlap between those communities, and both are 37signals projects.</div>
</div>
Stanko Krtalic Rusendichey@stanko.io