Running Campfire behind traefik
I rent a bare metal server from Scaleway where I host all my apps. For me, that's much more economical than running multiple small VPS boxes. 50€ a month buys me 2TB of disk space, 64GB of RAM and an 4 core / 8 thread CPU on which I can host two or three dozen apps. Compare that to AWS or DigitalOcean where for the same money I get about one quarter of those resources.
All my apps are deployed as Docker containers, most of them using Kamal, in front of which I put traefik as a reverse proxy to direct traffic and provide HTTPS via Let's Encrypt.
Last weekend I wanted to setup a Campfire instance for Ruby Zagreb - my local Ruby user-group - on that machine.
So I ran the official installer and got an error.
#!/usr/bin/bash sudo once setup 1111-1111-1111-1111 ██████╗ ███╗ ██╗ ██████╗███████╗ ██╔═══██╗████╗ ██║██╔════╝██╔════╝ ██║ ██║██╔██╗ ██║██║ █████╗ ██║ ██║██║╚██╗██║██║ ██╔══╝ ╚██████╔╝██║ ╚████║╚██████╗███████╗ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝ Failed to start the software The software couldn't be started. Please try again.
I expected the installer to fail since it can't know my setup, and it didn't ask me anything besides the domain of the instance.
Sadly, the email you get with your purchase doesn't explain what the installer does, and I can't check what it's doing since the installer is a binary blob. So I had to snoop around for a bit.
The first thing I found was the installer's error log which said that the container couldn't bind to port 443.
#!/usr/bin/bash cat .once-error.log 2024/01/27 19:33:11 Error response from daemon: driver failed programming external connectivity on endpoint campfire (e32f1a334f8ac25b9f3df0c283adb3eb3e8a554074f0827d17f8b566d03940a5): Bind for 0.0.0.0:443 failed: port is already allocated
Since traefik is bound to port 80 and 443 this makes perfect sense.
The error looks like it came from Docker so I checked my images, and sure enough there was a new "registry.once.com/campfire:latest" image and a stopped container which I promptly inspected.
Within the container runs a custom proxy called thruster that terminates SSL and handles the sendfile protocol. That's what's bound to port 443 and 80 of the container.
Inside the container is also a Redis instance, which implies that the installer sets at least "sysctl vm.overcommit_memory=1" else Redis wouldn't start.
I also saw that there were ENV variables injected into the container so I looked where those came from and found a config file in "/root/.config/once/config.json".
#!/usr/bin/bash sudo cat /root/.config/once/config.json | jq { "token": "1111-1111-1111-1111", "product": "campfire", "product_name": "Campfire", "email_address": "[email protected]", "ssl_domain": "chat.rubyzg.org", "validation_token": "PHONY_VALIDATION_KEY", "secret_key_base": "PHONY_SECRET_KEY_BASE", "vapid_private_key": "PHONY_PRIVATE_KEY", "vapid_public_key": "PHONY_PUBLIC_KEY", "storage_location": "/var/once/campfire", "cron_hour": 2, "once_binary_etag": "" }
Seems pretty straight forward. This is roughly what the installer does
#!/usr/bin/bash sudo sysctl vm.overcommit_memory=1 docker run \ -d \ --name campfire \ --restart unless-stopped \ --env-file campfire/.env \ -p 443 -p 80 -v campfire/storage:/rails/storage \ registry.once.com/campfire:latest \ bin/boot;;
Where "campfire/.env" contains the values from "/root/.config/once/config.json" plus a few extras.
# campfire/.env SECRET_KEY_BASE=PHONY_SECRET_KEY_BASE VAPID_PRIVATE_KEY=PHONY_PRIVATE_KEY VAPID_PUBLIC_KEY=PHONY_PUBLIC_KEY SSL_DOMAIN=chat.rubyzg.org DISABLE_SSL=YES
To run Campfire without the installer, behind traefik, all I had to do is
#!/usr/bin/bash sudo sysctl vm.overcommit_memory=1 docker run \ -d \ --name campfire \ --network apps \ --restart unless-stopped \ --env-file campfire/.env \ --label-file campfire/labels \ -v campfire/Procfile:/rails/Procfile \ -v campfire/production.rb:/rails/config/environments/production.rb \ -v campfire/storage:/rails/storage \ registry.once.com/campfire:latest \ bin/boot;;
Put this into "campfire/labels"
# Enable traefik for the container traefik.enable=true # Convert HTTP traffic to HTTPS traefik.http.routers.campfire.entrypoints=websecure traefik.http.routers.campfire.tls.certresolver=letsencrypt # Register the container to a domain - CHANGE THIS TO YOUR DOMAIN traefik.http.routers.campfire.rule=Host(`chat.rubyzg.org`) # Return a service not available screen if Campfire isn't running traefik.http.middlewares.campfire.retry.attempts=10 traefik.http.middlewares.campfire.retry.initialinterval=1s traefik.http.services.campfire.loadbalancer.healthcheck.path=/up traefik.http.services.campfire.loadbalancer.server.port=80 traefik.http.services.campfire.loadbalancer.server.scheme=http
Finally, I had to disable thruster and bind Puma to port 80 by updating the "web" entry in the Procfile (you get the source files in the purchase email)
# campfire/Procfile web: bin/rails server -b 0.0.0.0 -p 80 # ...
This will boot the server, it will allow you to setup an admin account, create rooms, add a logo, but you won't be able to post messages because Action Cable can't connect to the server.
Traefik has a bug where it doesn't forward x-forwarded-for headers for WebSocket handshakes.
To fix that I had to update "production.rb" with the exact URL that Action Cable would get requests from
To fix that I had to update "production.rb" with the exact URL that Action Cable would get requests from
# production.rb
# ...
config.action_cable.url = "wss://#{ENV.fetch("SSL_DOMAIN")}/cable"
config.action_cable.allowed_request_origins = ["https://#{ENV.fetch("SSL_DOMAIN")}"]
# ...
That's it.
It's isn't as straight forward as I'd like it to be, and it would be nice if Kamal and Campfire could play along nicely out of the box. I'd bet there is a large overlap between those communities, and both are 37signals projects.