DocsDeploy & operate

Swarms

Run several Sluurp servers as one: one writer, the same data on every node, rooms across nodes, a node lost without losing a write.

A swarm is several Sluurp servers acting as one. Every node answers reads from its own copy of the data, writes go to one node at a time, and a node can be lost without losing a write. Adding a node is one command. There’s no shared folder and no database server to run next to it.

To make a swarm, start the first machine with cluster init:

sluurp cluster init --cluster-addr 10.0.0.1:7946
sluurp serve

Features include:

  • One writer, with each write stored on a majority before it’s answered
  • Reads from every node, which always see their own writes
  • Files kept on 3 nodes, in different zones
  • Rooms, sync and live views across nodes
  • Upgrades with no downtime, and a rollback if a release fails to start
  • Nodes that talk over TLS, with the swarm’s own certificate authority

Swarms are new. Everything on this page works and is tested; a Helm chart and a way to keep the log short are still to come.

Make a swarm

init prints a join token. On every other machine:

sluurp serve --join SLUURP1-… --cluster-addr 10.0.0.2:7946

--cluster-addr is where the other nodes reach this one; the cluster port (7946) is for the nodes only and stays off the internet. The token can also be given as SLUURP_JOIN, which suits Docker and Kubernetes. A node joins once; after that it starts as part of the swarm.

Each node has a data folder of its own. Never point two nodes at the same folder, such as one Docker volume or network share mounted into several containers: SQLite is not made to be written by two servers through a shared disk, and Sluurp locks the folder (.sluurp.lock), so a second node there stops at once. The nodes share their data by replicating it, not through a disk. Give each container its own volume.

sluurp cluster status    # the nodes, which one leads, how far behind each is
sluurp cluster token     # a join token for another node

In front of the nodes

The nodes don’t spread requests among themselves: a load balancer in front does, and any node can answer any request. The nodes tell it who they are, as DNS: start any node (or all of them) with --dns, and it answers with the nodes taking requests now. A node that joins gets traffic, and one that stops answering stops getting it, with nothing to edit.

sluurp serve --join SLUURP1-… --cluster-addr 10.0.0.2:7946 --addr 0.0.0.0:8090 --dns 127.0.0.1:8053

With Caddy in front, terminating TLS:

app.example.com {
	reverse_proxy {
		dynamic a {
			name sluurp
			port 8090
			resolvers 127.0.0.1:8053
			refresh 5s
		}
		lb_policy least_conn
		health_uri /api/health
	}
}
  • dynamic a asks the nodes for every live node’s address, every few seconds, and sends requests to each on the web port given. Each node says where it takes requests when it starts: its --advertise address, or else --addr (with the machine’s cluster address when it listens on 0.0.0.0).
  • Nodes on different web ports: ask for SRV records instead (dynamic srv _http._tcp.sluurp), one per node with its port. Their targets are names like <node>.sluurp, which the balancer looks up through the machine’s own DNS, so that has to send .sluurp to the nodes too (with systemd-resolved, DNS=127.0.0.1:8053 and Domains=~sluurp in a file under /etc/systemd/resolved.conf.d/).
  • The nodes answer over UDP and TCP; Caddy asks over TCP.
  • Run --dns on the machine Caddy runs on, or on two nodes for a second resolver; keep the port off the internet. Any free UDP port works, but not 5353: that’s multicast DNS, which avahi and browsers already hold on most machines.
  • lb_policy cookie keeps each browser on one node, which helps live views a little. Nothing depends on it: a request on another node still reads its own writes.

What the nodes share

Records. One node, the leader, does all the writing. A write that reaches any other node is passed to the leader, and the answer comes back the same way, so you can send requests to any node. On the leader, a transaction commits only once a majority of the nodes have it stored: if the leader dies right after answering, the write is still there. If it can’t reach a majority, the write fails and nothing is changed anywhere. Every node then applies the transaction whole, in the same order, so no node ever shows half of one.

Because there’s only one writer, writes never conflict. Two people saving the same field one after the other get exactly what a single server would give them. likes+: 1 from many places at once adds up exactly (see adding to a number), a unique value can only be taken once, and a batch lands whole or not at all. Collections made or changed through any node are made on every node. Realtime subscriptions, sync and live views hear about every change, whichever node they’re connected to.

Reading your own writes. Reads are answered by the node they reach, from its own copy, which can be a few milliseconds behind the leader. Every answer to a write carries its position in the log, and the Sluurp client sends it back with its next request. A node that hasn’t applied that position yet waits for it (usually a millisecond), so a page never shows something older than what it just saved.

The same data everywhere. Every few seconds the leader compares each node’s copy with its own at the same point in the log. If a copy ever differs, say because someone edited a database file by hand, the node is logged as an error and named in sluurp cluster status.

Files. Uploads and backups are kept on 3 nodes, in different zones when nodes say which zone they are in (--zone). An upload is answered once 2 nodes have it. A node that needs a file it doesn’t hold fetches it in pieces from several nodes at once.

Rooms. People in the same room (typing, cursors, who is here) see each other whichever node they are on.

Plugins. Install, update and remove plugins through any node, in the admin’s Marketplace or /api/plugins: the change runs on the leader, and every node then gets the same plugin files, fetched from the nodes that hold them. A node that missed a change, or lost its copy, catches up within a minute. sluurp plugin add refuses on a node of a swarm, because only that node would get the plugin. Dev tools (Storyboard, Hawkeye) are not copied to other nodes, and in production they don’t run. A model a plugin fetches, such as the AI plugin’s embeddings model given as a Hugging Face address, is downloaded once by each node.

Upgrade

A release is a new Sluurp binary and your app’s folder, together. Only the leader changes the data model: it applies schema.json and runs the app’s and its plugins’ migrations, and the changes reach every node through the log. A node whose code is for another data model than the one the swarm agreed doesn’t serve your API and is left out of the DNS your proxy reads. New code never runs against an old data model, and old code never writes to a migrated one.

On one machine: sluurp cluster upgrade

sluurp cluster upgrade replaces the node running on this machine with a new release, with no downtime and no write lost. Put the release in a folder of its own, then run it beside the running node’s data folder:

sluurp --dir data cluster upgrade \
  --next data-next \
  --cluster-addr 127.0.0.1:7947 --web 127.0.0.1:8092 \
  --start "sudo systemctl start myapp@next" \
  --stop "sudo systemctl stop myapp@next" \
  --stop-old "sudo systemctl stop myapp@live" \
  --check /api/health

It goes in four steps, and says each one as it goes:

  1. Join. A fresh data folder (--next) joins the swarm and copies its data.
  2. Start. --start starts the new release on that folder: a systemd unit, a script, any command. The upgrade waits until the new node votes and has caught up, and answers its health check. While it waits for its data model, the running node goes on serving every request.
  3. Hand over. The running node steps aside and the new one leads. It runs the migrations, and once --check answers it serves. The old node’s code is now behind the data model, so it stops serving your API.
  4. Leave. The old node leaves the swarm, and --stop-old stops it.

Nothing is lost on the way. A write made before the handover is in the log the new node copied, and every write after it goes to the new leader.

Rollback. If anything fails before the handover (the release doesn’t start, doesn’t catch up, or isn’t healthy), the new node is taken out of the swarm and --stop stops it. The running node goes on as it was, and the command says the upgrade was rolled back. After the handover the data model has moved on, so a failure there is fixed forward with the next release. Try a release on a copy of production first (sluurp --dir data clone data-copy copies it without touching it), so a failing migration shows up there.

A server that isn’t in a swarm yet becomes one with sluurp cluster init; a swarm of one node is enough.

Several nodes

Upgrade one node at a time:

  1. Followers first. Stop one, put the new release in place, start it. Until the leader has the new version too, it waits: sluurp cluster status shows “waiting for the leader’s data model” beside it, and the other nodes go on serving.
  2. The leader last. On an upgraded node, run sluurp cluster lead: the leader steps aside, the upgraded node leads and runs the migrations, and every upgraded node starts serving. Then stop the old leader, put the new release in place and start it.

sluurp cluster status lists each node’s version. sluurp cluster leave takes a node out of the swarm for good, handing over the lead first if it has it.

When a node is lost

Nodes check on each other every two seconds. If the node leading the swarm is lost, the others choose a new one within a few seconds. A node silent for longer than the grace period (10 minutes, SLUURP_SWARM_GRACE) is taken out of the swarm, and the remaining nodes copy its files again so each is on 3 nodes. Every node also re-reads what it holds over time, and replaces a damaged copy with a good one.

A node that was offline catches up on every write it missed when it comes back. A new node first copies each project from the swarm, then follows the log from there; anything it had before joining is replaced. Each node’s databases commit to disk before answering, so a power cut doesn’t lose a write that was confirmed.

To restore a backup into a swarm, stop every node, restore on one, start it on its own with sluurp cluster init, and have the others join it. sluurp restore refuses to restore a single node in place, because that node would then hold different data from the rest.

Trust

The swarm has its own certificate authority, made by init. Nodes talk over TLS both ways, each showing a certificate it signed; the join token pins the authority and carries a secret, and a node that joins makes its own key, which never leaves it. Anyone without a certificate gets nothing from the cluster port but the way in.

View as Markdown