Why SQLite

Sluurp's pitch is one binary, one data file, nothing else to run. That requires an embedded database: a backend that starts with "first, set up Postgres" has already broken that promise.

What you get

  • No network hop. A query is an in-process function call, not a round trip. For the small indexed reads most pages do, that removes most of the cost.
  • Nothing to operate. No database server to install, upgrade, tune, secure or get paged about. sluurp serve is the whole deployment.
  • A project is a file. Each project is a separate SQLite file. Multi-tenancy is a folder of files, a backup is a file copy (see Backups in the admin UI), and moving a tenant is moving a file.
  • Concurrent reads. In WAL mode, readers don’t block each other or the writer. The connection pool is sized to the CPU count, so reads scale with cores.
  • Same engine in the browser. Sync can store rows in browser SQLite (the official WebAssembly build), so SQL works the same on both sides, FOR SYSTEM_TIME AS OF included.
  • Collections are real tables, with real indexes, unique constraints and SQLite’s query planner, not rows in a generic key-value blob. Rules and filters compile to that SQL with bound parameters.

The tradeoff

One writer at a time. SQLite serialises writes: reads scale with cores, writes don’t. A Sluurp write is a short transaction (the row, its history if enabled, the rule re-check), so the queue moves fast. For most apps (a school, a shop, an internal tool, a SaaS with one file per tenant) that’s far more headroom than needed. If you need tens of thousands of writes per second into one table, use something else.

Why not something else

Why not
PostgresA separate server, which is exactly what Sluurp exists to eliminate. It remains the escape hatch for installs that outgrow a single file: storage is behind a trait, so Postgres support is a new driver, not a rewrite.
CockroachDBWritten in Go, with no embeddable library. Since v24.3 the self-hosted Core edition is gone, and the source-available licence restricts redistribution.
pgrustPostgres rewritten in Rust, moving fast. But it’s AGPL-3.0, warns against storing important data, and the embeddable build is still on the roadmap.
TursoThe Rust rewrite of SQLite: MIT, in-process, file-format compatible, with async I/O and concurrent writes. The likely future engine; switching would be a driver change, not a migration, which is why storage sits behind a trait.

How fast

sluurp bench benchmarks collection access through Sluurp’s storage layer, exactly as a request handler calls it: rule compilation, filter parsing and binding, SQLite, and JSON serialisation. It uses a throwaway data directory and reports throughput plus median and p99 latency per operation.

sluurp bench            # table and score
sluurp bench --json     # machine-readable

On AMD Ryzen 7 9800X3D 8-Core Processor, 16 threads, Windows, 10,000 rows (Sluurp 0.1.0):

OperationCallers at oncePer secondMedianp99
Create, one callerone6,3080.124 ms0.336 ms
Create323,9394.656 ms60.301 ms
Read one row by id, one callerone21,7280.04 ms0.098 ms
List 20, filtered on an index, one callerone7,9750.116 ms0.199 ms
Read one row by id3225,8470.66 ms14.915 ms
List 20, filtered on an index3223,2221.354 ms3.417 ms
Count by group (10000 rows)3224,0411.084 ms5.323 ms
List 20 under a row rule3241,6180.681 ms2.423 ms
Update one field327,7212.186 ms41.342 ms
Create with history kept322,9466.399 ms72.496 ms
List 20 as it stood (asOf)326,5134.044 ms29.912 ms
Delete321,91210.319 ms83.838 ms

Score: 2,048. The geometric mean of each operation’s speed relative to a reference run, × 1,000. Every operation weighs equally regardless of scale; a machine twice as fast at everything scores 2,000. Run-to-run noise is a few percent. This machine scores about 2× the reference because the reference predates the asOf speed-up and the write-lock retry fix.

Against bare SQLite

The same operations directly on SQLite (rusqlite, same settings, one caller, prepared statements), with no rules, filter parsing, pool or JSON. This is Sluurp’s overhead over the raw engine:

Operation, one callerPer secondMedianp99
Create, one at a time11,4040.065 ms0.161 ms
Read one row by id, one caller214,2100.005 ms0.005 ms
List 20, filtered on an index, one caller47,4750.021 ms0.03 ms
Count by group (10000 rows), one caller4,6510.21 ms0.348 ms

Reading a row by id takes ~5 µs in bare SQLite and ~45 µs through Sluurp: the difference is the compiled view rule, dispatch to a pooled connection, and JSON serialisation. A create takes ~0.07 ms bare and ~0.11 ms through Sluurp, which also re-checks the create rule inside the transaction.

Takeaways:

  • Reads scale. Get-by-id, an indexed filtered page, a page under a row rule and an aggregate over 10,000 rows each take about a millisecond, at tens of thousands per second across cores.
  • Writes queue. One caller creates a row in ~0.1 ms. Concurrent writers share SQLite’s single writer, so total throughput stays about the same and each waits its turn.
  • Time travel is fast. An asOf list is rebuilt by SQLite in one statement from the change log and cached on the connection until something at or before that point changes. It used to take about a second on this data.
  • Deletes keep up. A delete also writes to the recycle bin and still runs about as fast as a create with history. Waiting writers used to sleep up to 100 ms between retries while the lock sat mostly free; they now retry within a fraction of a millisecond, which made 32 concurrent deletes 2.5× faster (from 778/s) and updates 2× faster.

Beyond one machine

Several Sluurp nodes can serve one data directory behind a load balancer. Each node gets --advertise and --node-id, and the balancer pins each tab to a node with a sticky cookie. All nodes need the same --dir on shared storage, which is where the real constraints are; see “Sharing the data directory” in the README.

Since projects are separate files, spreading them across machines later is a routing problem (move a file, update a map), not a distributed-join problem.