DocsData

Migrations

Versioned changes to an app’s data model and data, run once each, in order.

An app’s migrations/ folder holds one file per version, as Flyway does. When the server starts, each file runs once, in order: a new install runs all of them, and an existing one runs only those it hasn’t.

V1__init.json          # collections, in the same format as schema.json
V2__sample_data.json   # records (leave the file out for no sample data)
V3__catalog.js         # make or transform data in JavaScript
V4__orders_total.json  # a schema change, as a patch
V5__tidy.sql           # plain SQL
R__views.sql           # repeatable: runs again whenever it changes

Features include:

  • Collections and records in JSON, data made in JavaScript, and plain SQL
  • Placeholder ids, so records can point at each other in one file
  • Patches that change or remove fields and collections by name
  • A snapshot first, and all pending migrations or none
  • A record of what ran, with each file’s text and its hash

Versions

Files are named V<version>__<description>: V1, V2, V2_1 (between V2 and V3). Versions compare as numbers, with _ or . between parts and leading zeros ignored: V2_1 is V2.01, and V2_2 comes before V2_10.

  • A version runs once per database. One that already ran is always skipped, even if its file has changed since. A database that ran it keeps what it did, and a new database runs the file as it is now. A change to a file that already ran is recorded and logged.
  • To change a database that already ran a version, add a new version.
  • R__ files run after the versioned ones, and again whenever their contents change.

migrations/ goes in the app’s folder. To share it between apps in one repository, such as an app and its reports/, put it next to them; it then runs once for all of them.

JSON files

A .json file is an import document in the same format as schema.json: collections, then records. Sluurp inserts records in the order their references need, so a row exists before anything points at it.

Placeholder ids

When the server makes a record’s id, you can’t write it in the file. Give the record a placeholder id instead, starting with $id_, and use the same placeholder wherever another record points at it:

migrations/V2__sample_data.json
{
  "records": {
    "teams": [{ "id": "$id_red", "name": "Red", "lead": "$id_ana" }],
    "people": [
      { "id": "$id_ana", "name": "Ana", "team": "$id_red" },
      { "id": "$id_bo", "name": "Bo", "team": "$id_red" }
    ]
  }
}

Each record gets a real id from the server, and every placeholder is replaced by it. This works whichever record comes first, even when two records point at each other, as a team and its lead do here. It also works for placeholders inside a list, such as a JSON field that holds several ids.

A placeholder that no record in the file has as its id stops the migration, and the error names it. Records with a real id, or none, go in as written.

Settings and plugin data

A document can also hold settings, and data that plugins import: conversations with their messages (the conversations plugin), pages with their blocks and rows (pages), and feeds with their posts (social).

A post needs id, feed, author and body, and can have reply_to (a reply, in the thread’s feed), created_at, likes (user ids) and pinned. Posts go in the same way as posts made in the app, so counts, #tags and search stay right.

migrations/V5__news.json
{ "posts": [
    { "id": "welcome", "feed": "news", "author": "editor0000001", "body": "Welcome to the new site! #launch", "pinned": true },
    { "id": "welcome-1", "reply_to": "welcome", "author": "reader0000001", "body": "Looks great!" }
] }

Existing pages are left as they are, unless the document sets "merge": true. That updates folder settings, properties, template, navigation, sign-ups and look, and keeps content and rows.

Patches

A patch changes the schema by name. It is a JSON merge patch: objects add or change, and null deletes, along with its data. It is the only kind of migration that removes anything, and it removes only what it names.

migrations/V4__orders_total.json
{ "patch": {
    "orders": { "fields": { "total": { "type": "number" }, "old_note": null } },
    "invoices": null
} }

A file can hold a patch, collections and records together. The patch runs first, then the collections, then the records, so the schema and the data of one version travel together.

JavaScript files

A .js or .ts file exports a function that runs like a job, for example ctx.collection("orders").update(…).

To make a lot of data, return a document ({ records: … } or a patch). Sluurp imports it like a .json migration:

migrations/V3__catalog.js
export default function () {
  const products = categories.flatMap((c) => productsFor(c));
  return { records: { products } };
}

SQL files

A .sql file runs as one script, in one transaction.

Run migrations

Migrations run when sluurp serve starts, before the app’s agents. To preview or run them without serving:

sluurp migrate ./app --plan
sluurp migrate ./app

A database may already have what some files make, because it was set up by hand or before it had migrations. To say so, pass --baseline. It records every file up to that version as run, without running it, and later files run as usual:

sluurp migrate ./app --baseline V14

The record of what ran

Each database keeps a record of its migrations in __app_migrations, for auditing. For every file it holds:

  • the file’s text as it ran, and that text’s SHA-256 (with line endings as \n, so you can check one against the other);
  • when it ran, how long it took, and the version of Sluurp that ran it;
  • whether it ran, or was recorded by --baseline;
  • if the file changed after it ran: when the change was first seen, with the new text and its SHA-256.

To list it:

sluurp migrate ./app --history

All or nothing

Before it runs pending migrations, Sluurp takes a snapshot of the database in snapshots/<timestamp>-migration/. If any migration fails, Sluurp puts the snapshot back (schema, data and the record of migrations), and the error names the file.

Nothing is left half done: every write from the failing file, and from earlier migrations in the same run, is undone. An app gets all its pending migrations or none. Fix the file and start the server again.

The snapshot is kept either way, as a backup from before the upgrade, listed under Backups in the admin.

An app with only a schema.json works as before: on each start, collections are added or changed, never removed.

View as Markdown