DocsServer

Queues and workflows

Work done later, durably: task queues with retries, and workflows that sleep, wait for events and survive restarts.

Send work to a queue and it is done later, at least once, even if the server restarts halfway. Write a workflow and it runs step by step over minutes or days, sleeping and waiting for events, and picks up where it was after a crash. There is no broker and no worker process: the queue is in your database. To add a task, call ctx.queue from any server code, whether a function, a hook, a job or another task:

functions/sign-up.js
export default async function (ctx) {
  const person = ctx.collection("people").create(ctx.body);
  ctx.queue("emails").add({ to: person.email }, { key: `welcome-${person.id}` });
  return person;
}

Features include:

  • Retries with a growing pause, and leases so a crash loses nothing
  • Keys, so retrying a request never does the work twice
  • Delays, times and priorities
  • Workflows whose steps run once, with sleeps and waits for events that last days
  • Tasks and runs as records you can browse, filter, retry and cancel
  • Work shared out across a swarm

Queues

To say what to do with each task, add a file named after the queue:

queues/emails.js
export const concurrency = 4; // tasks of this queue at once, at most
export const attempts = 5;    // tries before the task is given up

export default async function (ctx) {
  const { payload, attempt } = ctx.body;
  ctx.mail({ to: payload.to, subject: "Welcome", text: "Glad you are here." });
  return { sent: true }; // kept as the task's result
}

Options

OptionWhat it does
delayRun it later: "30s", "10m", "2h", "1d", or milliseconds.
atRun it at a time, in milliseconds since 1970.
priorityHigher runs first. Tasks of the same priority run oldest first.
keyNames the work. Adding a task with a key the queue already has returns that task instead of adding another, so retrying a request never sends two emails.
attemptsTries before the task is given up, for this task only.

What happens to a task

  1. It waits as pending until it may run.
  2. A runner claims it: it is running, and its runner holds a lease on it, renewed every few seconds while the handler works.
  3. When the handler returns, the task has succeeded, and its result is kept for a week.
  4. When the handler throws, the task is tried again after a growing pause (2 seconds, 4, 8… up to a minute). When its attempts are used up, it is dead, with its last error kept, until you retry it.
  5. If the server stops in the middle, the lease runs out after 30 seconds and the task is pending again, its attempt counted.

A task can run more than once, for example after a crash that came after the work but before the result was saved. Write handlers that can safely run twice: key what they write by the work, or check whether it was already done.

Watch and manage tasks

Tasks are kept in the _tasks collection. Browse them in the admin under Managed, and filter them like any records.

  • GET /api/admin/tasks gives each queue’s counts, how long its oldest task has waited, and how fresh its running tasks’ heartbeats are.
  • POST /api/admin/tasks/{id}/retry runs a dead task again.
  • POST /api/admin/tasks/{id}/cancel stops a pending one.

Workflows

A workflow is code that runs in steps over a long time. Each step runs once and its result is saved. In between, the workflow can sleep or wait for something to happen.

workflows/onboarding.js
export default async function (ctx) {
  const { input } = ctx.body;
  const account = await ctx.step("create account", () => ctx.collection("accounts").create(input));
  await ctx.step("welcome email", () => ctx.mail({ to: input.email, subject: "Welcome" }));

  await ctx.sleep("a day later", "1d");

  const answer = await ctx.waitFor("survey", "survey.answered", { match: { account: account.id }, timeout: "7d" });
  if (answer === null) await ctx.step("reminder", () => ctx.mail({ to: input.email, subject: "Two minutes?" }));
  return { account: account.id, answered: answer !== null };
}

To start a run, call ctx.workflow from any server code. With a key, starting twice is one run:

ctx.workflow("onboarding").start({ email: "ann@example.com" }, { key: "ann@example.com" });

And somewhere else, when the survey is answered:

ctx.emit("survey.answered", { account: account.id, score: 9 });
Inside a workflowWhat it does
ctx.step(name, fn)Runs fn once for the run and saves what it returns. Later, the saved result is returned without running it again.
ctx.sleep(name, duration)Pauses the run for "30s", "2h", "7d"… No server is busy meanwhile.
ctx.waitFor(name, event, { match, timeout })Pauses until an event of that name is emitted whose data has the fields in match, and returns its data; or returns null after timeout.

How it survives

Each time a run is resumed, after a sleep, an event or a crash, its code runs again from the top. Steps it finished before don’t run again: their saved results are returned at once, so the code arrives where it stopped. Two rules follow:

  • Put everything that touches the world, such as writing records, sending mail or calling an API, inside a step.
  • Code between steps should only compute, since it runs again on every resume.

A run that throws is retried like a task. When its tries are spent, it has failed, with its error. Runs are kept in the _workflow_runs collection, with their input, the result of every step and what they are waiting for. POST /api/admin/workflows/{id}/cancel stops a run.

In a swarm

In a swarm, tasks and runs are records like any other: every node has them, and only the leader writes them, through the swarm’s log.

  • The leader runs every queue and workflow. If it fails, the new leader finds tasks whose leases ran out and runs them again, so nothing is lost.
  • A queue whose handler says export const anywhere = true runs on every node. The other nodes claim its tasks and save their results through the leader, so the work is shared out while the queue stays in one place.
queues/thumbnails.js
export const anywhere = true; // any node may do this work
export const concurrency = 4; // on each node

export default async function (ctx) {
  const image = await fetch(ctx.body.payload.url).then((r) => r.arrayBuffer());
  return { size: image.byteLength };
}

A handler that runs anywhere computes, or calls out over HTTP, mail or AI, and returns its result. It doesn’t write records itself: on a node that doesn’t lead, the database takes writes only from the swarm’s log. By default, queues run on the leader; keep queues that write records there.

View as Markdown