---
title: Queues and workflows
description: "Work done later, durably: task queues with retries, and workflows that sleep, wait for events and survive restarts."
section: Server
order: 6
---

# Queues and workflows

Work done later, durably: task queues with retries, and workflows that sleep, wait for events and survive restarts.

Send work to a queue and it is done later, at least once, even if the server restarts halfway. Write a workflow and it runs step by step over minutes or days, sleeping and waiting for events, and picks up where it was after a crash. There is no broker and no worker process: the queue is in your database. To add a task, call `ctx.queue` from any server code, whether a function, a hook, a job or another task:

```js title="functions/sign-up.js"
export default async function (ctx) {
  const person = ctx.collection("people").create(ctx.body);
  ctx.queue("emails").add({ to: person.email }, { key: `welcome-${person.id}` });
  return person;
}
```

Features include:

- Retries with a growing pause, and leases so a crash loses nothing
- Keys, so retrying a request never does the work twice
- Delays, times and priorities
- Workflows whose steps run once, with sleeps and waits for events that last days
- Tasks and runs as records you can browse, filter, retry and cancel
- Work shared out across a swarm

## Queues

To say what to do with each task, add a file named after the queue:

```js title="queues/emails.js"
export const concurrency = 4; // tasks of this queue at once, at most
export const attempts = 5;    // tries before the task is given up

export default async function (ctx) {
  const { payload, attempt } = ctx.body;
  ctx.mail({ to: payload.to, subject: "Welcome", text: "Glad you are here." });
  return { sent: true }; // kept as the task's result
}
```

### Options

| Option | What it does |
| --- | --- |
| `delay` | Run it later: `"30s"`, `"10m"`, `"2h"`, `"1d"`, or milliseconds. |
| `at` | Run it at a time, in milliseconds since 1970. |
| `priority` | Higher runs first. Tasks of the same priority run oldest first. |
| `key` | Names the work. Adding a task with a key the queue already has returns that task instead of adding another, so retrying a request never sends two emails. |
| `attempts` | Tries before the task is given up, for this task only. |

### What happens to a task

1. It waits as **pending** until it may run.
2. A runner claims it: it is **running**, and its runner holds a lease on it, renewed every few seconds while the handler works.
3. When the handler returns, the task has **succeeded**, and its result is kept for a week.
4. When the handler throws, the task is tried again after a growing pause (2 seconds, 4, 8… up to a minute). When its attempts are used up, it is **dead**, with its last error kept, until you retry it.
5. If the server stops in the middle, the lease runs out after 30 seconds and the task is pending again, its attempt counted.

> [!NOTE]
> A task can run more than once, for example after a crash that came after the work but before the result was saved. Write handlers that can safely run twice: key what they write by the work, or check whether it was already done.

### Watch and manage tasks

Tasks are kept in the `_tasks` collection. Browse them in the admin under **Managed**, and filter them like any records.

- `GET /api/admin/tasks` gives each queue's counts, how long its oldest task has waited, and how fresh its running tasks' heartbeats are.
- `POST /api/admin/tasks/{id}/retry` runs a dead task again.
- `POST /api/admin/tasks/{id}/cancel` stops a pending one.

## Workflows

A workflow is code that runs in steps over a long time. Each step runs once and its result is saved. In between, the workflow can sleep or wait for something to happen.

```js title="workflows/onboarding.js"
export default async function (ctx) {
  const { input } = ctx.body;
  const account = await ctx.step("create account", () => ctx.collection("accounts").create(input));
  await ctx.step("welcome email", () => ctx.mail({ to: input.email, subject: "Welcome" }));

  await ctx.sleep("a day later", "1d");

  const answer = await ctx.waitFor("survey", "survey.answered", { match: { account: account.id }, timeout: "7d" });
  if (answer === null) await ctx.step("reminder", () => ctx.mail({ to: input.email, subject: "Two minutes?" }));
  return { account: account.id, answered: answer !== null };
}
```

To start a run, call `ctx.workflow` from any server code. With a key, starting twice is one run:

```js
ctx.workflow("onboarding").start({ email: "ann@example.com" }, { key: "ann@example.com" });
```

And somewhere else, when the survey is answered:

```js
ctx.emit("survey.answered", { account: account.id, score: 9 });
```

| Inside a workflow | What it does |
| --- | --- |
| `ctx.step(name, fn)` | Runs `fn` once for the run and saves what it returns. Later, the saved result is returned without running it again. |
| `ctx.sleep(name, duration)` | Pauses the run for `"30s"`, `"2h"`, `"7d"`… No server is busy meanwhile. |
| `ctx.waitFor(name, event, { match, timeout })` | Pauses until an event of that name is emitted whose data has the fields in `match`, and returns its data; or returns `null` after `timeout`. |

### How it survives

Each time a run is resumed, after a sleep, an event or a crash, its code runs again from the top. Steps it finished before don't run again: their saved results are returned at once, so the code arrives where it stopped. Two rules follow:

- Put everything that touches the world, such as writing records, sending mail or calling an API, inside a step.
- Code between steps should only compute, since it runs again on every resume.

A run that throws is retried like a task. When its tries are spent, it has **failed**, with its error. Runs are kept in the `_workflow_runs` collection, with their input, the result of every step and what they are waiting for. `POST /api/admin/workflows/{id}/cancel` stops a run.

## In a swarm

In a [swarm](/docs/swarms), tasks and runs are records like any other: every node has them, and only the leader writes them, through the swarm's log.

- The leader runs every queue and workflow. If it fails, the new leader finds tasks whose leases ran out and runs them again, so nothing is lost.
- A queue whose handler says `export const anywhere = true` runs on **every** node. The other nodes claim its tasks and save their results through the leader, so the work is shared out while the queue stays in one place.

```js title="queues/thumbnails.js"
export const anywhere = true; // any node may do this work
export const concurrency = 4; // on each node

export default async function (ctx) {
  const image = await fetch(ctx.body.payload.url).then((r) => r.arrayBuffer());
  return { size: image.byteLength };
}
```

A handler that runs anywhere computes, or calls out over HTTP, mail or AI, and returns its result. It doesn't write records itself: on a node that doesn't lead, the database takes writes only from the swarm's log. By default, queues run on the leader; keep queues that write records there.
