JavaScript Web Workers logo JS Web Workers

Restarting Crashed Workers with Exponential Backoff

A worker can die in three ways, and only one of them tells you: an uncaught exception fires error, an out-of-memory kill fires nothing at all, and a deadlocked thread stays technically alive while answering nothing. A supervisor has to handle all three, and a retry loop without backoff turns the second one into a crash loop that consumes a core.

This is the recovery half of Main Thread vs Worker Thread Lifecycle, part of Web Workers Architecture & Communication.

The Supervisor

// supervised-worker.ts
interface Pending { reject: (error: Error) => void; timer: ReturnType<typeof setTimeout> }

export class SupervisedWorker {
  #worker: Worker | null = null;
  #pending = new Map<number, Pending>();
  #failures = 0;
  #openUntil = 0;                 // circuit breaker: no restarts before this time
  #seq = 0;

  constructor(
    private create: () => Worker,
    private onReady: (worker: Worker) => void = () => {},
    private opts = { baseMs: 250, maxMs: 30_000, maxFailures: 6, requestTimeoutMs: 15_000 },
  ) {}

  #start(): Worker {
    const worker = this.create();
    worker.addEventListener('error', (event) => this.#crashed(new Error(event.message)));
    worker.addEventListener('messageerror', () => this.#crashed(new Error('message could not be deserialised')));
    worker.addEventListener('message', (event) => this.#settle(event));
    this.onReady(worker);         // replay state: config, loaded datasets, warm caches
    this.#worker = worker;
    return worker;
  }

  #crashed(error: Error): void {
    // 1. Fail everything in flight — those promises will never settle otherwise.
    for (const { reject, timer } of this.#pending.values()) { clearTimeout(timer); reject(error); }
    this.#pending.clear();

    // 2. Discard the corpse before scheduling a replacement.
    this.#worker?.terminate();
    this.#worker = null;

    // 3. Back off, with full jitter, and give up after enough consecutive failures.
    this.#failures++;
    if (this.#failures > this.opts.maxFailures) {
      this.#openUntil = Date.now() + 5 * 60_000;      // breaker open
      return;
    }
    const ceiling = Math.min(this.opts.maxMs, this.opts.baseMs * 2 ** (this.#failures - 1));
    setTimeout(() => this.#start(), Math.random() * ceiling);
  }
}

Three responsibilities, in a strict order. Rejecting in-flight requests comes first, because a promise that never settles is worse than one that rejects — it hangs whatever awaited it, permanently and silently. Terminating the old worker comes second, since an error event does not necessarily mean the thread stopped. Only then is a replacement scheduled.

Supervisor states and the transitions between them A state machine with four states. Healthy: the worker answers requests and the consecutive-failure counter resets on every success. Crashed: reached by an error event, a message error, or a request timeout; in-flight promises are rejected and the old thread is terminated. Backing off: a replacement is scheduled after a randomised delay that doubles with each consecutive failure up to a thirty-second ceiling, returning to healthy once the new worker answers. Breaker open: entered after six consecutive failures, where no restart is attempted for five minutes and callers fail fast with a clear error. What the supervisor does, and when healthy answers requests success resets the counter crashed reject in-flight, terminate error · messageerror · timeout backing off delay doubles, jittered 250 ms → 30 s ceiling breaker open 6 consecutive failures fail fast for 5 minutes death detected replacement answers too many failures Without the breaker, a worker that crashes on startup restarts forever and burns a core doing it.
The dashed transition is what separates a supervisor from a retry loop: some failures are permanent, and hammering them costs the user battery for nothing.

Detecting the Silent Deaths

Only an uncaught exception announces itself. Two other mechanisms cover the rest.

Per-request timeouts. Every request gets a timer; expiry is treated as a crash. This catches out-of-memory kills, infinite loops and deadlocks with one mechanism, and it is the minimum viable liveness check:

request<T>(message: unknown): Promise<T> {
  if (Date.now() < this.#openUntil) return Promise.reject(new Error('worker unavailable'));
  const id = ++this.#seq;
  return new Promise<T>((resolve, reject) => {
    const timer = setTimeout(
      () => this.#crashed(new Error(`request ${id} timed out after ${this.opts.requestTimeoutMs} ms`)),
      this.opts.requestTimeoutMs);
    this.#pending.set(id, { reject, timer });
    (this.#worker ?? this.#start()).postMessage({ id, ...(message as object) });
  });
}

A heartbeat. For a worker that is idle much of the time, a ping every few seconds detects death before the next request rather than during it. The reply must be cheap and must not queue behind a long task — which requires the worker to yield periodically, the same requirement cancellation has.

Set the timeout generously. A threshold below the worker’s realistic worst-case task time turns slow work into a restart, which discards a warm worker and re-runs the task that was nearly finished — the most expensive possible outcome.

Backoff, Jitter and the Breaker

Why exponential. A worker that crashes because the machine is out of memory will crash again immediately if restarted immediately. Doubling the delay gives the condition time to clear, and caps the damage at a handful of attempts per minute rather than hundreds.

Why jitter. A pool of eight workers that all die together — a shared bad input, a memory ceiling — would otherwise restart in lockstep, produce the same load spike, and crash together again. Full jitter (Math.random() * ceiling) spreads them out and is both simpler and better than the fixed-plus-random variants.

Why a breaker. Some failures never resolve: a syntax error in the worker script, a missing chunk after a deploy, an unsupported API on this engine. Six consecutive failures is enough evidence to stop trying and start telling the user, which is a far better outcome than a page that is slow and quiet for the rest of the session.

Resetting the counter. On the first successful reply, not on successful construction. A worker that starts and then dies during initialisation would otherwise reset the counter every time and never trip the breaker.

Restart attempts over time with and without backoff Two attempt patterns over sixty seconds for a worker that fails on every start. Without backoff, restarts occur continuously — dozens per second — consuming a full core and producing hundreds of identical error reports. With exponential backoff and jitter, attempts occur at roughly a quarter second, half a second, one second, two, four, eight and sixteen seconds, and then stop entirely as the circuit breaker opens, for seven attempts in total and negligible CPU. A worker that fails every time: attempts in the first minute no backoff continuous restarts — a full core, hundreds of identical error reports backoff + jitter breaker opens — no further attempts 0 s 20 s 40 s 60 s Seven attempts is enough to recover from anything transient, and enough evidence to stop when the cause is permanent. The user gets an error instead of a warm laptop.
Both lines represent the same broken worker. The difference is entirely in how much of the user's machine the failure consumes.

Restoring State in the Replacement

A new worker starts empty. Whatever the old one had loaded — configuration, a parsed dataset, a compiled WebAssembly module, a warm cache — has to be re-established before the replacement can serve requests, which is what the onReady hook is for.

Two rules keep this honest. Keep the replay idempotent and cheap: a list of setup messages, not a transcript of everything that ever happened. And queue requests until replay completes, so a caller cannot get a wrong answer from a half-initialised worker:

#start(): Worker {
  const worker = this.create();
  this.#ready = (async () => {
    worker.postMessage({ kind: 'CONFIG', config: this.config });
    if (this.dataset) worker.postMessage({ kind: 'LOAD', dataset: this.dataset });
  })();
  return worker;
}
// request(): await this.#ready before posting.

Where replay is expensive — a 200 MB dataset, a multi-second WebAssembly compile — it is worth reconsidering what the worker holds. State that can be rebuilt from the page cheaply belongs in the worker; state that cannot should live on the page and be transferred per request, so a crash costs a restart rather than a re-download.

What to Do With the Failed Request

Restarting the worker is only half of recovery; the request that was in flight when it died still has an owner waiting. Three policies, and the right one depends on the operation.

Reject and surface. The default, and the only safe choice for anything with side effects. The caller learns the operation failed and decides — usually by showing an error with a retry affordance. Silence is the failure mode to avoid here: a promise that never settles leaves a spinner on screen indefinitely.

Retry once, automatically, if the operation is pure. A parse, an aggregation or a render has no side effects, so re-running it on the replacement worker is safe and usually invisible to the user. Retry exactly once per request: a request that kills two workers in a row is very likely the cause of the crashes — a payload that exhausts memory, an input that triggers an infinite loop — and retrying it forever recreates the crash loop that the backoff was meant to prevent.

Degrade to the main thread. For small, critical work, computing inline is better than failing. This requires the algorithm to be a neutral module rather than living inside the worker file, which is the same structure that makes it testable — see Testing Workers in CI.

async run<T>(message: Request, opts = { retryPure: false }): Promise<T> {
  try {
    return await this.request<T>(message);
  } catch (error) {
    if (!opts.retryPure || message.attempt) throw error;      // never retry twice
    return this.request<T>({ ...message, attempt: 1 });
  }
}

Whichever policy applies, record the crash with the request that was in flight. The correlation between a specific input and a repeated death is the single most useful signal for diagnosing the underlying bug, and it is lost if the telemetry only reports “worker crashed” — the wiring for that is in Production Error Telemetry.

Cost of recovering a worker that held a compiled WebAssembly module Three recovery paths measured on the same crash. A naive restart that recompiles everything costs about 95 milliseconds, most of it WebAssembly compilation. A restart that transfers a cached compiled module costs about 28 milliseconds. Restarting without replaying state at all costs 20 milliseconds but leaves the worker unable to answer, so the next request pays the full load anyway. Recovering a worker with a 1.2 MB WebAssembly module recompile everything ≈95 ms — 75 ms compilation transfer the cached module ≈28 ms no replay at all ≈20 ms, then the next request pays 0 32 64 96 ms Replay cost, not restart cost, is what makes a crash visible to the user.
Caching the compiled module on the page turns a visible stall into a recovery nobody notices.

Gotchas

Restarting inside the error handler synchronously. A worker that throws during module evaluation fires error immediately on construction, so a synchronous restart recurses until the stack overflows. Always schedule the replacement on a timer, even for the first attempt.

Leaking timers for settled requests. Clearing the timeout in the success path is easy to forget, and hundreds of stale timers keep their closures — and the request payloads — alive.

Treating messageerror as a crash. It means one message could not be deserialised, not that the thread is dead. It usually indicates a non-cloneable payload, which is a bug in the caller; restarting hides it. Log it distinctly.

A heartbeat that queues behind the work. If the ping is answered on the same message queue as the jobs, a busy worker looks dead. Either answer heartbeats on a dedicated MessagePort, or make tasks yield often enough that the inbox drains between slices.

Counting failures globally in a pool. Each worker needs its own counter, or one repeatedly failing member trips the breaker for a pool that is otherwise healthy — see Worker Pool Management for where the supervisor sits relative to the pool.

Performance Note

Restarting a module worker that loads a 240 kB script and compiles a 1.2 MB WebAssembly module costs about 95 ms on a 2023 laptop: 12 ms of construction, 8 ms of script evaluation, and 75 ms of WebAssembly compilation — which is why replay cost, not restart cost, dominates. Caching the compiled module in the page and transferring it to the replacement cut that to 28 ms, and made the recovery invisible to a user who was mid-interaction.

Frequently Asked Questions

How do I tell a crashed worker from a busy one?
You cannot, from the outside, without a protocol — a thread that is 400 ms into a synchronous loop is indistinguishable from one whose process was killed. Add a heartbeat the worker answers between tasks, or a per-request timeout, and treat a missed deadline as death. Both need the worker to yield periodically, or a long task will be misdiagnosed as a crash.
Why does an out-of-memory kill produce no error event?
When the browser terminates a worker under memory pressure, it does not run any JavaScript in that thread, so self.onerror never fires and often nothing fires on the page either — the thread simply stops answering. That is why liveness has to be inferred from missing replies rather than from an event, and why every in-flight request needs a timeout.

See also

Related Guides

Web Workers Architecture Communication Handling Worker Termination Gracefully In Spas Open guide