The job that died without dying

Stuck at processing

A process can be killed between "I have started" and any possible "I have stopped". That sounds obvious written down, but it breaks a pattern almost every queue starts with, which is a status column that only the worker itself ever writes.

I found this the way you usually do. A row in a job queue sat at status processing for about twenty-three hours. The job was a long-running generation task on a serverless platform, and the code that should have marked the row failed was written, tested and correct. It just never ran.

The code looks fine, which is why it's worth showing.

await markProcessing(job.id);
try {
  await doTheLongThing(job);      // ← killed here, at the platform's cap
  await markDone(job.id);
} catch (err) {
  await markFailed(job.id, err);  // ← never reached
}

A catch block defends against a throw inside the process. The platform hard-kills a function at its 300-second cap, and a hard kill isn't a throw. There's no unwinding, no finally and no last write. The last thing the row says is "something picked me up", so it just stays at processing forever.

So the first principle is that any state a process can enter needs a way out that doesn't rely on that process still being alive.

The recovery depended on the thing that had stopped

There was already a reaper, a function that finds rows sitting in processing past a cutoff and marks them failed. That's the right idea, and it was already written.

It ran at the top of the queue processor, so stale rows got cleaned up whenever the queue was next poked.

That reads fine, and it's wrong in the one situation it exists for. A queue with nothing new arriving never runs its processor, so it never reaps itself. And a queue nobody's using is also the one most likely to have lost a worker without anyone noticing, so the recovery was weakest right where it was needed most.

The second principle follows from that, which is that a recovery mechanism mustn't share a failure mode with the thing it recovers.

Three separate fixes

The fix was three changes, and I kept them separate from each other.

An independent sweeper. The reaper now also runs from a daily scheduled task that exists for a completely unrelated reason and knows nothing about the queue. It runs alongside that task's own work and reports how many rows it reaped.

I attached it to an existing schedule instead of adding a new one. The host job has its own reason to fire every day, and if it stopped, the people watching something else entirely would notice. That's what I wanted, a sweeper that keeps running even if generation stops.

A step written twice. The job payload carries a step string, set once when the row is created and again at the moment the processor picks it up.

// at row creation
payload: { step: "Queued — waiting for processor" }

// at the moment status flips to processing
data: {
  status: "processing",
  payload: { step: "Starting generation..." },
}

Writing it at creation is the half that's easy to skip, and it carries most of the diagnostic value. Without it, a row with no step is ambiguous. It might never have been picked up at all, or it might have been picked up and died before it could say anything, and those are different bugs with different investigations. It's two cheap writes, both landing before anything risky happens, so a stuck row now tells you which step it got to.

Timeouts below the cap. The job calls out to three external API providers, and none of them had an explicit per-request timeout. A hang in any of them would run until the platform ended the function.

const PROVIDER_A_TIMEOUT_MS = 120_000;
const PROVIDER_B_TIMEOUT_MS = 90_000;
const PROVIDER_C_TIMEOUT_MS = 90_000;

// platform function cap: 300_000

The whole point is in that ordering. Every per-request timeout sits strictly below the platform's function cap, so a hung call throws inside the process instead of being killed from outside it. The existing catch (the one that was always correct and never reached) now catches it, marks the row failed and records a real error message. I didn't touch the error handling. I just moved the failure somewhere the catch could see it.

The timeouts bound each call, not the whole job. Three slow calls in a row that each stay under their timeout can still reach the cap, and no per-request setting can prevent that. That's why the sweeper is a separate fix I couldn't have skipped.

Why three and not one

Any one of these fixes would have closed the ticket, and none of them covers the others.

The timeouts turn the common case (a slow external dependency) into an ordinary handled failure. The step field doesn't prevent anything, but it makes whatever still gets through explainable in ten seconds instead of an afternoon. The sweeper catches everything else, including cases I haven't thought of yet. It's also the only one of the three that holds if the worker gets killed for a reason that has nothing to do with the external calls, like a deploy mid-flight, an out-of-memory kill or a platform incident.

They also fail independently. If the sweeper's schedule is misconfigured, the timeouts still work. If a fourth provider gets added without a timeout, the sweeper still cleans up after it.

I think both principles are more about who owns the state than about serverless. If only the worker can update its own status, a dead worker leaves it stuck, so something outside the process has to be able to move a row out of "in progress" based on elapsed time alone, because once a process is gone that's the only signal you've got. The reaper inside the processor had the same problem as a retry queue on the failing host or a health check served by the service it checks. They all look fine, but each one depends on the thing it's meant to rescue.