Yazam.ai is a multi-agent LLM pipeline. It turns an immigration applicant's intake answers into a finished, fact-checked E-2 visa business plan, a narrative plus a five-year financial model, which a lawyer then files with USCIS. Some runs need more than one function invocation to finish. Three parts handle that: the lock that guards each job, a watchdog that notices when a job has lost its coordinator, and a checkpoint that lets work pick up where it stopped.
How long should a distributed lock TTL be for a serverless job?
The tempting answer is "a bit longer than a typical run." That ties the lock to a number you don't control, because runtime varies from job to job. The platform's kill deadline is a fixed, published value. On Yazam.ai that deadline is the 1800-second maximum function timeout. The job concurrency lock's TTL is set slightly longer than that ceiling, on purpose.
The stated reason is narrow and concrete. A hard-killed coordinator must never be able to hold the lock forever. A bounded TTL derived from the kill deadline gives every lock a guaranteed end. It does not depend on how long a particular job happens to take.
What happens when the coordinator is hard-killed mid-run?
In general, a hard kill gives the process no chance to clean up after itself. Anything it would have done on the way out, such as releasing a lock or updating a status field, may never happen. In that situation the TTL guarantees that the lock does not stay held indefinitely.
The job itself is a separate problem. Its record can be left marked 'running' with nothing behind it. A lock that has lapsed doesn't restart the work by itself. Something has to notice the job and act on it.
How does the watchdog detect and relaunch stuck jobs?
A watchdog runs every 2 minutes. It looks for jobs stuck in 'running' whose lock is dead, and it relaunches them. Matching on both conditions matters. A 'running' status alone doesn't tell you whether a coordinator is still working, and a dead lock alone doesn't tell you whether the job still has work left. The combination points to the case the watchdog exists for: a job that claims to be in progress but has no live coordinator.
Why does checkpoint-based resume have to be idempotent?
Relaunching a job is only safe if starting again doesn't redo or corrupt work that already finished. On Yazam.ai, resume is idempotent via the checkpoint. A relaunched job therefore continues from saved progress rather than from zero.
That changes what a timeout means. A run that outlasts one function invocation is not treated as a failure. It simply spans several invocations, and the lock, watchdog and checkpoint carry it from one to the next. The lock TTL, the watchdog and the checkpoint only work as a set. The TTL makes sure a dead coordinator lets go, the watchdog finds the abandoned job, and the checkpoint makes relaunching it safe.
What are the parameters in this design?
| Parameter | Value |
|---|---|
| Job concurrency lock TTL | 35 minutes |
| Maximum function timeout | 1800 seconds |
| Watchdog interval | 2 minutes |
| Stale-job sweep | 14 days |
The first two values are coupled. The TTL is derived from the timeout, slightly longer than it, rather than chosen on its own.
What should you copy into your own long-running pipeline?
- Start from the hard timeout. Look up your platform's kill deadline and set the lock TTL slightly above it. Your expected runtime is the wrong input for this number.
- Treat the two values as one setting. If the function timeout changes, revisit the TTL so that it stays longer than the kill deadline.
- Trigger recovery on two signals. Relaunch a job when it claims to be running and its lock is dead, not on either condition alone.
- Run the watchdog on a schedule. A killed process cannot report its own death, so something external has to check for abandoned jobs.
- Make resume idempotent before you automate relaunch. In general, automatic relaunch without safe resume risks repeating or corrupting completed work. With a checkpoint, a timeout becomes a handoff between invocations.