Workflows that survive restarts

Prior-authorisation intake waits: days for a fax, hours for a nurse's approval. A process that holds that wait in memory (a sleeping thread, an asyncio task, a polling loop) loses its place when the pod restarts, the deploy rolls or the region fails. Durable execution persists every step. You write the process as workflow code that only orchestrates, and put every side effect (model calls, tool calls, emails) in activities. The engine records each activity's result in an event history. After a crash, a new worker replays the workflow code against the history: completed steps return their recorded results instead of running again. For replay to work, workflow code must be deterministic: no wall clock, no randomness, no I/O, no LLM calls. Fourteen-day timers, signals and updates that deliver documents and approvals, and status queries all become durable. Activities still run at least once, so writes still need idempotency keys.

Key takeaways:

• Workflow code decides; activities do. Every model call is an activity.
• Replay re-runs your workflow code from the start against recorded results; any difference in the commands it issues is a non-determinism error.
• Waits are durable timers and signals, not sleeping processes. A case waiting 14 days costs nothing while it waits.
• Changing workflow code with executions in flight is a versioning problem. Test it with replay before you deploy.
• Durable execution gives you exactly-once orchestration, not exactly-once side effects. Idempotency keys and compensation remain your job.