Durable execution and fault tolerance

LangGraph · Expert · 16 min

Long agent runs will hit failures; durable design turns them into resumable pauses instead of lost work.

Durable execution means a run can stop at any point (crash, deploy, timeout) and later continue from its last saved step without redoing finished work. LangGraph gets this from checkpoints: resume a thread with graph.invoke(None, config) and it continues from the last checkpoint.

Durability modes trade speed for safety, set per run with durability=:

  • "exit": save only when the run finishes or pauses. Fastest; a crash mid-run loses progress.
  • "async" (default): save each step in the background while the next one runs. Small risk of losing the last step.
  • "sync": save each step before starting the next. Safest, slower.

Design for replay. Resumption re-runs the interrupted node from its start, so:

  • make side effects idempotent (safe to repeat), for example with an idempotency key the external API recognizes,
  • put each side effect in its own node (or @task), so a completed one is never repeated,
  • keep anything nondeterministic (random IDs, timestamps, model calls) inside nodes or tasks, so its result is recorded.

Per-node controls on add_node:

  • retry_policy=RetryPolicy(max_attempts=..., retry_on=...) retries transient failures with backoff.
  • timeout= (1.2+, async nodes) caps an attempt; TimeoutPolicy separates a total run_timeout from an idle_timeout that resets on progress.
  • error_handler= (1.2+) runs after retries are exhausted, receives a NodeError, and can return a Command to record the failure and route to a compensation step (the saga pattern: undo earlier steps).

For planned shutdowns, RunControl.request_drain() stops after the current step and saves a resumable checkpoint.

Example

from typing import TypedDict
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.errors import NodeError
from langgraph.graph import StateGraph, START, END
from langgraph.types import Command, RetryPolicy


class State(TypedDict):
    order_id: str
    status: str


def charge(state: State) -> dict:
    # real code: payments.charge(..., idempotency_key=state["order_id"])
    raise ConnectionError("gateway timeout")


def on_charge_failed(state: State, error: NodeError) -> Command:
    return Command(update={"status": f"failed: {error.error}"}, goto="refund")


b = StateGraph(State)
b.add_node("charge", charge, retry_policy=RetryPolicy(max_attempts=3, initial_interval=0.1),
           error_handler=on_charge_failed)
b.add_node("refund", lambda s: {"status": s["status"] + " -> compensated"})
b.add_edge(START, "charge")
b.add_edge("charge", END)
b.add_edge("refund", END)
graph = b.compile(checkpointer=InMemorySaver())

cfg = {"configurable": {"thread_id": "order-9"}}
print(graph.invoke({"order_id": "9", "status": ""}, cfg, durability="sync")["status"])
# failed: gateway timeout -> compensated

Remember

  • Checkpoints make runs resumable: invoke(None, config) continues a thread
  • durability = 'exit' | 'async' (default) | 'sync'
  • Make side effects idempotent and isolate them in their own nodes/tasks
  • RetryPolicy, timeout and error_handler (1.2+) handle failures per node

Watch out for

  • Generating a random ID in the entrypoint body or before an interrupt, so replays produce a different ID and duplicate the side effect.

Check yourself

  1. Which durability mode guarantees each step is saved before the next starts?
  2. A node sends an email and then crashes before finishing. On resume it runs again. How do you avoid a duplicate email?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: docs.langchain.com/oss/python/langgraph/fault-tolerance