When an AI Agent Fails Mid-Task: The DN Runtime Recovery Test

The missing boundary is between remembering and doing

A workflow record can say “the next step is to create a payment request.” A receiver can already have created it. If the worker disappears between those events, its next instruction is ambiguous: continue, retry, reconcile or stop?

DN’s thesis is that recovery needs three separate contracts: a durable place in the workflow, a stable identity for the intended external effect, and current authority to perform it. A system can satisfy one contract while failing another. We built the local test to make those boundaries inspectable.

The workload contains no payments. Its “effect” is an entry in a separate local SQLite receiver database. That separation creates a concrete commit-before-acknowledgement window without putting real funds or business records at risk. The worker exits abruptly with code 86 at the selected boundary; a fresh process then resumes the same workflow identity.

The complete six-configuration result

Read the denominator correctly: a successful revocation test produces zero effects, while a successful restart or retry test produces exactly one. Combining the cells into a platform “success score” would conceal what was tested. Ten repetitions of a deterministic scenario do not establish a statistical production failure rate.

1. Restart: can the next worker find its place?

The first worker prepares a synthetic task, reaches a static breakpoint after that node and exits. The second worker uses the same thread identity. The durable configuration has a SQLite checkpoint; the comparison configuration has an in-memory saver that disappears with the first process.

In the final suite, the SQLite configuration completed one intended effect in each of ten trials without repeating preparation. The in-memory configuration had no checkpoint available after restart and completed no effects. Its recorded outcome was state unavailable.

This result is consistent with LangGraph’s persistence documentation, which distinguishes memory-only state from persistent checkpointing. DN tested a process restart after a prepared checkpoint. We did not cut power during a database write or measure a cloud service’s recovery objective.

2. Uncertain outcome: does a retry create another effect?

Both configurations in this scenario use SQLite checkpoints. The receiver commits an effect; the worker exits inside the action node before returning success to the graph. The resumed graph re-enters the pending action. What changes is the application’s request identity.

With a new key for the retry, the receiver recorded two effects per trial: 20 across ten trials. With a stable key and a receiver uniqueness constraint, it recorded one per trial: ten across ten. Durable workflow state alone did not supply that receiver contract.

The fixture uses a deliberately simple mapping: one task, one intent, one key. A real integration must bind the key to the caller and intended payload, handle conflicting reuse, define retention and expiry, and reconcile receipts. We did not test those additional cases. A key placed in a request has no deduplication power unless the receiver enforces the corresponding semantics.

AWS’s idempotent API design discussion explains why a missing response leaves uncertainty about a side effect. The DN demonstration applies that principle to a resumed workflow. It does not establish exactly-once execution across arbitrary distributed services.

3. Revocation: is yesterday’s permission still accepted?

The workflow prepares while its policy permits the action, then pauses. The runner changes the policy to deny before the second worker starts. Both configurations have durable checkpoints. One trusts the cached allow decision in workflow state. The other reads current policy at the simulated receiver, inside the same local transaction that would commit the effect.

The cached configuration committed an effect after revocation in all ten trials. The receiver-recheck configuration returned blocked and committed no effects in all ten. Remembered approval was therefore insufficient for this fixture’s current-authority rule.

The receiver transaction is an important constraint. A permission check in the client followed by an unrelated remote write can still leave a race. This test changed policy between phases; it did not exercise concurrent policy changes, token expiry, disconnected policy caches or a real service’s cancellation latency. OWASP’s authorization guidance supports checking permissions on every request, but does not turn this small suite into a complete access-control audit.

python -m venv .venv
.venv/bin/python -m pip install -r requirements-lock.txt
.venv/bin/python recovery_test.py run --repeats 10 --output my-results.json

A practical recovery acceptance card

Before expanding a workflow’s authority, record the following for its actual tool and receiver. This is a manual evidence checklist; it does not scan your system or certify it.

What the controls cost—and what we did not price

Persisting checkpoints creates storage and lifecycle work. Receiver-side deduplication needs retained request identities and a policy for conflicting requests. Current-authority checks need an enforcement point that can reject the effect. Those costs should be accounted for, rather than hidden behind a model-token comparison.

This release contains no latency or cost benchmark. The test used no paid model or managed runtime call, but that does not make real deployments free. Use the Agent Cost Stack to account for storage, tools, retries, verification and human review. Use the Runtime Portability Index to examine whether recovery state and receiver contracts survive a platform change.

Where this evidence fits in DN’s research

The Managed Runtime Dataset v0.1 remains a documentation baseline. This is a separate, versioned local-framework execution release. It adds a reproducible method and an actual receipt; it does not silently replace empty managed-provider measurements with local results.

The experiments complement Model Intelligence vs Harness Intelligence and Event-Driven MCP. They show why workflow state, effect identity and authority deserve separate acceptance tests. They do not change existing Agent Trust scores, classifications or platform rankings.

Methodology, limitations and change log

Release v0.1, 21 October 2026. The final suite reports all 60 trials: three scenarios, two selected configurations each, ten serial repetitions. Each trial uses fresh local databases and two worker processes. A first-phase exit must return code 86; a restart must return code zero. Setup or timeout errors abort the suite rather than silently excluding a trial. Development smoke runs are separate from the final published receipt.

The runner reads the simulated receiver database after execution and counts committed effects. It also records preparation count and the worker outcome. A restart passes only with one effect, one preparation and an accepted result; an uncertain-outcome retry passes with one effect and an accepted result; revocation passes with zero effects and a blocked result. The receiver ledger is separate from the graph’s success state, but remains DN-controlled local instrumentation—not an external auditor.

The worker really exits, including inside a node after a receiver commit in the uncertain-outcome case. The other cases exit after a static breakpoint. We do not test every possible crash boundary. Checkpointer configuration uses synchronous durability; the receiver uses local SQLite transactions. SQLite’s atomic-commit documentation describes database guarantees and their assumptions; no power-loss experiment was performed here.

One local host and one framework version cannot establish cross-platform superiority, a production incident frequency, model decision quality, cloud portability or comprehensive payment security. Concurrency, partitions, conflicting payload reuse, key expiry and real financial settlement remain untested. Future adapters should preserve these acceptance rules while reporting their own model, harness, receiver and infrastructure versions.

Primary sources and original evidence

• DN final execution receipt and reproduction instructions.
• LangGraph: persistence and memory-only checkpoint limitations.
• LangGraph: interrupts, thread identity and resuming execution.
• Amazon Builders’ Library: making retries safe with idempotent APIs.
• OWASP: authorization checks and enforcement location.
• SQLite: atomic commit and environmental assumptions.

Frequently asked questions

Disclosure and licensing: No sponsored placement or affiliate link appears in this research release. No vendor funded the local run. DN original source is MIT-licensed; DN original results are CC BY 4.0. Third-party packages and documentation retain their own rights. This is educational research, not investment advice or a security certification. Send reproducible corrections through DN’s contact page; never submit credentials or private customer data.