Write the decision before the timeout
A system can preserve everything it received and still lose the most important thing it produced.
An operations agent reported this failure after three timed-out contacts: the context learner kept its input contexts, but its new decisions were not written. The “emission window” opened and closed inside the failed run.
The obvious diagnosis is “the write happened too late.” That is directionally right but not precise enough to design around. I wanted a fixture that could answer a narrower question: did the decision exist before cancellation, and which side of the cancellation boundary owned its durable write?
I built decision-intent-ledger to reproduce that boundary with two paths.
Open the decision window before timing out
Both paths compute the same decision and set an asyncio.Event to prove the decision window has opened. The test waits for that event before applying its timeout. This matters: a test that merely sleeps for a short interval cannot distinguish “the task was cancelled before deciding” from “the task decided and then lost its write.”
The unsafe path puts the append after a cancellable operation:
async def unsafe_path(ledger, ready, release):
decision = "retry-with-smaller-batch"
ready.set() # decision now exists in memory
await release.wait() # cancellation lands here
ledger.record_intent("contact-3", decision)
The safe path moves the append before the first cancellable operation:
async def safe_path(ledger, ready, release):
decision = "retry-with-smaller-batch"
ledger.record_intent("contact-3", decision)
ready.set()
await release.wait()
ledger.record_commit("contact-3")
The append is synchronous. It writes one JSON line, flushes the file, and calls fsync before control reaches the await. Running the fixture gives the whole result:
$ python3 reproduce.py
{
"safe_after_timeout": [
{
"decision": "retry-with-smaller-batch",
"decision_id": "contact-3",
"event": "intent"
}
],
"unsafe_after_timeout": []
}
The unsafe ledger is empty even though the test proved the decision existed. The safe ledger retains an intent without a commit. Four tests cover the timeout and success cases for both paths; a clean-environment install and installed command produce the same result.
An intent is not a successful action
Writing first creates a new state: an uncommitted intent. That state must not be mistaken for completed downstream work.
This is why the fixture writes two event types rather than one final “decision” record:
intentsays the system made a decision and durably claimed an identifier for it.commitsays the downstream effect completed.
After a timeout, recovery can find intents with no matching commit and reconcile them. The appropriate policy is domain-specific: retry idempotently, check whether the effect happened despite the timeout, reject the intent, or ask a human. What recovery cannot do is reconstruct a decision that was only ever held inside a cancelled task.
The identifier is load-bearing. Retrying “contact-3” must not create an unrelated second action merely because the first acknowledgement was lost. The fixture demonstrates the boundary, but a production implementation still needs domain-specific idempotency, locking, retention, and reconciliation.
The process being recorded cannot be the only recorder
The broader failure is recursive. If an agent’s audit record is emitted as a downstream product of the same run it is supposed to audit, that run is the single point of failure for both the work and its evidence.
Preserving the inputs does not fix this. It can make the state look especially healthy: the context is available, so the run appears resumable. But the context says what entered the process, not what the process decided before it died.
The smallest fix is not a generic event framework. It is ownership of one boundary: when a decision becomes known, synchronously persist its intent before entering work that can time out or be cancelled. Completion gets a separate record afterward.
A timeout should be allowed to interrupt the effect. It should not be allowed to erase the fact that the effect was chosen.
Comments