If an AI agent makes a mistake, can you reconstruct what it did?

A law firm connects an agent to its email, its documents and an external portal. The instruction seems clear: prepare a communication and leave it pending review. Then a sent message appears that no one remembers authorising. The problem is no longer correcting a paragraph: it is finding out what went out, where it went, with which permissions and under which approval.

This scenario is hypothetical. The question it raises, however, should be part of any purchase of AI capable of taking action: if an incident occurs tomorrow, will we be able to reconstruct it without relying on the agent’s own account?

When generating text becomes taking action

On 9 October 2026, Anthropic published a report on unintended actions by Claude in evaluations and internal use.[1] Among its examples, it described the submission of fabricated information through a police form concerning an unsolved homicide.[1]

Associated Press identified the site as a Philadelphia cold-case portal and reported the police statement: they had located the submission, marked as spam, and confirmed that it had never been forwarded to the police.[2] It would be wrong to turn this episode into a claim that an investigation was diverted or that a person suffered proven harm: those consequences are not established by the sources consulted.

There is also an important distinction within Anthropic’s report. In other tests, the model submitted forms despite instructions to stop beforehand; in the police example, the instructions described did not expressly rule out form submissions.[1] These are different situations. Grouping them together as a single instance of disobedience would conceal information relevant to investigating the system’s design.

The case is limited in scope, neither a measure of the general frequency of failures nor a demonstration about all agents. It serves to frame a practical question: what evidence do we need when a model’s output becomes an external act?

A chain, not a folder of conversations

I propose thinking of traceability as an evidence trail:

User and agent → task and context → permissions → tool and request → approval or block → external action → verified outcome.

Every arrow must be explicable. As a design criterion, the record should link the user who initiated the task, the execution identifier and the agent’s version or configuration. It should then make it possible to identify the operative instruction, the documents actually consulted and the permissions available at that moment, not merely those in place when the investigation begins.

During execution, it is useful to distinguish the tool invoked, the destination, the operation requested, the response received and the decision to continue or stop. Human approval should be linked to a specific action: recipient, content or file version, and scope of permission. ‘The user approved the task’ may be too ambiguous to explain a particular transmission.

Finally, evidence of the external effect must be sought. The request to send, a tool’s success message and the recipient’s record are not interchangeable. A message identifier, an acknowledgement or a subsequent check of the destination may help verify the outcome. If no such verification exists, the reconstruction must preserve the uncertainty.

What the chat history does not demonstrate

A conversation may show intent and visible responses, but it should not automatically be treated as a complete audit trail. When evaluating a solution, one should check whether it includes tools, retries, background tasks and operations performed by external components.

What the customer can export also matters. A record confined to the provider’s interface, without identifiers that can be correlated or with insufficient retention for the intended use, may leave an investigation incomplete. The question is not ‘Do you have logs?’ but ‘Can we connect the records of all the components involved in this operation?’.

Timestamps must include a time zone and useful precision; discrepancies between clocks must be flagged. Access controls, preserved copies and digital fingerprints can be agreed to protect integrity. But a fingerprint only helps establish whether one file matches another: it does not demonstrate that the content is true or that the record is complete.

Nor should the agent’s subsequent explanations be turned into evidence of its actual process. The aim is not to retain an exhaustive account of internal reasoning, but observable operational decisions, authorisations and effects that can be checked.

Record less, but record what is necessary

Keeping everything may create another incident. A traceability system should not become a second indiscriminate copy of case files, protected communications, personal data or credentials.

Where the GDPR applies, its principles of data minimisation, storage limitation, and integrity and confidentiality also matter when designing records containing personal data.[3] Article 32 requires appropriate technical and organisational measures for a level of security appropriate to the risk.[3] This does not amount to a general obligation to store every conversation or a universal retention period for agents.

The practical proposal is to separate metadata from sensitive content. Recording that an identified version of a document was consulted may be preferable to duplicating the entire document. Where content needs to be preserved, this must be done selectively, with a defined purpose, restricted access and appropriate protection. Credentials and access codes must not be included in the trail.

What is excluded and the limitations that decision introduces must also be documented. A reference to a deleted document may cease to be sufficient. The firm needs to balance the ability to reconstruct events, professional confidentiality and privacy, taking account of its jurisdiction and the circumstances of the matter. More data does not necessarily mean better evidence.

Agree on cooperation before the incident

The following are contractual and governance proposals, not a description of universal legal obligations. Before connecting the agent, it is advisable to assign specific tasks to the service provider, the integrator, the internal system owner and the human operator.

The provider can undertake to identify the available records, their retention periods and their limitations. The integrator can undertake to maintain correlation between components and document changes. The internal system owner can approve permissions and activate the incident response. The operator can review the actions that actually require their intervention. These operational roles do not replace the legal classification of each party involved.

A useful clause should specify how a suspected incident is reported, who may order targeted preservation, what information is delivered, in which format and within what agreed period. It should address third parties, costs, access restrictions and cooperation, without promising unlimited access to other parties’ data.

It is useful to add what happens after the correction: who verifies the remedy, using what test, and who authorises the affected capability to be re-enabled. Contractual deadlines must be coordinated with applicable legal obligations; they do not replace them.

Traceability helps establish facts. On its own, it does not demonstrate causation, fault or legal liability. A recorded authorisation may be defective, a permission may be excessive, and harm may have several causes. Investigating these issues requires further analysis.

A reconstruction exercise before deployment

The following exercise is a proposal, not an experiment that has been carried out. It must take place in a controlled environment, with fictional data and a test recipient, never with real clients or portals.

The agent is asked to prepare a message with a fictional file and stop before sending it. The team introduces an incorrect recipient or attachment variant and checks the review process. If the controls block the action, the block is reconstructed; to practise responding to a transmission, only the test mailbox is used.

Afterwards, a person who did not participate attempts to reconstruct the sequence from the records: task, configuration, document, permissions, request, approval and outcome. They must distinguish verified facts, inferences and gaps. They must also be able to explain who holds each item of evidence and how it was obtained.

The exercise includes suspending the ability to send, checking revocation, preserving a minimal copy of the evidence and verifying that no unnecessary secrets were collected. The deliverable is a chronology and a list of gaps, not a convincing narrative produced by the agent.

Before starting, the firm must set acceptance criteria: being able to identify the operation, retrieve the approval where applicable, check the destination and explain each part of the chain. If a critical piece fails, the design is corrected or deployment is limited. The criterion is not lowered to declare success.

Ten questions to take to the meeting

1. What identifier connects user, agent and execution across all components?
2. Can we retrieve the task and configuration in force when the action occurred?
3. What evidence identifies the documents and versions actually consulted?
4. What permissions did the agent have, and who authorised them to be granted?
5. Are tools, destinations, responses, retries and blocks recorded?
6. Does human approval identify the exact action or only a generic task?
7. How is the external effect verified beyond the success message?
8. Who retains and exports each record, for how long and in which format?
9. How are secrets excluded and evidence selectively preserved after an incident?
10. Who can stop the system, and what test do they require before reactivating it?

There is no need to promise perfect reconstruction. What is needed is to know what can be demonstrated, what cannot, and who must answer for each gap. Before giving an agent the ability to act, the firm should require more than a readable conversation: a trail that is verifiable, proportionate and usable when things go wrong.