Reproducing an agent run is not as straightforward as rerunning the same application code. An agent’s behavior may depend on the model response it received, the context assembled at that moment, the state of external tools and data, stored memory, available permissions, and any human decisions made during the run. By the time someone needs to investigate an unexpected action or compare it with a later execution, several of those conditions may have changed.
That raises the practical question - what would a system need to record if we wanted to reconstruct an earlier agent run well enough to understand what happened, and perhaps reproduce parts of it later?
For conventional software, reproducing a problem often starts with the code version, configuration, inputs and environment. Those still matter for an agent, but they describe only part of the execution. The agent may also have retrieved information that has since changed, called an external service whose state is different today, read memory that was later updated, or received a model response that a fresh call will not reproduce exactly.
Reproducibility for agents therefore needs to account for the wider execution around the application itself.
What does reproducibility mean for an agent?
There are several different things we might mean when we say that we want to reproduce a run.
The first is reconstruction: being able to inspect the earlier execution and establish what inputs, state, decisions, tool calls and external results contributed to the outcome.
The second is replay: taking recorded state and results from an earlier execution and using them to reproduce some or all of its execution path.
The third is a fresh rerun: executing the agent again, potentially against current models, tools and external data, to see what it does now.
These are related, but they are not interchangeable.
Durable workflow systems already make this distinction explicit. Temporal records the results of external Activities in workflow history. If the workflow is replayed, completed Activities are not called again; their recorded results are reused so that the workflow logic can reconstruct the same history.
Agent frameworks can expose a similar difference. LangGraph checkpoints graph state during execution and can resume from an earlier checkpoint. Its time-travel documentation notes that work completed before the checkpoint does not run again, while later nodes do, including model calls and API requests that may now produce different results.
For agents, it is therefore useful to separate the question of whether we can understand an earlier run from whether we can make today's system generate the same behavior.
The execution record needs more than the prompt
Consider an agent handling a support request. It reads the user's message, retrieves account information and policy documents, sends selected context to a model, decides that a refund is appropriate, asks for approval and then calls a payment tool.
Keeping the original user request and final response would preserve only the outside edges of that process.
To understand the decision later, an investigator may need to know which instructions were active, which model was called, which material was placed into context, what the retrieval system returned, which tools were available, what arguments were sent to them and what came back. If an action depended on stored memory, authorization or a human approval, those are also part of the conditions under which the run took place.
Current GenAI observability work in OpenTelemetry already reflects part of this execution surface. Depending on configuration, telemetry can include model information and can also capture system instructions, prompt messages, tool definitions, tool arguments and tool results. OpenTelemetry leaves detailed content capture disabled by default in its example instrumentation because this information may be sensitive.
Reproducibility depends on knowing which parts of the execution may need to be identifiable later. That does not mean every agent should retain every input indefinitely.
Versions matter beyond the application code
A source commit or application build identifier is an obvious starting point, but agents depend on components that can change independently of the code that orchestrates them.
Instructions may be edited without an application release. A tool can keep the same name while its schema or implementation changes. A retrieval system can return a different version of a document. Authorization policy can be updated. A model alias can point to different underlying deployments over time.
A useful execution record should therefore distinguish changing dependencies where the system supports that distinction.
For the model, this might include the requested model identifier, relevant generation settings and any model or deployment identifier returned by the provider. For tools, it might include a schema or implementation version. Retrieved material can carry a document version, stable identifier or content hash. Agent configuration and instructions can be tied to their own version rather than treated as an unchanging property of the application.
None of this guarantees that a fresh run will produce the same result. It does, however, make it possible to identify which parts of the environment were the same and which were not.
External results belong to the history
External systems create one of the clearest reproducibility problems.
Suppose an agent checks stock before deciding whether an order can be fulfilled. During the original run, the inventory tool reports three units available. When the same request is investigated a week later, the tool reports 27.
Calling the tool again has not reproduced the original condition. The response returned during the original run is part of the evidence needed to understand the decision.
The same applies to search and retrieval. Preserving the query tells us what the agent asked for, but not necessarily what it saw. If the search index, underlying documents or ranking system have changed, a fresh search can produce a different context.
For an investigation of an earlier run, the useful record is therefore the result that was actually supplied to the agent at the time. A current rerun remains useful, but it answers a separate question about how the agent behaves under current conditions.
Keeping those two purposes separate makes comparisons more informative. If the new execution differs, the team can ask whether the agent itself changed or whether it was operating on different external information.
State and memory are inputs too
Stateful agents add another variable because the current state of the system may not represent what the agent knew earlier.
Conversation history, workflow state, previous tool results, user information, summaries and longer-term memory can all influence later decisions. Some of those records may be updated or replaced after the execution being investigated.
LangGraph's persistence model illustrates one way of retaining this history. Its checkpointers save graph state during execution, organized into threads, which supports resumption, fault recovery and inspection of earlier states. Separate stores can retain information outside an individual thread.
For reproducibility, the important information is not only the latest stored value, but the state that was visible when a particular decision was made. If a memory item was corrected the following day, inspecting the corrected version does not explain the decision that used the original one.
This is one reason checkpoints or versioned state can be useful even when an application already has conventional logs.
Permissions can change the execution path
An agent with permission to read a system can behave differently from the same agent with permission to modify it. A user may be allowed to request an action while another can approve it. An external service might reject the same tool call later because authorization policy has changed.
The effective permission context is therefore part of the conditions of a run.
Recording that context does not require putting passwords, tokens or other reusable secrets into telemetry. What matters for later analysis is the relevant identity or role, the authorization decision that was made, and whether an approval or policy check affected what happened next.
Without that information, a later rerun can appear to show changed agent behavior when the real difference is that the agent is operating with different authority.
Human decisions need to remain visible
Human intervention can also alter an execution in ways that cannot be reconstructed from model and tool events alone.
A reviewer may approve a proposed transaction, reject an action, edit a plan or provide information that allows an interrupted task to continue. If later steps depend on that intervention, it belongs in the execution history.
The appropriate level of detail will depend on the system. Some applications may need to retain the identity of the approver, while others need only the role, decision and time of approval. The important point for reproducibility is that an investigator can see that the execution path changed because of a human event rather than attributing the change to the agent itself.
More telemetry is not automatically better
One obvious but problematic response to all of this is to store everything.
Agent executions can contain personal information, confidential documents, tool arguments, retrieved records, model prompts and business data. Retaining complete execution histories indefinitely can increase both cost and security exposure.
OpenTelemetry's approach to GenAI content reflects that tension. Basic telemetry can retain operational metadata without automatically capturing complete prompts and tool content, while more detailed capture can be enabled where appropriate.
A reproducibility strategy therefore also needs a retention strategy.
A team might keep detailed execution content for a limited period, retain references or hashes for longer, and preserve selected metadata after the underlying content has been deleted. Higher-impact actions may justify a stronger historical record than low-risk conversational interactions.
The right amount of evidence depends on what the system needs to explain later, not on how much data it is technically possible to collect.
Testing how reproducible a run is
One practical way to assess an existing agent is to choose a completed run and assume that the surrounding environment has since changed.
Could someone still establish which application and agent configuration were running, which model was used, what instructions and context it received, what state was available, which tools it called and what those tools returned?
Could they determine which permissions applied, whether a human decision affected the run, and which external actions succeeded?
More importantly, could they distinguish information preserved from the original execution from information being fetched from today's system?
Gaps do not necessarily mean that every missing field should immediately become permanent telemetry. They identify the point at which an investigation would begin relying on assumptions rather than historical evidence.
Reproducibility is not the same as determinism
For many agent systems, expecting a fresh execution to reproduce every earlier model response, tool result and decision exactly is unlikely to be a useful standard.
A more practical objective is to preserve enough evidence to understand the original conditions and to explain meaningful differences between executions.
If an agent behaves differently six months later, an engineer should have some way to determine whether the model changed, the instructions changed, the retrieved context changed, a tool changed, permissions changed, stored state changed or the outside world simply returned a different answer.
That level of reproducibility does not make an agent deterministic. It makes its history more inspectable, and gives teams something more useful than rerunning the system and hoping that the same behavior appears twice.
Share
Author
