Evaluating LLMs Under Production Parity: A Replay Pipeline for Safe Model Swapping in Conversational Agents

Providers release new model families every few months and discontinue previous ones within short timeframes, without necessarily preserving the same cost, latency, or success rate. For anyone operating a conversational agent in production, each such change raises the same question: does this new model behave like the one already validated?

Public benchmarks don't answer that. They measure the model against generic answer keys, not against your product's behavior, or using your skills, your tools, your tone of voice, and your transactional flows. We built a pipeline to answer this empirically, and the most interesting finding ended up being not about any particular model, but about the design of the verdict.

TL;DR

• A replay engine re-executes curated, approved synthetic interactions (Base 0) under full operational parity with production, isolating the test variable to the model swap (LLM-swap).
• We evaluated 8 models (GPT-4.1/5 families, Gemini 2.5, and the open-weight Kimi-K2.5 and GPT-OSS-120B) across hundreds of runs. Three were approved: GPT-5.4 mini, GPT-5.4 nano, and Kimi-K2.5.
• Three others had a Final Score above 79% and were rejected only for exceeding the Hallucination Rate threshold, an eliminatory gate applied afterward and independently of the weighted average.
• Recalculating the verdict run by run, two rejections turned out to be statistical ties and one turned out to be structural. A single aggregated verdict does not justify discarding a model.

In agents that make decisions about real customers, safety criteria need to be independent filters, not just another component diluted into a weighted average. In an average, a serious failure can be offset by good performance on other dimensions and still result in a satisfactory score.

---

1. The Platform the Pipeline Runs On

The platform separates what the agent should do from how the model decides to do it. Expected behavior is declared in a Skill (a YAML with instructions, topics, and steps), and integrations with external systems are exposed to the model as Actions, Python functions it invokes when the conversation requires a concrete operation rather than just text. It is this separation that allows swapping the underlying model without touching the business logic.

On top of this foundation, the test follows three invariants. The first is that the Skill, the Actions, the memory manager, and the prompt remain unchanged: it is literally the same prompt validated in production, not a simplified version for research. The second is that the approved conversation is not resubmitted to a live agent — a replay engine reconstructs the exact context in which each turn happened and re-executes only the LLM's decision under test, which includes filling memory and time in the system prompt with the real values from that moment, restoring the baseline's initial tools and messages, dynamically rebuilding the tool surface on every Skill switch, filtering history after a transfer between agents, and applying the same termination rule. The third is that the only variable modified is the LLM that decides each turn.

It's worth emphasizing that this is not prompt injection: at no point does the reference script or the expected response enter the prompt as an instruction. The candidate receives the same informational context it would have in production — user data, conversation history, available tools — and from there decides on its own how to act. Any variation in performance, latency, or cost is therefore attributable to the model swap, not to differences in the reconstructed context.

There is a single point where fidelity is deliberately broken, and that is tool execution, which does not happen live. It would be neither safe nor reproducible to actually cancel a policy during a test: the action would have a real, irreversible effect on the customer's system, and the result would depend on that system's state at the time of the test, not at the time of the original interaction. That's why, when the candidate's call matches the expected step, the pipeline injects the real, historical response recorded in production; when it diverges, it falls back to a simulated response that does not claim success — it reports that the tool failed and that the interaction should continue with the information available, discouraging a new attempt as if the first one had worked. Combined with a cap of five tool calls per turn, this choice prevents a divergent call from degenerating into a loop.

2. Platform Architecture

Before describing the methodology, it's worth understanding the platform it operates on, because it is this architecture that makes the LLM-swap experiment possible.

The Tech4.AI platform separates what the agent should do from how the model decides to do it. Expected behavior is declaratively described in a Skill (YAML); external integrations are exposed to the model as Actions (Python functions via tool call). This separation allows swapping the underlying model without changing the instance's business logic.

A conversation's scope is not fixed: a skill-redirection Action hands control over to a different Skill, dynamically rebuilding the Action surface offered to the model. This behavior is faithfully replicated by the replay engine.

3. Methodology

Testing a candidate model follows simple logic: reconstruct exactly the situation in which the reference agent handled the interaction, swap out only the model behind the decision, and compare the result against the approved response.

3.2 Simulated Tool Execution

Action execution is not live. When the candidate's call matches the baseline's expected step, the pipeline injects the reference response for that step. When it diverges, it falls back to a simulated failure response that discourages a new attempt at the same action as though it had worked.

This choice, combined with a recursion cap on tool calls per turn (MAX_TOOL_CALL_RECURSION = 5), prevents a divergent call from degenerating into a loop within the same turn.

3.3 Batch Execution and Concurrency

The pipeline processes multiple session-model combinations simultaneously via real threads, which are more reliable in this synchronous context than asyncio.

Since more recent families (GPT-5) operate under more restrictive request quotas on Azure, the pipeline implements an adaptive concurrency cap: it automatically reduces parallelism only for those families (a maximum of 4 concurrent runs), while preserving greater parallelism for the other models. Transient failures are retried with exponential backoff.

3.4 Metrics Engine

The calculation combines two strategies.

Deterministic metrics, reproducible and invariant across runs:

• Tool Calling Accuracy (TCA): syntactic validity of the JSON, existence of the action in the catalog, parameter typing, and equivalence of the tool sequence with the reference transcript.
• Task Completion Rate (TCR): a continuous score between 0 and 1, a weighted average of step completion (45%), path efficiency (30%), and closing satisfaction (25%).
• Business rules: validations via regular expressions or fixed logic.

Non-deterministic metrics, which require semantic interpretation:

• LLM-as-a-Judge: assessment of how well the complete session fits the user's intent and Base 0.
• Semantic Similarity: a fallback within TCA for free-text narrative parameters, via token overlap or embedding similarity.
• Consistency evaluation: multiple generation rounds with the same input, to measure response variance.

When a tool diverges from the expected step, the deterministic base is supplemented by a tool-competence judge, whose score forms part of the TCA and TCR scores. This way, a model that solves the problem via a different but sensible path is not penalized by the rigidity of the deterministic comparison.

3.5 Judge Panel (LLM-as-a-Judge)

The protocol requires the judge model to have superior reasoning capabilities and a different architecture from the models being evaluated, in order to mitigate self-preference bias. The panel's temperature receives no special research adjustment (JUDGE_LLM_TEMPERATURE, default 1).

Score combination follows two rules: quality and session verdict use the median across judges; hallucination detection uses the most severe classification found among them, prioritizing safety over statistical convenience.

The judging prompt combines quantitative session data, a verdict rule (golden rule: "poor" reserved for serious failure, "adequate" is the minimum if the agent resolved the request), explicit handling of masking artifacts, and structured JSON output on a 1-to-4 Likert scale.

4. Dataset (Base 0)

Base 0 is the reference corpus: validated synthetic interactions, with PII replaced by fictitious but valid values — CPF and CNPJ numbers get a correct check digit, and the substitution is deterministic within each session, so the same real datum always turns into the same fake datum, preserving narrative coherence while never reintroducing the original.

Automatic customer approval, however, is a necessary but not sufficient condition: not every approved session demonstrates strict adherence to the agent's expected behavior. Each of the 106 initial sessions then went through manual, turn-by-turn review against a four-dimension protocol — skill routing, adherence to the system prompt, message sequencing, and correctness of tool calls. 20 sessions remained, distributed across four service instances, averaging 7.85 user turns per session (maximum of 33) and 10.5 tool calls (maximum of 32). The reduction is not a loss of coverage — it ensures that the studied baseline dataset is reliable enough that the research won't reproduce noise.

5. Approval Criteria

Four evaluation dimensions: cost (Cost per Session, including reasoning and tool cycles), latency (a 30-second cap per turn), resolution capability (Pass Rate, requiring TCR, TCA, and factual integrity), and behavioral quality (PT-BR experience and Hallucination Rate).

The Final Score (S) is calculated as:

S = 50% · Q + 20% · L_norm + 30% · C_norm

where Q, L_norm, and C_norm are scores normalized from 1 to 4. The decision matrix first evaluates the hallucination gates, which reject the model regardless of its other scores, then the APPROVED criterion (Score ≥ 75%, Pass Rate ≥ 70%, critical Hallucination Rate = 0%, overall ≤ 2%), then PARTIAL, with REJECTED as the residual case.

6. Providers and Technical Adaptations

The eight models were accessed through two providers. Seven of them ran via Azure AI Foundry: the entire OpenAI family (GPT-4.1 nano, GPT-5 mini, GPT-5 nano, GPT-5.4 mini, and GPT-5.4 nano), plus the open-weight models GPT-OSS-120B and Kimi-K2.5, each configured as a deployment in the Foundry catalog. Gemini 2.5 Flash was the only one served by Google Cloud, via Vertex AI. In addition, a few adaptations were needed to make these models work:

• Vertex AI uses its own message and function calling format, requiring bidirectional translation.
• Azure, GPT-5 family: refuses any temperature other than 1 and requires a different parameter name for output tokens.
• Azure, GPT-OSS-120B: natively uses the Harmony format, incompatible with the rest of the pipeline's chat completions standard. It was the model with the highest rate of turns with no recognizable decision.

7. Candidate Models

Eight models were evaluated as genuine candidates: Gemini 2.5 Flash, GPT-4.1 nano, GPT-5 mini, GPT-5 nano, GPT-5.4 mini, GPT-5.4 nano, GPT-OSS-120B, and Kimi-K2.5.

Three models appear in the pipeline without being part of the comparison: GPT-4o mini functions as a validation control, since Base 0 is made up of conversations generated by that very model (circular comparison); GPT-4o and GPT-4.1 act as judge models in the LLM-as-a-Judge panel.

8. Results

8.1 Final Score and Hallucination Rate

Three models score above 79% on Final Score with competitive Pass Rate, but are rejected for exceeding the overall Hallucination Rate limit. GPT-5 mini is the only one with a critical occurrence.

8.2 Model Ranking: By Score and By Verdict

Ranking the 8 models purely by composite score, rank position does not equal approval: GPT-4.1 nano appears in 3rd place by Final Score, ahead of Kimi-K2.5, but was rejected for exceeding the hallucination limit.

The same 8 models, grouped by final verdict, with score ordering within each group. The side-by-side comparison makes it explicit that it's the hallucination gate, not the raw ranking position, that decides approval.

8.3 Failure Patterns by Model

Four patterns stand out, not always for the most intuitive reason:

1. Wrong tool choice (most common, GPT families): a judgment error about which action to take, not an inability to follow the required format.
2. Text response instead of action (Gemini 2.5 Flash): different calibration between "when to act" and "when to just talk."
3. Provider rate limiting, not reasoning (Kimi-K2.5): 100% of recorded technical failures are HTTP 429s.
4. Decision not recognized by the pipeline (GPT-OSS-120B): in 87% of sessions, neither a tool call nor a text response was identified, due to the Harmony format.

8.4 Verdict Consistency Across Runs

Since LLMs are probabilistic systems, we recalculated the verdict run by run, not just on the volume-weighted aggregate. Gemini 2.5 Flash (47.2% approved / 44.4% rejected) and GPT-5 mini (33.3% in each category) hover near a tie, suggesting dependence on the specific session sample. GPT-OSS-120B never achieved approval in any run, consistent with its structural cause.

Among the approved models, Kimi-K2.5 (80.0%) and GPT-5.4 mini (84.6%) maintain consistent approval across runs. GPT-5.4 nano is the least consolidated (50.9%).

8.5 Real Cost vs. Token Proxy

The Cost score (C_norm) is normalized by token volume, not monetary value, since the pipeline has no per-call dollar-cost telemetry. Converting to USD using public list prices:

This estimate does not reproduce the C_norm ordering used in the Final Score. GPT-5.4 mini gets C_norm = 0.89, the best normalized cost score among the 8 candidates, but in practice has the highest estimated CPS — about 5.3× the cost of GPT-5.4 nano. The cause lies in the output-token list price (US$ 4.50/million vs. US$ 1.25/million), not in the volume of tokens processed.

9. Discussion

The hallucination gate as a methodological finding. If approval depended only on the Final Score, at least three models scoring above 79% would have been recommended even while exceeding the hallucination limit. Treating the Hallucination Rate as an eliminatory gate, applied afterward and independently of the weighted average, is what prevented this risk. In agents that make decisions on critical flows, a weighted quality average should not, on its own, decide approval.

Variance across runs. A single isolated rejected verdict should not be enough to discard a model. The GPT-OSS-120B case, rejected in every single run without exception, is a qualitatively different and stronger piece of statistical evidence than an isolated aggregate rejection.

LLM-swap is operationally simple, but not without monitoring. Unified routing turns swapping models, in most cases, into swapping a configuration name. But the four failure patterns identified show that each family tends to fail in its own specific way, even behind the same interface. Adopting LLM-swap does not eliminate the need for continuous monitoring per model family.

Limits of approved interactions as a quality criterion. Of the 106 candidate sessions, only 20 survived manual review. Automatic approval of a synthetic interaction is not, on its own, a guarantee that the agent correctly followed the expected routing.

11. Future Work

Phase 2 will investigate incremental capabilities, such as improved Function Calling and extended context windows. Other directions: expanding Base 0, resolving Kimi-K2.5's rate limiting, reincluding GPT-OSS-120B after completing extraction from the Harmony format, replacing the public list-price estimate with actually contracted cost, expanding provider coverage, and further investigating GPT-5 mini's critical hallucination.

12. Conclusion

Of the eight models evaluated, only GPT-5.4 mini, GPT-5.4 nano, and Kimi-K2.5 met the approval criteria. The most interesting result, however, is not found in the approved models, but in the pattern observed among the five rejected ones: in almost every case, the determining factor was the incidence of hallucinations, not the overall quality of the responses.

This behavior reinforces an important distinction between quality and safety. In agents that operate on critical flows, safety properties should be treated as eliminatory criteria, not as just another component of the final score. In a weighted average, a serious failure can be offset by good performance on other dimensions and still result in a satisfactory score. A hard filter eliminates that possibility: fundamental requirements are non-negotiable. If a model exhibits a critical hallucination, it must be rejected regardless of its performance on the other criteria.

---

References

1. Microsoft Azure, "Azure OpenAI Service: Pricing," 2026.
2. Microsoft Azure, "Foundry Models Pricing: Kimi," 2026.
3. M. Grace, J. Hadfield, R. Olivares, and J. De Jonghe, "Demystifying Evals for AI Agents," Anthropic Engineering Blog, Jan. 2026.
4. M. Finio and A. Downie, "What is AI Agent Orchestration?," IBM Think, 2025.
5. S. G. Patil et al., "The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models," Forty-second International Conference on Machine Learning, 2025.
6. L. Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," NeurIPS: Datasets and Benchmarks Track, 2023.
7. S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains," arXiv:2406.12045, 2024.
8. Google, "Gemini Developer API Pricing," 2026.

Download the full technical study (PDF)