AI Evaluation Framework: Catch Accuracy Drift Automatically

An AI feature ships with a strong demo. Leadership signs off because the answers in the sales meeting were sharp, the latency was acceptable, and the cost projection looked reasonable. Six weeks later the support team starts forwarding screenshots — answers that used to be right are now subtly wrong. A customer pastes a transcript where the model invented a policy that does not exist. The engineering team checks the prompt. It has not changed. The application code has not changed. Yet the system is behaving differently, and nobody can say when the change began or how much of the production traffic was affected.

The cause is almost always one of three things. The foundation model received a silent update from the provider and the new version interprets the prompt differently. The retrieval corpus drifted as documents were edited or added. Or the inputs themselves drifted because users started asking new types of questions the system was never tested against. What separates teams that catch this in a sprint from teams that learn about it through a churn email is whether an AI evaluation framework is sitting between the model and the user. This article walks through what the framework actually contains, how it gets wired into the delivery pipeline, and what it costs to build one alongside an AI engagement rather than after one.

Why Silent Drift Is The Most Expensive Class Of AI Failure

The failure mode an evaluation framework prevents is the one that is hardest to see and slowest to escalate. A web service returning 500 errors gets paged within minutes. An AI system that drifts from 92% to 84% accuracy on its core task produces no error signal — it just produces answers that are a little worse, sometimes wrong, and increasingly distrusted. By the time the trend is visible in customer behaviour, some users have stopped using the feature, a handful of decisions made on bad answers have already landed in production data, and the diagnostic surface has been buried under thousands of new log lines.

The cost compounds across three axes. Retention: users who lose confidence in an AI feature rarely come back to it, even after the underlying issue is fixed. Operations: investigating drift without an evaluation suite means re-running tests by hand against historic queries, which takes days and is rarely conclusive. And governance: under the EU AI Act, ISO 42001, and the NIST AI Risk Management Framework, the burden of evidence for higher-risk use cases sits with the operator, and "we noticed in the metrics" is not the standard those regimes describe. The Built to Last™ 2.0 framework treats this risk as a P05 Right Code concern because the discipline lives in the pipeline. Without it, every AI release is a release nobody is qualified to validate.

What An AI Evaluation Framework Actually Is

The AI evaluation framework sits in the Right Code pillar alongside Prompt Version Control and CI/CD Pipeline Implementation. It is a systematic, version-controlled test suite for AI behaviour — the equivalent of CI/CD for an AI system. It runs on every prompt change, every model update, every retrieval pipeline tweak, and on a continuous cadence in production. It produces a result engineers, product managers, and stakeholders can read the same way they read a green or red unit test. The framework breaks into five constituent parts.

The benchmark dataset

The benchmark dataset is the labelled, version-controlled set of inputs and expected behaviours the system is measured against. It is not a list of demos. It is a representative sample of the queries the system will receive in production, with the expected response — exact, partial, or behavioural — captured next to each. A workable starting size is 100 to 500 labelled examples spanning the query categories the use case must handle. The set lives in source control alongside the application code, has a changelog like the code, and is reviewed in pull requests when it changes. The first sign an evaluation effort is going wrong is a benchmark that lives in a spreadsheet nobody owns.

The dataset grows with the product. Every production failure that surfaces — a wrong answer flagged by a user, a hallucinated citation found by support, an edge case the team missed — becomes a new example. The discipline is to add the failing case to the dataset before fixing the underlying issue, so the fix is verified against the case that motivated it, and so the regression cannot recur silently.

Edge case and adversarial inputs

A benchmark of expected queries catches degradation on expected queries. The system also has to be tested against the queries that break it. Edge cases are inputs at the boundaries — extremely long, very short, in unsupported languages, deliberately ambiguous, or containing conflicting instructions. Adversarial inputs are crafted to provoke a known failure mode — prompt injection attempts, attempts to extract the system prompt, attempts to bypass safety constraints.

The edge case and adversarial suite is smaller than the benchmark set — typically 30 to 80 inputs — but it runs on every release. The point is not to prove the system is impregnable; the point is to detect when a model update or a prompt change has weakened a defence that was previously holding. OWASP's Top 10 for Large Language Model Applications is the right place to start when scoping this suite.

Hallucination tests

Hallucination is the failure mode that costs the most credibility per occurrence and is hardest to detect from the model's output alone. The hallucination subset is a slice of the benchmark where the expected behaviour is bounded — the answer must come from the retrieval context, must cite a real source, or the system must refuse when it lacks evidence. Each example carries a verification rule the harness applies automatically. The hallucination rate becomes a percentage on every release, not a vibe.

For systems that do not ground answers in a retrieval pipeline, the test is harder but still possible. The set captures factual claims about identifiable entities and verifies them against a source of truth. Where verification must be manual, the set is smaller and the cadence is weekly rather than per release. The discipline that matters is that hallucination is being measured as a number.

Response consistency tests

A model that gives a correct answer once and a different correct answer the second time is technically right and operationally broken — users lose confidence when the same question produces different responses on different days. Consistency tests run the same query multiple times, ideally across model versions, and measure whether the responses agree on substance. The test framework needs a similarity rule that allows surface variation while flagging meaningful divergence. Embedding-based similarity works for many cases; rule-based extraction works better where the response has structure the test can parse.

Consistency tests are load-bearing for systems where the provider's foundation model is on a versioned-but-rolling-update cadence. A model update that improves average performance can quietly increase variance, and the user-visible result is a system that feels less reliable even though average accuracy has not dropped.

Continuous evaluation and drift detection

The four suites above are necessary; continuous execution is what makes them effective. Evaluation runs on every prompt change in CI, on every model update, on every retrieval-corpus update, and on a scheduled cadence — typically daily — against the live production endpoint. Results are versioned, charted, and alerted on. An accuracy drop below threshold pages the on-call engineer. A statistically significant change in hallucination rate raises a ticket. A consistency divergence triggers a model-version comparison. The decision the framework produces is binary on every release and continuous in production, where the trend chart shows whether the system is holding, drifting, or improving.

A team can build all five suites and still ship a system that drifts unnoticed. Three failure modes recur. First, the benchmark is too narrow — it covers what the team thought to test, not what users actually ask, and the eval keeps passing while production quality degrades. Second, the suite runs on prompt changes but not on model updates, because the provider's model is implicitly trusted; this is the single most common cause of the six-week silent regression. Third, the alerts fire but nobody acts — the channel is noisy, the threshold was set too tight, or the on-call rotation does not include an engineer who can interpret the result. Each failure mode is structural and each has a fix; none is solved by having a framework on paper.

How To Implement An AI Evaluation Framework Without Slowing Delivery

The discipline does not require pausing the build. It requires sequencing the right work alongside the early sprints of an AI engagement. A realistic four-to-six-week implementation runs in parallel with feature development and reaches steady-state operation before the system enters the Production Readiness Review™.

Week one is benchmark scoping. The team works through the use case with the product owner and writes down the query categories the system must handle: lookup, comparative, generative, summarisation, refusal, whatever applies. Each category gets an initial set of 10 to 30 labelled examples drawn from real-world queries — historical support logs, sales-call transcripts, or validation work from the Discovery Workshop™ if one was run. The output is a versioned bench.jsonl checked into the repository with a README explaining the labelling rules.

Week two is metric definition and thresholds. Each suite needs a clear pass criterion the team commits to. The criteria are not aspirational; they are the floor below which the system does not ship. Common forms: exact match for structured outputs, embedding similarity above a threshold for free-form outputs, refusal rate within a band for guardrail tests, hallucination rate below a percentage on the bounded subset. The thresholds are reviewed monthly against production data; tight thresholds catch drift faster but produce more false alerts, and the team has to live with that trade-off honestly.

Week three is harness construction. The eval harness is a thin piece of code that, given a system version and a benchmark file, runs every example through the system, scores the output against the expected behaviour, and produces a structured result. It needs four properties: it has to run locally in seconds for a developer iterating on a prompt, it has to run in CI for every pull request, it has to run against the production endpoint on a schedule, and it has to write results to a store that supports trend analysis. The store can start as a flat file in object storage and graduate to a dedicated eval platform when the team outgrows it.

Week four is CI/CD integration. The harness runs on every pull request that touches prompts, retrieval configuration, or model selection. A failing run blocks merge by default; a sign-off process exists for the cases where a known regression is being shipped deliberately, with the new failing example added to the benchmark first. The integration uses the same CI tooling the application uses, so the engineers who maintain the deployment pipeline also maintain the evaluation gate.

Weeks five and six move evaluation into production. The harness runs daily against the live system using a frozen subset of the benchmark. Results are charted, thresholds are tightened or loosened based on the first week of real data, and alerts are wired to the on-call rotation. The benchmark itself moves to a write path: production failures flagged by users, support, or sampling get triaged into the benchmark before the underlying fix ships. The framework now operates as infrastructure, not as a project.

Three obstacles recur. The team that defers benchmark construction because building examples is slower than building features ships a framework that runs on examples nobody trusts. The team that tests only at release time and not continuously in production misses the foundation-model updates that arrive between releases. The team that runs evaluation against the model directly rather than the full pipeline — prompt, retrieval, post-processing, guardrails — catches model drift but misses the common case where a retrieval-corpus update changes behaviour. Implementation depends on Prompt Version Control, the CI/CD Pipeline Implementation, and the Production Readiness Review.

A Foundation Model Update That Cost Six Weeks Of Silent Drift

A mid-sized SaaS client (engineering team of around fifteen, an AI-powered classification feature in a customer-facing workflow) we worked with launched with strong demo accuracy. Internal sampling at launch put correct classification at the upper end of the team's target band. The team shipped, moved to the next feature, and treated the AI component as done.

Six weeks later a support pattern emerged — customers were correcting outputs at roughly twice the rate seen in week one. Root cause investigation took most of a sprint. The application code had not changed. The prompts had not changed. The retrieval corpus had not drifted materially. The foundation-model provider had pushed a minor version update on a date the team had not tracked, and the new version's interpretation of a key instruction in the system prompt had shifted by a few percentage points on the team's specific query distribution. The drop was small per query and compounded across millions of monthly classifications.

The remediation was an evaluation framework retrofitted in four weeks. A benchmark was built from production logs since launch, including support-logged corrections. Hallucination and consistency suites were added. The harness ran in CI from week three and daily against production from week four. The first piece of value the framework delivered was retrospective: it confirmed the regression point, quantified the drop, and supported the rollback to the previous model version while prompts were re-tuned. Three months on, the same update would have been caught the day it landed.

When This Component Is Critical, And When You Can Defer It

The framework is critical the moment an AI feature sits in a customer-facing path, in a regulated workflow, or in a decision loop whose outputs affect business records. It is critical for systems that depend on a foundation model the team does not own — which is most production systems — because silent provider-side updates are the dominant cause of unexplained drift. It is critical under the EU AI Act, ISO 42001, and NIST AI Risk Management Framework for higher-risk use cases, where measurable post-deployment monitoring is part of the compliance expectation.

It can be deferred — though not skipped — for narrow internal proofs-of-concept where the user base is small, outputs are advisory, and the use case will be retired before production scale. Even there, the benchmark dataset is worth building from week one; a proof-of-concept that succeeds becomes a system that needs the framework retrospectively, and retrospective benchmarks built from imperfect logs are always weaker than benchmarks built from labelled examples at design time.

What To Do Next

If you have an AI system in production without a versioned benchmark, an edge-case suite, a hallucination metric, or a continuous evaluation cadence, the sequence above is the structured way to close the gap before the next provider-side model update finds you. For the broader view of how AI evaluation sits inside the delivery flow, see how we deliver agentic AI. The next BTL component most AI teams need alongside this one is Prompt Version Control — the source-of-truth discipline that lets the evaluation framework compare apples to apples across releases.

Frequently Asked Questions