Executive Takeaway: One of the disciplines I valued most in traditional machine learning was rigorous testing—understanding where a model performed well, where it failed, how design choices affected results, and whether improvements held up on unseen data. AI-assisted workflows raise those evaluation stakes even higher. A production system may combine RAG, prompts, GenAI models, predictive signals, deterministic rules, services, and human review, with each layer introducing its own failure modes. The right mental model is therefore layered: Did RAG retrieve the right evidence? Given the right evidence, did the prompt produce the right behavior? Given the same evidence and prompt, which model performs best? Did deterministic rules execute correctly? And did the complete workflow improve the decision? A Decision Contract provides the specification tying these tests together by defining the evidence, signals, constraints, AI role, human judgment, and success measures required for the decision. The objective is not simply better model performance, but a system that is transparent, testable, governable, and demonstrably improves decision quality, timing, and accountability.
Introduction
One of my favourite aspects of traditional machine learning model development was always the rigor around testing and evaluation.
A good evaluation framework helped you understand where a model was strong, where it struggled, which components and design choices were actually contributing to performance, and whether an improvement was real or simply looked promising on a handful of examples. It gave you a disciplined way to optimize the model while also building confidence that the final solution would perform when exposed to new data.
As I have moved further into AI Engineering, I have found myself returning to that same discipline.
But in many ways, AI-assisted workflows built with retrieval-augmented generation (RAG), prompting, generative AI models, business rules, and human review raise the performance-measurement stakes even higher.
There is no longer just one model to evaluate.
There is a chain of interdependent components, each of which can introduce error, uncertainty, or variability into the final decision. The process for measuring each component individually—and then measuring the performance of the complete system—is every bit as important as it was in traditional machine learning, but often more nuanced and more specific to the workflow being designed.
That leads to a deceptively simple question:
How do you actually test an enterprise AI system?
Traditional machine learning gives us a familiar framework: training data, validation data, holdout data, accuracy, precision, recall, and other well-defined evaluation metrics.
Generative AI systems require something broader.
A governed AI-assisted workflow should not be evaluated as one opaque system with a single “AI accuracy” score. Retrieval can fail while the model performs perfectly. A prompt can fail even when the correct evidence is supplied. A model can extract the right information but behave inconsistently. A recommendation can sound reasonable while violating a deterministic business rule. And an entire workflow can be technically accurate while doing little to improve the business decision it was designed to support.
The approach I have increasingly adopted is therefore to evaluate the system in layers:
Retrieval → Prompt → Model → Rules and Services → Decision Workflow → Business Outcome
The general mental model is:
- RAG test: Did we retrieve the right evidence?
- Prompt test: Given the right evidence, did the instructions produce the right behavior?
- Model test: Given the same evidence and prompt, which model performs best?
- Rules and services test: Did the deterministic logic execute correctly?
- Workflow test: Did the complete system improve the business decision?
And the unit around which those tests are designed is not the model.
It is the decision.
That is where a Decision Contract becomes useful. It establishes what decision is being supported, which evidence and signals are required, which rules and constraints apply, what the AI is allowed to recommend or explain, when it must escalate, where human judgment remains necessary, and what action should follow.
The Decision Contract therefore does more than govern the workflow.
It provides the specification against which the AI system can be tested.
“AI accuracy” is the wrong abstraction
A common approach to evaluating generative AI is to assemble several examples, run them through the system, inspect the answers, and ask whether the results “look good.”
That may be sufficient for a demonstration.
It is not sufficient for an enterprise system.
Consider a workflow in which AI must identify a business situation, retrieve supporting documentation, interpret that evidence, apply business rules, generate a recommendation, and route it to a human decision-maker.
If the final recommendation is wrong, what failed?
The retrieval layer may have returned the wrong document.
The correct document may have been retrieved but ranked too low.
The correct evidence may have been supplied to the LLM, but the prompt failed to extract an important fact.
The prompt may have worked, but the selected model produced an unsupported inference.
The model may have correctly interpreted the evidence, but a calculation or business rule was implemented incorrectly.
Or every technical component may have worked while the recommendation arrived too late, created too much review effort, or failed to give the decision-maker enough confidence to act.
These are fundamentally different failure modes.
Treating them all as “AI accuracy” makes the system almost impossible to diagnose.
The better approach is to isolate each layer, establish measurable expectations for it, and only then test the complete workflow.
Start with a Decision Contract
This is one reason I continue to believe enterprise AI should be designed backward from the decision.
A Decision Contract defines the operating conditions for a specific AI-assisted decision.
It identifies the decision owner and decision point; the evidence and signals required; the business rules and constraints that govern the choice; what the AI should retrieve, interpret, recommend, explain, or escalate; where human judgment is required; what happens after the decision; and how success will be measured.
Those same requirements can become evaluation criteria.
If the Decision Contract requires evidence from three authoritative source categories, for example, the evaluation should measure whether the retrieval system consistently finds evidence across all three.
If a recommendation may only proceed when specified inputs are available, the evaluation should test whether the system abstains when one is missing.
If a business rule determines whether a recommendation is permissible, that rule should be tested deterministically.
If the AI must explain why it reached a recommendation, the explanation should be evaluated against the evidence and rules actually used.
This changes the evaluation question from:
“Did the AI give a good answer?”
to:
“Did the system satisfy the requirements of the decision?”
That is a much more useful question.
Hold out decisions, not arbitrary document chunks
There is another lesson I have carried over from traditional machine learning.
We still need development, validation, and holdout testing.
But RAG changes what should be held out.
A traditional supervised model may split individual labeled rows into training, validation, and test datasets. A RAG system has an external knowledge environment available at inference time. Randomly splitting document chunks therefore does not necessarily replicate the business problem we are trying to test.
The more meaningful protected unit is often the decision case. A decision case is a complete, representative instance of the business decision being evaluated: the situation or entity under review, the evidence and signals legitimately available at that point in time, the applicable rules and constraints, and the expected disposition or action. In other words, it is the unit of work the AI-assisted workflow is ultimately trying to support—not an individual document, chunk, or model response.
Historical cases can be divided into development cases used while configuring the system, validation cases used to choose retrieval strategies, prompts, models, and thresholds, and a locked set of unseen cases that remain untouched until the system configuration is frozen.
The knowledge corpus remains the environment the AI is allowed to search.
But it needs leakage controls.
If an unseen historical case can retrieve the completed recommendation, approved decision, or another artifact that directly reveals the answer being evaluated, the benchmark is no longer measuring the system’s ability to reconstruct the decision from legitimate evidence.
It is measuring its ability to retrieve the answer.
That distinction matters.
For historical evaluation, I increasingly think in terms of an “as-of” evidence environment: what information would legitimately have been available to the decision-maker when the original decision was made?
This gives generative AI testing something much closer to the discipline of a traditional holdout set.
Test RAG before testing the answer
For document-intensive enterprise AI, retrieval should be evaluated as its own system.
If the correct evidence never reaches the model, prompt engineering cannot recover it.
Classic information-retrieval metrics remain useful. Precision@k tells us how much of the retrieved material is relevant. Recall@k tells us how much of the relevant evidence was found. Metrics such as MRR and nDCG help evaluate ranking quality.
But enterprise RAG needs more than semantic relevance.
Was the authoritative source retrieved?
Was the correct revision used?
Was an obsolete document incorrectly promoted?
Did retrieval cover every evidence category required by the Decision Contract?
Did the citation actually point to the passage supporting the claim?
Was information retrieved from within the user’s permitted access scope?
A search result can be semantically relevant and still be operationally wrong.
This is why I increasingly think of trustworthy RAG as an evidence system, not simply a vector-search problem.
Chunking strategies, metadata filters, hybrid search, reranking, source authority, document revision, top-k settings, and citation construction should all be benchmarked rather than selected because a handful of queries happened to work well.
Ablation testing is particularly useful here. The idea is to deliberately remove, disable, or change one component at a time while holding the rest of the system constant. We might compare retrieval with and without metadata filters, reranking, source-authority weighting, or different chunking strategies and top-k settings. The objective is not simply to determine which configuration performs best, but to understand which components are actually contributing to the improvement—and what trade-offs they introduce in relevance, coverage, latency, or cost. Paired bootstrap resampling can then help quantify how confidently those observed improvements generalize beyond the particular benchmark cases tested.
Only once retrieval reliably supplies the required evidence should downstream reasoning become the primary evaluation target.
Test prompts with the evidence held constant
Prompt evaluation becomes much cleaner when retrieval is removed as a variable.
Give competing prompt versions the same benchmark cases, the same validated evidence, the same model, the same tools, and the same output schema.
Then change only the prompt.
Now it becomes possible to evaluate whether the instructions themselves are improving the system.
For structured extraction, I might measure precision and recall of the extracted fields. For workflow integration, I would look at schema validity and instruction compliance. For evidence-grounded reasoning, I would examine grounded claims, unsupported claims, citation fidelity, conflict handling, and appropriate abstention.
Repeated runs are useful as well.
If the same evidence and instructions repeatedly produce materially different recommendations, that variability itself is an evaluation result.
This leads to a principle that I think enterprise AI teams need to adopt more broadly:
A prompt change is a software change.
A production prompt is part of the system’s behavioral logic. Changing it should be governed, tested, versioned, and released with the same discipline as a software change.
It should not be promoted because five examples suddenly “look better.”
It should outperform—or at least remain non-inferior to—the existing version on a governed benchmark without creating regressions somewhere else.
Model selection should be task-specific
The same principle applies to choosing a model.
When comparing models, hold the benchmark case, evidence, prompt, rules, output schema, and evaluation rubric constant.
Change the model.
Now the comparison becomes meaningful.
Task accuracy obviously matters. But so do groundedness, structured-output reliability, tool-use correctness, run-to-run stability, latency, context handling, deployment constraints, and cost.
The largest or newest model does not automatically win.
In fact, I increasingly think the right objective is:
Use the lowest-complexity, lowest-cost model that reliably satisfies the requirements of the task.
A structured extraction task may not require the same model as a difficult evidence-synthesis task.
And neither should require an LLM to perform logic that belongs in deterministic code.
That separation is important because a good enterprise AI architecture is rarely purely probabilistic.
Probabilistic and deterministic components require different tests
LLMs are useful precisely because they can interpret ambiguity.
They can read documents, extract information, compare passages, summarize evidence, explain a recommendation, and work with natural language in ways traditional software cannot.
Predictive models can add another probabilistic input by estimating risk, propensity, probability, demand, failure likelihood, or another business signal.
But not every part of the workflow should behave probabilistically.
Business rules, required fields, calculations, schema validation, revision checks, permission checks, thresholds, and other deterministic controls should generally remain in rules and code.
And they should be tested accordingly.
A quantity formula needs unit tests and boundary tests.
A required-field rule should fail every time the required input is absent.
A source-authority rule should deterministically prevent an invalid source from becoming the governing evidence.
A permission rule should never depend on whether an LLM happens to interpret the policy correctly on that run.
The model interprets.
The governed workflow decides what that interpretation is allowed to influence.
Keeping those responsibilities separate not only improves reliability. It makes failures far easier to test and explain.
The most important tests may be the cases where the AI should not answer
Happy-path examples make good demos.
Failure cases make good systems.
A mature evaluation suite should deliberately create situations in which required evidence is missing, authoritative sources conflict, obsolete documents rank highly, two entities have nearly identical descriptions, a source contains prompt-injection text, a calculation lacks required inputs, a tool call fails, a user lacks permission to access a source, or a citation points to a passage that does not actually support the claim.
The desired behavior is often not to produce a better answer.
It is to stop, flag, or escalate.
This is one of the more important conceptual shifts in governed AI.
An unresolved result is not necessarily a failure.
If the Decision Contract says that a recommendation requires specified evidence and that evidence does not exist, “insufficient evidence” may be exactly the right answer.
A system that confidently completes every workflow can be considerably more dangerous than one that knows when the decision boundary has been reached.
End-to-end UAT should test the decision workflow
Once individual components are working, the complete workflow still needs to be tested.
This is where technical evaluation begins to meet business evaluation.
The release candidate should be frozen: corpus and index version, retrieval configuration, prompt version, model configuration, rule set, schemas, and application services.
Then the locked holdout cases can be executed.
Representative users should evaluate whether the system found the right evidence, whether that evidence was sufficient, whether the recommendation was understandable, whether they accepted or overrode it, why they overrode it, and whether the workflow actually made their work easier.
And this is where the metrics should begin shifting beyond model performance.
How much research effort was reduced?
Did cycle time fall?
Was rework reduced?
How often were recommendations accepted, changed, or rejected?
Were overrides explainable?
Could the decision be reconstructed later?
Were users better able to identify uncertainty?
Did fewer material defects escape the workflow?
Ultimately, technical quality matters because of what it enables at the decision point.
Version everything
There is one additional implication that becomes obvious once AI evaluation is treated this way.
Almost every material component of the system needs a version.
The corpus.
The index.
The retrieval configuration.
The prompt.
The model.
The model parameters.
The business-rule package.
The schema.
The benchmark dataset.
The application release.
A change to any one of these can create an AI regression even when the conventional application code has not changed.
Changing a chunking strategy can affect retrieval.
Changing a prompt can alter abstention behavior.
Changing the model can improve extraction while reducing consistency.
Updating a rule can change downstream recommendations.
Adding documents to the corpus can change which evidence gets ranked.
Enterprise AI therefore needs regression testing in much the same way conventional software does—but across a broader system boundary.
And this is where the benchmark itself becomes an important enterprise asset.
A small set of carefully adjudicated cases with known evidence, known constraints, expected outcomes, and protected holdout governance can be more valuable than thousands of synthetic examples.
Over time, that benchmark becomes a form of governed organizational memory.
It captures not only what the organization decided, but what evidence, rules, exceptions, and judgment define an acceptable decision.
The real evaluation target is decision quality
The more enterprise AI systems I work with, the less interested I become in whether a model can produce an impressive answer in isolation.
That is not the problem most organizations need to solve.
They need AI systems that can reliably find the right evidence, interpret it appropriately, combine it with signals, respect deterministic rules, expose uncertainty, show their work, involve humans at the right point, and improve a consequential business decision.
That requires more than model evaluation.
It requires decision-system evaluation.
The Decision Contract provides the specification.
RAG provides governed evidence.
Predictive models provide signals.
LLMs provide interpretation and synthesis.
Rules and services provide deterministic control.
Human judgment handles the ambiguity and accountability that should remain human.
Together, those components create a governed AI-Assisted Decision Intelligence Workflow designed around a specific business decision.
This is also how I increasingly think about the value AI should create inside an organization.
The goal is not simply to generate more AI output.
It is to create better decision currency: evidence-grounded, governed, explainable information that the organization can confidently use to make and manage better decisions.
If we cannot identify the decision being improved, define what a good decision requires, and demonstrate that the AI-assisted workflow improves the quality, timing, or accountability of that decision, then we may still have an interesting AI system.
But we are probably still experimenting.
AI only creates value when it improves the quality, timing, and accountability of business decisions.
About the Author
Robb is the President and Principal Decision Intelligence Architect at Scope Analytics, helping organizations combine predictive signals, enterprise knowledge, AI reasoning, and human judgment into better decision systems.
Learn more: https://www.scopeanalytics.com
Leave a Reply