Building AI Around the Decision: What I’ve Learned Moving from Decision Science into AI Engineering

A retrospective on RAG, Agents, Signal Engineering, and the evolution from Predictive Models to AI-Assisted Decision Workflows

Executive Takeaway: Over the past year, my work has increasingly moved from traditional Decision Science into AI Engineering—using RAG, LLMs, agents, enterprise knowledge, and governed AI-assisted workflows. Yet the transition has reinforced rather than replaced the core principles of Decision Science. The objective is still not the model, prediction, dashboard, or LLM; it is the decision. What has changed is the machinery available to support it. Analytical models generate predictive and optimization signals, RAG makes enterprise knowledge accessible as evidence, LLMs provide reasoning and synthesis, and agentic orchestration brings these capabilities together within operational workflows. This creates an important role for Signal Engineering: integrating models, evidence, business rules, and human context into governed inputs for AI-assisted recommendations. For Decision Scientists, the opportunity is to bring strengths in problem framing, uncertainty, evidence, and decision quality into the architecture of AI systems. The future is not LLMs replacing Analytics, but Analytics, enterprise knowledge, AI reasoning, and human judgment working together around better decisions.

Introduction

Over most of my career, I have thought of myself as a Decision Scientist (as opposed to Data Scientist). That distinction has always mattered to me because the objective was never simply to build a better model, produce a more sophisticated analysis, or create another dashboard. The objective was to use data, analytics, modelling, and quantitative reasoning to help someone make a better decision.

Over the past two years, however, the nature of the systems I have been building has changed considerably. I have found myself working much closer to what we now call AI Engineering: building Retrieval-Augmented Generation (RAG) systems, designing prompts and structured outputs, working with enterprise knowledge, orchestrating agents, defining APIs and schemas, thinking about traceability and evaluation, and integrating probabilistic AI into governed operational workflows.

At first, this felt like a significant career transition. In retrospect, I see it somewhat differently. I did not stop being a Decision Scientist and become a Software Engineer. Instead, the technical surface area available to Decision Science expanded. The tools changed, the architecture changed, and the kinds of information we can use changed, but the underlying question remained remarkably consistent:

What decision are we trying to improve, and what evidence should support it?

That question has become even more important in the age of Generative AI.

From Models to Decision Systems

Traditional Data Science has generally followed a familiar pattern:

Data → Features → Model → Prediction → Human Interpretation → Decision

We collect structured data, engineer features, train models, and generate predictions, scores, forecasts, classifications, or recommendations. Then someone has to interpret those outputs and decide what to do. For years, much of my work lived inside this paradigm.

The models could be extremely valuable. A propensity score might identify the customers most likely to buy. A forecast might identify future revenue risk. An optimization model might recommend how to allocate resources. A churn model might identify accounts most likely to leave. But there was usually a gap between the analytical output and the business decision.

A model might tell you that the probability of conversion is 72%, but the decision-maker still wants to know why. What changed?

A model might tell you that the probability of conversion is 72%, but the decision-maker still wants to know why. What changed? What does the customer actually care about? What happened in the last meeting? What contractual constraints exist? What have we done in similar situations? Is there evidence supporting this recommendation? What should I do next?

Most predictive models were never designed to answer those questions, and much of the information required to answer them does not live neatly inside a relational database. It lives in language: emails, contracts, presentations, meeting notes, technical manuals, policies, research reports, support tickets, PDFs, and institutional knowledge.

Generative AI—particularly when combined with retrieval—allows us to begin connecting those worlds. That changes the architecture of decision support.

The New AI-Assisted Decision Workflow

The systems I increasingly find myself thinking about look less like isolated models and more like integrated decision workflows. Structured data produces analytical signals. Documents become retrievable evidence. Business rules define constraints and governance. Human context contributes judgment and intent.

Those inputs can then feed an AI-assisted workflow that produces something closer to:

Analytical Signals + Retrieval Evidence + Business Rules + Human Context → Agentic Orchestration → Grounded Recommendation + Rationale → Human Decision / Action → Outcome / Feedback

The LLM is part of that system, but it is not the whole system. That distinction has become one of the most important lessons of my transition into AI Engineering.

A useful enterprise AI system is rarely just:

Prompt → LLM → Answer

The real work is designing everything around the model.

Traditional modeling has not disappeared inside the AI-assisted workflow. Its role has changed. Predictions, forecasts, probabilities, optimization outputs, and other model results now enter the workflow as Analytical Signals. Signal Engineering prepares those signals for use alongside a second, distinct source of information: Evidence retrieved through RAG. Business Rules and Human Context provide additional constraints and judgment. Agentic orchestration can then bring these inputs together to generate a recommendation and rationale for human decision-making.

What Transferred from Decision Science?

More than I expected.

When generative AI first exploded into mainstream business conversation, it was easy to assume that a completely new discipline had arrived. In some respects, it had. But many of the skills that made someone effective in Data Science or Decision Science remain highly transferable.

Problem framing

One of the most important skills in Decision Science has always been translating an ambiguous business problem into something analytically tractable. That becomes even more important with AI. Before asking which model to use, which vector database to implement, or which agent framework to adopt, someone still needs to define what decision the system is supposed to improve.

Many AI projects begin with questions such as “Where can we use AI?”, “Can we build a copilot?” or “Can we put an LLM over our documents?” Those are technology questions. Decision Science teaches us to work backward from the decision itself:

Who owns the decision?
What information matters?
What constitutes sufficient evidence?
What does a good outcome look like?
What are the costs of being wrong?
When should the system abstain?
When should a human review the recommendation?

Those questions are foundational to good AI architecture.

Thinking in signals

Data Scientists are trained to search for signal inside noise. We ask which variables contain predictive information, how reliable they are, how they interact, whether they generalize, and whether they actually contribute to the outcome we care about.

That mindset transfers directly into AI systems, except now the signals may not all be numerical. A useful signal could be:

a probability score;
an anomaly;
a forecast;
a customer statement;
a clause inside a contract;
a maintenance interval inside a technical manual;
a business rule;
a historical precedent;
an expert judgment; or
the absence of required evidence.

The analytical mindset still matters. The evidence surface simply became much larger.

The analytical mindset still matters. The evidence surface simply became much larger.

Understanding uncertainty

Traditional Data Science forces us to think about uncertainty. Predictions are not facts. Models make errors. Thresholds matter. False positives and false negatives carry different costs. That same discipline is essential when working with LLMs.

An AI-generated recommendation should not automatically be interpreted as truth simply because it is expressed fluently. We still need to ask how confident we are, what evidence supports the recommendation, what evidence contradicts it, what information is missing, and whether the system should proceed, ask a question, or escalate to a human.

Generative AI makes uncertainty easier to hide behind confident language. Decision Science makes us suspicious of that confidence. That is a useful instinct.

What I Had to Learn

The transferable skills were substantial, but so was the new learning curve. AI Engineering introduced an entirely new technical vocabulary and a new set of architectural considerations.

Retrieval-Augmented Generation

RAG was one of the first major shifts. A general-purpose LLM knows a great deal, but enterprise decisions often depend on information the model could not possibly know reliably: internal documents, customer-specific history, technical standards, organizational policies, current records, and proprietary knowledge.

RAG changes the problem. Instead of asking the model to rely entirely on what it learned during training, we retrieve relevant information at runtime and provide that evidence as context.

Conceptually:

Question → Retrieve Evidence → Provide Context → Generate Grounded Response

The concept is straightforward. In practice, almost every arrow in that sequence becomes an engineering problem:

How do we parse the documents?
How should they be chunked?
Which metadata should be extracted?
Should retrieval use keyword search, semantic search, or both?
How do we filter by customer, asset, document type, revision, or authority?
How many results should we retrieve?
How do we rerank them?
How do we distinguish authoritative evidence from merely similar text?
How do we preserve citations and provenance?
How do we evaluate whether the right evidence was retrieved?

I quickly learned that the quality of a RAG system often has less to do with writing a clever prompt than with engineering the information environment surrounding the model.

Prompting Was Not the Hard Part

Prompt engineering matters. Clear instructions matter. Structured output matters. System prompts matter. Examples matter. But prompting is only one layer of the system.

The more difficult problems tend to involve deciding what context the model should receive, ensuring that context comes from trusted sources, resolving conflicting evidence, defining what the AI is allowed to infer, and determining when deterministic rules should override probabilistic reasoning. The engineering problem continues downstream as well: outputs need to conform to schemas, human-review rules need to be explicit, system performance needs to be evaluated, and the rationale behind recommendations needs to remain auditable.

In practice, the hard part is less about finding a magical sequence of words and more about engineering a governed reasoning environment. That distinction matters.

From Feature Engineering to Signal Engineering

This evolution eventually led me to a concept I increasingly think of as Signal Engineering.

Feature engineering has always been central to traditional Data Science. We take raw data and transform it into representations that allow models to detect useful patterns. Signal Engineering extends that idea beyond the predictive model.

In an AI-assisted decision system, a useful signal might come from statistical modeling, machine learning, optimization, simulation, retrieval, business rules, or enterprise knowledge.

Consider a revenue decision. Suppose the question is:

Should we invest additional resources in this account?

A modern AI-assisted workflow might incorporate traditional analytical signals such as:

probability to buy;
expected revenue;
pipeline velocity;
churn or renewal risk;
historical response rates;
customer lifetime value;
product adoption;
account growth; and
forecast uncertainty.

Those are signals we already know how to build. But we might now combine them with retrieved evidence:

recent account notes;
executive meeting summaries;
product feedback;
contractual obligations;
support issues;
competitor mentions; and
historical proposals.

Then we can layer in business rules, such as territory ownership, investment thresholds, margin requirements, product eligibility, and approval requirements. We can also incorporate human context—things like strategic importance, executive relationships, unusual circumstances, and judgment that may never be captured cleanly in a database field.

The role of the AI system becomes the orchestration and interpretation of these different forms of evidence. The LLM does not replace the predictive model; it helps place the prediction in context. It does not replace business rules; it reasons within them. It does not replace human judgment; it helps prepare the evidence on which that judgment can operate.

The LLM does not replace the predictive model; it helps place the prediction in context. It does not replace business rules; it reasons within them. It does not replace human judgment; it helps prepare the evidence on which that judgment can operate.

That is what I mean by Signal Engineering, and I believe it is an important bridge between traditional Analytics and modern AI systems.

What Does the Decision Scientist Bring That a Software Engineer May Not?

This comparison needs to be made carefully.

AI Engineering unquestionably requires strong Software Engineering. Reliable systems need good architecture, APIs, authentication, deployment pipelines, observability, security, testing, versioning, scalability, and maintainable code. Those are disciplines in which experienced Software Engineers have substantial advantages.

But AI-assisted decision systems also require another kind of reasoning. Decision Scientists are accustomed to asking:

What exactly are we optimizing?
What outcome are we trying to change?
Which signals actually matter?
How reliable is this evidence?
What does the model miss?
What happens when two sources disagree?
What does a false positive cost?
What does a false negative cost?
Is the system calibrated?
Is the intervention causal or merely correlated?
What threshold should trigger action?
What is the expected value of acting?
When does uncertainty become too high?
How will we measure whether the recommendation improved the decision?

Those are different instincts.

A Software Engineer is often trained to ask:

How do we make this system work reliably?

A Decision Scientist is often trained to ask:

How do we know this is the right thing for the system to recommend?

Enterprise AI increasingly needs both.

The strongest AI teams will not be composed of one discipline replacing another. They will combine Software Engineering, Data Engineering, Machine Learning, Decision Science, domain expertise, governance, and product design.

My own advantage is not that I suddenly became a better Software Engineer than people who have spent their careers building production systems. It is that I can bring Decision Science much closer to the architecture. And that turns out to matter.

My own advantage is not that I suddenly became a better Software Engineer than people who have spent their careers building production systems. It is that I can bring Decision Science much closer to the architecture. And that turns out to matter.

Why LLMs Matter for Decision-Making

The biggest conceptual change for me is not conversational interfaces, chatbots, or even agents. It is the ability to make language part of the analytical system.

Historically, Analytics has been strongest where information could be structured. A customer record might tell us:

Revenue: $3.2 million
Growth: 14%
Probability to Buy: 72%
Support Cases: 7
Renewal Date: 94 days

We can model those variables. We can predict outcomes. We can prioritize accounts.

But real decisions rarely depend exclusively on structured fields. A sales leader also wants to know what the customer said in the last executive meeting, why they postponed the last purchase, whether a competitor is involved, what business problem they are trying to solve, what objections they have raised, and what happened with similar customers.

Before LLMs and modern retrieval systems, that information was difficult to incorporate systematically into decision support. It existed, but it was often trapped inside documents and prose.

RAG gives us a mechanism for retrieving that information dynamically. LLMs give us a mechanism for interpreting it. That means a decision-support system can begin to combine two fundamentally different types of intelligence.

A predictive model can tell us:

What is likely to happen?

Retrieved enterprise evidence can help explain:

What do we know about this specific situation?

Business rules can determine:

What are we allowed to do?

Optimization can help answer:

Where should we allocate scarce resources?

And an LLM can help synthesize:

Given the signals, evidence, constraints, and uncertainty, what should the decision-maker consider doing—and why?

That is a much richer form of decision support.

From Analytics Products to AI-Assisted Workflows

This also changes where Analytics lives.

For much of the last two decades, the dominant analytical interface has been the dashboard. The dashboard represented an enormous improvement over static reporting, but it still placed a substantial burden on the user: here are the metrics, here are the charts, here is the model score—now interpret everything and determine what to do.

AI-assisted workflows create the possibility of moving closer to the decision itself. Instead of simply surfacing information, the system can help assemble:

the relevant signals;
the supporting evidence;
the applicable rules;
the important uncertainties;
the recommendation;
the rationale; and
the actions available to the decision-maker.

The user still owns the decision where human judgment is required, but the system does more of the analytical preparation.

This is where agents become interesting. Not because autonomous agents should suddenly make every business decision, but because agentic orchestration can coordinate specialized analytical tasks that historically existed in separate systems. One component retrieves evidence. Another interprets analytical signals. Another applies business rules. Another checks completeness. Another generates the recommendation. Another records the Decision Trace.

The value is not autonomy for its own sake. The value is coordinating the work required to make a better decision.

**The Important Architecture Is Around the LLM**

This has changed the way I think about AI strategy.

It is easy to focus on the model: which LLM, which vendor, which agent framework, which vector database. Those decisions matter, but they are not usually the most strategically important ones.

The more enduring architecture sits around the model:

What decision is being supported?
What signals does that decision require?
What evidence is authoritative?
What rules constrain the recommendation?
What requires human judgment?
What should trigger escalation?
How do we measure the quality of the decision?
How do we learn from the outcome?

That is the decision layer. Increasingly, that is where I believe enterprise AI creates—or fails to create—business value.

Traditional Analytics Is Not Going Away

One of the least convincing narratives around generative AI is that it somehow makes traditional Data Science obsolete. I see almost the opposite.

A language model is extremely powerful, but if I need an accurate probability estimate, I would still prefer a well-designed predictive model. If I need to determine an optimal allocation under constraints, I want optimization. If I need to understand causal impact, I want causal methods. If I need a forecast with known statistical characteristics, I want forecasting methods designed for that problem.

LLMs add an extraordinary new capability. They do not repeal statistics.

The opportunity is integration.

The next generation of enterprise AI systems can combine:

Statistics;
Machine Learning;
Optimization;
Simulation;
Business Rules;
Retrieval;
Enterprise Knowledge;
LLM reasoning; and
Human judgment

within the same operational workflow.

That is far more interesting than asking which of those technologies will replace the others.

I Am Still a Decision Scientist

So, have I become an AI Engineer?

In practical terms, yes. I now spend considerably more time thinking about retrieval pipelines, prompts, agents, schemas, orchestration, APIs, evaluation, cloud architecture, and AI governance than I did a few years ago. Those are real new skills, and there is still plenty more to learn.

But I do not see the transition as leaving Decision Science behind. If anything, building AI systems has reinforced why I preferred the term Decision Scientist in the first place.

The model was never the objective. The prediction was never the objective. The dashboard was never the objective. And now the LLM is not the objective either.

The objective is the decision.

What has changed is the machinery available to support it. We can now combine predictions with documents, models with language, structured data with enterprise knowledge, rules with probabilistic reasoning, and automation with human judgment. We can orchestrate those capabilities inside workflows that do more than tell someone what happened. They can help determine what should happen next.

That, to me, is the real evolution from Decision Science to AI Engineering.

The engineering surface area expanded. The possibilities expanded. But the goal stayed the same:

Improve the quality, timing, and accountability of business decisions.

About the Author

Robb is the President and Principal Decision Intelligence Architect at Scope Analytics, helping organizations combine predictive signals, enterprise knowledge, AI reasoning, and human judgment into better decision systems.

Learn more: https://www.scopeanalytics.com