Why System 3 thinking needs an operational layer
When AI helps people find procedures, review documents or summarise live information, it is no longer just a background tool. It starts to influence how decisions are made. That creates value, but it also creates a risk of handing over too much judgement to a system that feels clear, quick and capable. This is why System 3 thinking needs an operational layer.
From quick thinking and careful thinking to outsourced thinking
A helpful place to start is with a simple idea that became well known through the Nobel Prize winner Daniel Kahneman’s work. People make decisions in two broad ways. Sometimes decisions are quick and come from habit, familiarity, instinct and pattern recognition. Other times decisions are slower. People pause, weigh things up, compare options and think more carefully.
Most people already know this from everyday life, even if they do not use the labels. In marketing, for example, persuasion often works by making something feel easy. A message is short, familiar, repeated, visually clean and simple to process. That ease can make it feel more convincing before anyone has really examined it closely.
Recent work by Steven Shaw and Gideon Nave argues that AI is starting to act as a third way of making decisions: a form of artificial cognition that sits outside the brain and helps retrieve, structure and present information for us. Their argument is especially relevant because they also warn about “cognitive surrender”, where people may hand off more judgement to AI than they realise.
That is why this matters in high-consequence decision-making. If a person uses AI for something low-stakes, the downside of over-trusting it may be limited. If a duty manager, control room operator or safety lead starts leaning too heavily on AI during an incident, a document review or a live operational decision, the consequences can be much more serious. In those settings, the issue is not just whether AI is helpful. It is whether people are starting to outsource too much of the thinking itself.
Why this matters in real operations
This matters because real operational decisions are rarely made from instinct alone or from quiet reflection alone. Most of the time, people are working with a mix of things around them: approved procedures, previous incidents, live updates, handover notes, decision logs, and checks built into the task itself.
In other words, judgement in operations is not just something that sits inside one person’s head. It is shaped by the information they can reach, the records they trust, and the tools they use to make sense of what is happening.
That is why AI matters more than it may first appear. Once AI starts helping someone pull back the right procedure, compare one document with another, summarise several incoming reports, or highlight what looks most important, it is no longer just saving time in the background. It is helping shape the decision itself.
That can be genuinely useful. A duty manager under pressure may need the right emergency procedure quickly, not after ten minutes of searching. A safety lead may need help spotting whether a contractor submission misses something important. A control room operator may need a clear summary of several messages arriving at once.
In each case, the AI is doing more than helping with admin. It is becoming part of how the person understands the situation, what they focus on, and what they do next. That is what makes this more than a technology story. It is a decision-making story.
And in high-consequence settings, that matters a great deal. Once a tool starts affecting what people see, what they retrieve and what they treat as important, it starts to affect judgement as well.
The risk is not just bad answers. It is offloading too much judgement
When people talk about AI risk, they often focus on the obvious failure. The answer is wrong. The summary misses something important. The system pulls back the wrong document. Those risks are real, but they are not the whole problem.
A more important risk may be what happens when the answer sounds good. Recent work by Steven Shaw and Gideon Nave at Wharton draws a useful distinction here between task offloading and cognitive surrender. Task offloading is familiar enough. People have always used tools to reduce mental effort, whether that is a calculator, a sat nav or a checklist. Cognitive surrender is different. In their framing, it is the tendency to defer judgement, effort and responsibility to AI output, especially when the system appears capable and confident.
That distinction matters in operations. Offloading a simple task is not necessarily a problem. Letting the system do too much of the actual judging is. In the Wharton research, Shaw and Nave argue that once people begin to rely on this third layer too easily, their accuracy starts to track the quality of the AI itself. When the AI is right, performance rises. When it is wrong, people can go wrong with it. That is the real warning behind cognitive surrender.
In practice, that can look quite ordinary. A duty manager may accept a summary without checking the underlying procedure. A reviewer may trust that a contractor document has been assessed properly because the output looks thorough. A control room operator may follow the suggested next step without noticing that the system has pulled from an older version of the plan. None of this has to involve a dramatic failure. The risk is that the person is no longer just using the tool to reduce effort. They are starting to let it carry more of the judgement than they realise. That is much closer to surrender than support.
That is why outsourced thinking needs to be treated carefully in high-consequence work. The danger is not just poor output. It is the gradual habit of handing over too much judgement because the system feels useful, clear and easy to trust.
Why this needs an operational layer
If AI is becoming part of how people make decisions, then some degree of offloading is probably inevitable. That should not be surprising. People have always used tools to reduce effort. The difference now is that the tool is starting to help with parts of judgement itself.
In that sense, cognitive surrender is not something to deny as if it can be avoided completely. It is more realistic to treat it as a pressure that needs a counterbalance. Shaw and Nave’s work is useful here because their concern is not just that AI may be wrong. It is that people may gradually hand over too much judgement to a system that feels capable and easy to trust.
There is a simple everyday parallel. People did not start going to the gym because exercise suddenly became fashionable for no reason. In part, they did it because modern life made movement easier to avoid. Once the environment changed, people needed a counter to stay healthy. The same basic logic applies here. Once AI makes it easier to hand over parts of thinking, organisations need a counter that helps keep judgement strong.
Take RAMS review. If AI helps check structure, flag gaps, highlight vague responsibilities and point to likely weak spots, that can save time and improve consistency. But if people rely on that support every time without ever going back to review a full RAMS themselves, their own judgement can start to soften. They may become less sharp at spotting what the system misses, less confident in challenging a neat-looking answer, and less familiar with the detail of what good really looks like.
That is where the counter matters. In practice, it may mean that some reviews are still done fully by hand, even if AI could assist. Not because the technology has failed, but because the person needs to keep using their own judgement often enough to stay strong. In the same way that modern life makes movement easier to avoid, AI makes thinking effort easier to avoid. The answer is not to reject the convenience. It is to build in a healthy counter to it.
That is part of what an operational layer should do. It should not only improve the use of AI. It should also help preserve the human capability that sits alongside it. Otherwise the team may become faster in the short term, but weaker over time.
Ways to spot when System 3 is making people too dependent on AI
Human factors research has been warning about versions of this problem for years, long before today’s AI boom. The language is not always the same, but the pattern is familiar: when people rely too heavily on automation, they can monitor less carefully, question less often and become slower to catch mistakes. That older work on automation bias and overreliance fits closely with the newer warning from Shaw and Nave about cognitive surrender.
If AI is becoming part of how people think, organisations need a way to spot when support is turning into dependence. Some signs are easy to miss because they do not look dramatic at first. They often show up as small changes in working habits.
People stop going back to the source material
The summary, recommendation or flagged issues start to feel enough on their own. Fewer people open the underlying procedure, review the full document or check the original wording for themselves.
People become less confident doing the task without assistance
A reviewer who used to work through a RAMS confidently may begin to feel slower or less sure without the AI doing the first pass. That is often a sign that the underlying judgement muscle is weakening.
Polished answers start carrying too much weight
The output is neat, quick and well written, so it feels more reliable than it really is. The danger is not just a wrong answer. It is a wrong answer that does not look wrong.
People struggle to explain why an answer is right
If someone accepts an AI-supported recommendation but cannot explain the reasoning, point to the source or describe what made it sound right, the system may be doing too much of the thinking.
Checking becomes less consistent over time
At the start, people verify carefully. Later, once the tool feels familiar, the checking becomes lighter. This is often how over-dependence creeps in: not all at once, but gradually.
Manual skill starts to fade
When people rarely carry out the task unaided, they may become less sharp at spotting what the AI has missed. In RAMS review, for example, they may lose confidence in reading the document cleanly from first principles.
Teams start trusting speed over scrutiny
If quick answers are regularly valued more than careful checking, the organisation may be drifting towards convenience at the expense of judgement.
Repeated errors are missed because the AI usually sounds convincing
A system that is right most of the time can become harder to challenge. That can make familiar mistakes more dangerous, because people stop looking for them.
The point is not to remove AI support. It is to notice when useful support is starting to weaken the human judgement around it. That is the line organisations need to watch.
The real aim is not to remove judgement, but to strengthen it
The point of this article is not that organisations should avoid AI because people may lean on it too much. Offloading some effort is part of how tools work. The more useful question is whether that offloading is making decisions better, or simply making thinking easier to avoid.
That is why System 3 matters in high-consequence work. If AI is becoming part of how people retrieve information, compare options, spot patterns and shape next steps, then it is no longer just a convenience tool. It is becoming part of the decision process itself. Shaw and Nave’s warning about cognitive surrender matters because it names a real risk: the system may not just help people think, it may tempt them to hand over too much of the thinking.
The answer is not to pretend that risk will disappear. It is to build around it properly. That means treating AI as something that needs an operational layer around it, especially in serious environments. It means using AI to support judgement, while also preserving the habits, checks and manual capability that keep judgement strong. The older human factors literature points in the same direction. When automation becomes too easy to trust, people can become less effective at monitoring it and less likely to catch mistakes.
So the real aim is not to remove human judgement from operational work. It is to make that judgement more effective by surrounding AI with the right evidence, working habits and checks. In that sense, the best use of System 3 is not outsourced thinking without limits. It is supported thinking with boundaries.
That is the standard worth aiming for. In high-consequence settings, AI should not quietly replace judgement. It should help strengthen it.
Author bio: Andrew Tollinton
Andrew Tollinton is CEO and Co-Founder of SIRV, which builds operational AI for safety, security and resilience teams. He focuses on practical, controlled AI use in serious environments, with particular interest in evidence, accountability and human judgement. Andrew chairs the Institute of Strategic Risk Management’s AI in Risk Management Special Interest Group and speaks regularly on AI governance and operational resilience.
Frequently asked questions
What is System 3 thinking?
In this article, it means AI acting as a third layer in decision-making by helping retrieve, structure and present information for people. That usage is drawn from recent work by Steven Shaw and Gideon Nave.
What is cognitive surrender?
It is Shaw and Nave’s term for handing over too much judgement, effort or responsibility to AI when the system appears capable and convincing.
Why does this matter in high-consequence work?
Because overreliance on automation has long been associated in human factors research with weaker monitoring and reduced error detection, and similar concerns now apply to AI-supported decisions.
What is the role of an operational layer?
It is the practical structure around AI use that helps keep support bounded, checkable and useful in real work rather than allowing unchecked delegation.
"SIRV helped us move beyond basic reporting into a system that actively supports decision-making". Les O'Gorman, Director of Facilities, UCB - Pharma and Life Sciences
