Why System 3 thinking needs an operational layer
AI can make operational decisions faster. The harder question is whether it also makes people less likely to exercise their own judgement.
When AI helps people find procedures, review documents or summarise live information, it is no longer just a background tool. It begins to influence how decisions are made. That creates value, but it also creates a risk: handing over too much judgement to a system that feels clear, quick and capable.
From two systems of thinking to a third
Daniel Kahneman's familiar distinction is between fast, intuitive thinking and slower, more deliberate reasoning. Steven Shaw and Gideon Nave have proposed that AI introduces a third form: artificial cognition outside the individual that retrieves, structures and presents information.
This matters especially in high-consequence work. A duty manager, control room operator or safety lead may use AI to identify relevant procedures, compare submissions or interpret multiple reports. In each case, AI helps determine what the person sees and what they treat as important.
Offloading work is not the same as surrendering judgement
People have always used tools to reduce effort, from calculators to checklists. Shaw and Nave distinguish that ordinary task offloading from cognitive surrender: deferring judgement to AI because its output appears capable and convincing.
The risk is not confined to an obviously incorrect answer. A polished summary can be accepted without the source being opened. A convincing document assessment can discourage a reviewer from testing the underlying reasoning. A suggested action can be followed even when it relies on an outdated procedure.
As reliance increases, human performance can become more dependent on the quality of the AI output. When the system is wrong, people may follow it into error rather than independently detecting the problem.
What an operational layer should do
An operational layer should make AI useful within defined boundaries. It should connect outputs to approved sources, make uncertainty and gaps visible, preserve review points and maintain a record of consequential decisions.
Consider reviewing risk assessments and method statements (RAMS). AI can identify omissions, ambiguous responsibilities or gaps against expected requirements. But reviewers must remain capable of assessing the document independently, including noticing issues the system did not flag.
For some tasks, periodic unaided review may be worthwhile, not because AI has failed, but because the organisation needs to preserve human capability alongside it.
Eight signs that assistance is becoming dependence
Overreliance can emerge gradually. Research on automation bias identifies factors such as trust, workload and decision-support design. The following are practical warning signs, not a validated diagnostic score.
| What changes | What to examine |
|---|---|
| Sources are no longer opened | Whether staff verify original procedures and records |
| Unaided work feels difficult | Whether task competence is being maintained |
| Polished outputs are trusted | Whether presentation is being mistaken for reliability |
| Reasoning cannot be explained | Whether decisions are supported by identifiable evidence |
| Checking declines over time | Whether verification habits weaken as familiarity grows |
| Manual skills deteriorate | Whether reviewers can still recognise omitted or flawed analysis |
| Speed outweighs scrutiny | Whether performance measures reward checking as well as throughput |
| Convincing errors recur | Whether recurring failure patterns are detected and corrected |
Preserve the judgement that AI is meant to support
The purpose of an operational layer is not to make every decision slower. It is to distinguish work that can safely be accelerated from decisions that still require deliberate human scrutiny.
Good operational AI should improve access to evidence, reveal uncertainty, support comparison and make consequential decisions easier to review. The outcome to aim for is not outsourced thinking without limits, but supported thinking with boundaries.
Applying these principles to an operational workflow?
Discuss how evidence, verification and human approval can be built into a defined safety, security or resilience workflow.
Discuss a workflowSources and further reading
- Daniel Kahneman, Thinking, Fast and Slow (2011).
- Steven Shaw and Gideon Nave, research on artificial cognition and cognitive surrender. Find the paper on SSRN.
- Kate Goddard, Abdul Roudsari and Jeremy C. Wyatt, Automation bias: a systematic review of frequency, effect mediators, and mitigators, Journal of the American Medical Informatics Association, 19(1), 121–127 (2012).
