SIRV / Insights / AI governance
The checkbox trap: Why CCTV is a warning for enterprise AI
Deploying a technology is not the same as developing the operational capability to use it safely and effectively.
I recently spoke with students on the University of Portsmouth's MSc Risk, Crisis and Resilience Management programme. Our discussion returned to a familiar pattern: organisations can install technology faster than they can build the processes, skills and accountability needed to make it useful.
CCTV scales visibility. AI scales the production of outputs. Neither creates reliable outcomes without an operating model around it.
CCTV
“We have cameras.”
- Cameras installed, but feeds are not consistently monitored
- Alerts generated without a practical response
- Coverage measured instead of prevention or response
- Visible infrastructure creates false reassurance
Enterprise AI
“We use AI.”
- Outputs generated, but evidence is not verified
- Recommendations produced without clear ownership
- Adoption measured instead of decision quality
- Fluent answers create false confidence
The comparison is conceptual: technology deployment is an input, not proof of operational effectiveness.
Instrumentation scales faster than oversight
Cameras are tangible. They can be purchased, counted and installed. Yet their operational value depends on monitoring, triage, escalation, response and learning. An alert that nobody can act on is not the same as a functioning security control.
AI presents a similar challenge, often at greater speed. Teams can generate summaries, classifications, recommendations and draft actions in large volumes. The limiting factor may no longer be production. It may be the organisation's capacity to verify and act on what is produced.
When output grows faster than verification, apparent capability can increase while assurance deteriorates. That is the checkbox trap.
Why AI outputs are particularly persuasive
CCTV footage is recognisably a recording requiring interpretation. AI-generated text can look like a finished analysis. Its fluency may conceal missing evidence, incorrect assumptions or an untested conclusion.
AI systems are powerful pattern learners, not independent guarantees of truth. They can help retrieve, compare and synthesise information, but consequential conclusions still require evidence, context and accountable judgement.
Two paths for workforce capability
Experienced practitioners can use AI to explore more evidence, test alternatives and interrogate assumptions. Their domain knowledge helps them recognise omissions and challenge plausible but incorrect outputs.
Others may become dependent on AI without developing the ability to check its work. Output volume rises, but independent judgement does not necessarily improve. This makes deployment a question of training and operating design, not simply software access.
Well-designed workflows can reduce this gap by making evidence checks and escalation requirements part of the process rather than optional good habits.
The size of the opportunity does not remove the need for controls
The potential for automation is substantial. McKinsey Global Institute has estimated that demonstrated technologies could, in theory, automate activities accounting for approximately 57% of US work hours. This concerns activities, not a prediction that 57% of jobs will disappear.
Source: McKinsey Global Institute research. The figure describes technical potential, not observed adoption or realised productivity.
The larger the opportunity, the greater the temptation to deploy quickly. Speed is valuable, but in high-consequence work the ability to verify, escalate and account for decisions must develop alongside it.
From installed technology to governed capability
For CCTV, effective operation connects detection to assessment, escalation and response. Enterprise AI needs a comparable chain. The details differ by workflow, but the principle is the same: evidence and human accountability must survive the move from input to action.
Illustrative control sequence. The level of review should reflect the consequence of the task.
What leaders should measure
A deployment dashboard can tell you how many users have access or how many outputs were generated. It cannot, on its own, tell you whether operational decisions are more reliable.
| Deployment measure | Operational assurance measure |
|---|---|
| Number of CCTV cameras | Useful detections, response time and resolved incidents |
| Number of AI users | Validated outcomes in defined workflows |
| Number of AI-generated outputs | Accuracy, exceptions and time to verified decision |
| Number of workflows automated | Appropriate human review, traceability and error rates |
Measures should be selected for each workflow, not treated as universal KPIs.
Five practical controls
Bound the use case. Specify the task AI is allowed to support and what it must not decide.
Connect outputs to evidence. Show the source material, provenance and limitations behind consequential claims.
Make review explicit. Identify who verifies, approves and escalates, and under what conditions.
Test real scenarios. Evaluate performance against representative examples, including known failures and difficult edge cases.
Monitor outcomes. Measure accuracy, elapsed time, review burden and operational results, not adoption alone.
What about AGI?
Experts disagree about how quickly AI capabilities will develop and which technical approaches will prove sufficient. Organisations do not need to settle those forecasts before improving their current controls. The operational question is already present: can a system's outputs be inspected, challenged and used responsibly?
The takeaway
“We use AI” is a statement about deployment. Capability is demonstrated when outputs can be interrogated, checked against evidence and connected to accountable decisions.
Governance is not merely a brake on adoption. Properly designed, it is what allows AI to be used in consequential security, safety and resilience workflows.
Discuss a workflow
If you are assessing where AI could help your team, start with one defined workflow: its evidence, decisions, controls and measurable outcomes.
Discuss a workflow →Andrew Tollinton · CEO and Co-Founder, SIRV. This article draws on a guest lecture and discussion at the University of Portsmouth.
