AI, trust and the human factor: Rethinking risk and responsibility

📆  22 November 2023

📍Online

In the fast-moving world of artificial intelligence, it’s easy to get swept up in the breakthroughs, buzzwords, and bold predictions. But beneath the surface of every shiny new model lies a more fundamental challenge: trust.

At the heart of this challenge is an uncomfortable truth, human nature hasn’t changed. Our core desires, fears, and behaviors remain largely the same, regardless of the tools we use. Just as we once moved from gramophones to MP3s to streaming in pursuit of music on demand, we are now exploring how AI can deliver faster, better decisions, especially in high-stakes environments like healthcare and disaster response.

But are we really ready to trust the machine?

Introduction

At the ISRM conference on 22 November 2023, speakers from academia and industry explored a practical question. How do we build AI that risk professionals can trust. The answer begins with human factors: clear explanations, usable interfaces and governance that keeps pace with deployment.

The promise and the problem

AI can analyse images, events and text at speed, but adoption depends on confidence. In domains like healthcare and disaster response, human–AI teams only outperform when the workflow supports understanding, not blind acceptance.

Why black boxes stall adoption

People will challenge recommendations that defy experience. If decision-makers cannot see why a system suggests action, they will reject it. Explainability is not optional in high-stakes environments.

Design is a risk control

Trust grows when tools are built for the people who use them:

  • Plain language, predictable layouts and readable charts

  • Evidence surfaces next to recommendations

  • Simple controls to accept, override or request a re-check

  • Audit trails for accountability

Deployment in the real world

Damage-assessment models can classify buildings across a city in seconds from aerial imagery and guide search and rescue while manual surveys are still mobilising. When interfaces assume expert users, adoption lags. The fix is inclusive design: match capability to the operator, not the other way round.

Governance and incentives

Oversight needs to be practical, not performative. Align incentives so teams are rewarded for transparency and safe deployment, not just speed. Regulation will catch up, but buyers should demand clear model documentation, data provenance and routes to human review.

Key takeaways

  • Trust is built through workflow, not slogans

  • Explanations must live next to recommendations

  • Human-in-the-loop beats human-out-of-the-loop

  • Design for non-technical users first

  • Demand governance: documentation, provenance and audit

Transcript

Event: ISRM Conference — AI, trust and the human factor: rethinking risk and responsibility in the age of automation
Date: 22 November 2023
Format: Online
Speakers:

  • Dr David Rubens (Executive Director, ISRM)

  • Andrew Tollinton (Co-founder, SIRV)

  • Ritwik Gupta (Research Fellow, Berkeley Risk and Security Lab)

  • J Scott Christianson (Associate Teaching Professor and Director, Centre for Entrepreneurship and Innovation, University of Missouri)

Opening

Human nature has not changed. Our desires and behaviour are consistent over time, even as technology changes the way we meet them. Just as we moved from gramophones to streaming to access music on demand, we are now exploring how AI can deliver faster, better decisions. The question is not only what AI can do, but whether we trust it.

Power, trust and regulation

Power can corrupt priorities. Even well-intentioned innovators can drift as influence grows. Oversight and governance matter, yet regulation often fights yesterday’s battles. Translating high-level principles into daily safeguards is the hard part.

Human–AI performance depends on trust

A study paired radiologists with a disease-detection model that outperformed most individuals. Results got worse in the human–AI team because doctors distrusted the model’s decisions. In contrast, centaur teams in chess outperform either humans or machines alone. The difference is workflow and trust.

The black box dilemma

Leaders will challenge recommendations that cut against experience. If a system cannot explain why it suggests an action, confidence collapses. This is not only technical. It is a human–computer interaction issue that needs iterative design with end users from the start.

Cutting-edge deployments

From satellite damage assessment to medical support, AI can deliver rapid, high-quality outputs. In disaster response, models can label damaged buildings across a city within seconds and align closely with later manual surveys. Yet uptake is slow when tools assume technical familiarity that does not exist. That is a design failure, not a user failure.

Building usable, trustworthy AI

Priorities for deployment:

  • Design for non-technical users with clear language and legible interfaces.

  • Explain decisions with evidence, not only scores.

  • Keep human-in-the-loop controls to accept, override or request verification.

  • Involve UX specialists alongside data scientists and domain experts.

Closing thought

AI will not replace decision-makers, but it will redefine decision-making. To work for everyone, not just the technically confident, systems must embed empathy, ethics and usability at their core.

Get in touch

Frequently asked questions

Q1. What stops AI adoption in risk management
Lack of understandable explanations and interfaces that assume technical expertise. Fixing these increases trust and uptake.

Q2. What does human-in-the-loop mean here
Humans can accept, reject or request re-checks on AI outputs, with clear evidence and an audit trail.

Q3. How should recommendations be explained
Show the factors that influenced the recommendation, relevant evidence (for example, imagery snippets or event matches) and confidence ranges.

Q4. Do high-performing models always improve teams
No. Teams can perform worse if they do not trust or understand the system. Workflow design matters as much as accuracy.

Q5. What governance should buyers require
Model documentation, data provenance, tests for bias, routes to human review and clear incident response if outputs are challenged.

Q6. How do we start if our team is not technical
Begin with a narrow use case, run a short pilot, use plain-language interfaces and measure a small set of outcomes tied to operations.

Q7. Where does SIRV fit
SIRV supports describe, predict and prescribe workflows. Cal helps surface signals, make recommendations visible and keep humans in control.

Q8. What metrics show success
Time to first insight, decision accuracy, operator confidence scores, reduction in disruption minutes and post-incident review quality.

css.php