Soft failure: when AI makes progress feel real

When activity is mistaken for progress

A prospective client shows interest. We have an initial conversation and, for the next meeting, they invite more colleagues. Eight or ten people join. The meeting lasts an hour. People ask questions. There is discussion and apparent enthusiasm.

It feels like progress.

Then the meeting ends. There is no agreed problem to solve, no defined use case, no budget discussion, no owner and no next commercial step.

The meeting was real. The engagement was real. But curiosity is not the same as buying intent.

All the visible signals of progress were present, even though little had actually moved forward.

I think generative AI can create the same problem. It can give us the feeling that useful work has happened even when the underlying task is incomplete, weakly supported or unresolved.

An apparently successful interaction is not necessarily a successful outcome.

What is soft failure?

Most traditional software has recognisable limits. Ask Microsoft Word to make a financial transaction and it cannot do it. The failure is obvious.

Generative AI behaves differently. Ask it for something outside its actual capability and it may still produce something plausible: options, analysis, a plan or a polished answer. Nothing visibly breaks.

That is what I mean by soft failure: the system appears to make progress without necessarily achieving the intended objective.

Many operational AI tasks are not binary. We ask systems to assess a risk, review a document, challenge an argument or recommend an action. An answer may be excellent, partially correct or weakly supported, yet each may appear similarly fluent.

Why it can feel like a dream

Frequent use makes this harder. A few minutes with generative AI can produce pages of analysis, ideas, options and actions. There is so much visible output that the interaction itself feels productive.

This is where I find the analogy with dreams useful. I do not mean that AI is dreaming. I mean the experience for the user can become dream-like.

While we are inside a dream, events can feel coherent and convincing. It is only when we wake up that we test them against reality.

AI can create a similar feeling. We finish a long interaction believing that a problem has been analysed, a decision improved or a piece of work completed.

But what has actually changed? Has the evidence been checked? Has somebody made a decision? Has an action been taken? Has the underlying problem moved forward?

Or have we mainly produced a convincing description of progress?

Fluency is not evidence

Generative AI can make complex subjects readable at extraordinary speed. This is part of its value, but it also makes weak answers harder to spot.

Readable prose can suggest command of a subject even when relevant sources are missing, assumptions are unresolved or the task has been misunderstood. After a succession of good answers, the next one may receive less scrutiny, including when the consequences are greater.

The critical distinction is between the quality of the presentation and the reliability of the work beneath it.

Why this matters in operational work

In a low-consequence task, soft failure may waste time. In safety, security, risk and compliance, it can influence a decision that the organisation later needs to explain and defend.

Consider an AI-supported assessment of contractor RAMS, a security incident or an emergency plan. The output looks reasonable, so the reviewer accepts it and the workflow moves on.

Months later, an incident or challenge prompts different questions. Which evidence supported the assessment? What information was available? What was missing or uncertain? Who reviewed and approved the conclusion?

At that point, a polished answer is not enough. The organisation needs to establish how its decision was reached.

Designing workflows that expose failure

The response to soft failure should not be to abandon useful AI capability. It should be to design workflows that make the difference between an answer and an assured operational result visible.

Controls depend on the task. They may include approved sources, checks against the original evidence, explicit treatment of missing information, escalation thresholds, human verification, and a record of what was decided and why.

These measures cannot guarantee that every error will be detected. They make the system's limitations more likely to be recognised and challenged.

Three questions to test apparent progress

Whenever AI appears to have helped, I think three questions are worth asking:

Generative AI is unusually good at making progress feel real. The challenge is making sure it is.