Why Testing AI for Safety Is Necessary — But Still Not Enough

<div class=”tw:border-b tw:border-slate-200 tw:pb-4″>
<h2 class=”tw:mt-0 tw:mb-1 tw:text-2xl tw:font-heading”>Key Takeaways</h2>
<ul class=”tw:font-normal tw:font-serif tw:text-base tw:marker:text-slate-400″>
<li>Testing shows what happened; formal methods define what failures are impossible.</li>
<li>AI safety is a leadership risk decision, not an engineering optimization problem.</li>
</ul>
</div>
<p>I have spent a lot of time watching smart teams do the wrong thing for very rational reasons. When leaders <a href=”https://www.entrepreneur.com/leadership/why-ai-is-a-leadership-problem-not-a-tech-one/502359″ rel=”” target=”_self”>talk about AI risk</a>, the conversation usually collapses into testing, with bigger eval suites, red teams and synthetic data.</p>

<p>The instinct is understandable. Testing feels concrete. You can point to dashboards. You can say you ran 10,000 cases instead of 1,000, and it looks like progress.</p>

<p>The problem is that this approach quietly assumes something false: That you can test your way to safety. There are infinitely many possible inputs to any <a href=”https://www.entrepreneur.com/growing-a-business/most-companies-say-they-use-ai-but-few-can-pass-this/500725″ rel=”” target=”_self”>nontrivial AI system</a>. No matter how large your test suite is, it is still a rounding error.</p>

<p>At best, testing tells you what happened on a narrow slice of reality. It does not tell you what cannot happen. For leaders, that distinction is everything.</p>

<p>This is a decision about how you manage risk in systems you’re responsible for and that are already reshaping how <a href=”https://strixus.com/entry/neel-somani-on-the-new-invisible-hand-how-ai-is-reprogramming-capital-attention-and-human-value-18453″>capital, attention and value flow</a>, even if you do not fully understand them yet.</p>

<h2 class=”wp-block-heading”>Sampling feels like control, but it’s not</h2>

<p>Executives are used to operating in environments where sampling works. Instead of interviewing every customer, a leader might talk to a handful and infer broader patterns.</p>

<p>In systems with bounded behavior, this is efficient. But AI systems are not stable in that way.</p>

<p>A modern model is a large, adaptive software artifact. Small changes in input can produce qualitatively different behavior.</p>

<p>When you test such a system, you are making a probabilistic claim. You are saying, “We did not see a failure in these cases, so we believe failures are unlikely.”</p>

<p>That belief has failed before.</p>

<p>In Australia, the federal government deployed an automated welfare debt recovery system, <a href=”https://pursuit.unimelb.edu.au/articles/the-flawed-algorithm-at-the-heart-of-robodebt”>later known as Robodebt</a>. It had been reviewed, scaled and used across hundreds of thousands of cases. On paper, it appeared to be working.</p>

<p>What it had not done was rule out a specific class of invalid assumptions about how income could be averaged over time. That single design flaw produced false debt notices, ultimately forcing the program’s shutdown.</p>

<p>This is the core problem with sampling. It tells you what happened in the cases you tried, but nothing about what <i>cannot</i> happen.</p>

<p><a href=”https://www.entrepreneur.com/leadership/quality-leadership-should-be-the-top-priority-in-the-ai-era/497746″ rel=”” target=”_self”>Leaders care about tail risk</a>. They care about the rare cases that matter most, because those are the ones that trigger regulatory scrutiny, reputational damage and real-world harm. A single bad output can outweigh thousands of successful tests.</p>

<p>If your safety strategy relies entirely on testing, you are accepting that certain failures are inevitable and hoping they do not happen on your watch.</p>

<h2 class=”wp-block-heading”>This is a risk-management bet</h2>

<p>The right question for leaders is not, “What is the best AI safety technique?” It is, “Where should we invest if we want fewer catastrophic surprises?”</p>

<p>That is why I think formal methods matter. Formal methods are not magic. They do not prove that a system will never do anything bad. That framing is a category error.</p>

<p>What they do is narrower and more powerful. They let you make exact claims about specific properties of a system and let you say that, within a defined boundary, certain classes of behavior are impossible.</p>

<p>That is fundamentally different from testing.</p>

<p>Testing asks, “Did this happen in the cases we tried?” Formal methods ask, “Can this happen at all, given these constraints?” Leaders should recognize the difference immediately. One is reactive. The other is preventative.</p>

<p>For example, Rust did not make software bug-free. What it did was eliminate entire classes of errors by construction. Memory safety issues went from being something you tested for to something the system would not allow.</p>

<p>That shift changed how teams <a href=”https://www.entrepreneur.com/leadership/the-hidden-risks-of-ai-and-ai-powered-digitization-and/452777″ rel=”” target=”_self”>thought about risk</a>. They stopped playing whack-a-mole with bugs after deployment and started preventing them before code shipped.</p>

<h2 class=”wp-block-heading”>Fewer surprises, clearer guarantees</h2>

<p>From a leadership perspective, the value of formal methods shows up in three places:</p>

<p><b>1. First, they force clarity.</b></p>

<p>To apply any formal technique, you have to specify what you care about.</p>

<ul class=”wp-block-list”><li>What inputs are in scope?</li><li>What behaviors are forbidden?</li><li>What invariants must hold?</li></ul>

<p>Many AI failures happen because <a href=”https://www.entrepreneur.com/science-technology/why-every-company-will-need-an-ai-specialist-by-2026/499538″ rel=”” target=”_self”>nobody was explicit about boundaries</a>.</p>

<p><b>2. They reduce surface area.</b></p>

<p>If you can formally guarantee that a certain pathway cannot activate under certain conditions, you do not need to test every edge case related to that pathway. You have collapsed an infinite space into a finite claim.</p>

<p><b>3. They enable credible communication.</b></p>

<p>When something goes wrong, leaders are asked hard questions.</p>

<ul class=”wp-block-list”><li>What happened?</li><li>Why did it happen?</li><li>Could it happen again?</li></ul>

<p>“We tested extensively” is not a satisfying answer. Being able to say, “This behavior is impossible within this defined scope, and here is why,” is much stronger.</p>

<p>None of this requires formal methods to be ready at full scale today. Leaders invest in directions. The question is whether you want your organization to build muscle around pre-deployment guarantees or to double down on post-hoc testing forever.</p>

<p>Every time I look at large, safety-critical systems, the lesson is the same. The most effective risk reduction comes from eliminating entire classes of failure early, not from catching individual failures late.</p>

<p>AI will not be different.</p>

<h2 class=”wp-block-heading”>Choosing where to be early</h2>

<p>There is a temptation to wait, because formal methods sound academic, the tooling is immature and the talent pool is small.</p>

<p>All of that is true, but it was also true for many technologies that later became table stakes.</p>

<p>Leaders are paid to make asymmetric bets, and this is one of them. You can keep pouring resources into larger test suites and hope that covers the risk you care about, or you can start investing in approaches that give you stronger guarantees, even if they are not yet perfect.</p>

<p>The payoff is confidence. Confidence that when you deploy a system, you understand what it can and cannot do, that when something breaks, you are not flying blind and that you are managing AI like the engineered system it is, not like a black box you poke and pray.</p>