AGI Doesn't Need a Verdict. It Needs an ATO.

AIAGIevaluationAI ethicsinfrastructure

Every few weeks someone announces that AGI has arrived, and every few weeks someone else explains why it hasn’t. I’ve been reading both sides, and I’ve come to think they’re arguing about the wrong thing.

I’m not an AI researcher. I’m a systems administrator. I’ve spent about 30 years keeping military networks running, and a big part of that job is getting systems approved to operate on those networks. That process taught me something the AGI debate keeps missing: nobody serious asks whether a system is “good.” They ask whether it’s good enough, for a specific job, in a specific configuration, right now, and who is willing to sign for it.

That’s the question I want to bring to AGI.

Why nobody can agree

AGI stands for artificial general intelligence. The rough idea is an AI that can do most of the thinking work a capable person can, across many fields, instead of being brilliant at one narrow thing. The trouble starts the moment you try to pin that down.

There is no agreed definition of intelligence. Psychologists have argued about it for over a century, and that’s just for humans. Every AGI claim quietly picks a definition first.

“Human-level” isn’t one level. People vary enormously. Is the bar an average adult, a trained professional, or the best expert alive? An AI that beats a junior programmer and loses to a senior one is both above and below “human level” at the same time.

Today’s AI is jagged. It can earn a gold medal at a math olympiad, then forget what you told it last week. An average score hides that shape completely.

The word has money attached. Declaring AGI brings attention and investment, and at least one major industry partnership has had contract terms tied to it. Announcements aren’t neutral.

So when two people argue about whether we’re at AGI, they’re often using different definitions and different thresholds. Both can be right, which means the argument goes nowhere.

What the serious attempts look like

Researchers have already done a lot of careful work on measuring this. Before I add anything, it’s worth knowing the main approaches.

Approach What it measures What it found
A Definition of AGI (Hendrycks, Bengio and 31 others, 2025) Ten cognitive abilities borrowed from human psychology tests, weighted equally, scored 0 to 100% against a well-educated adult GPT-4 scored 27%, GPT-5 57%. Long-term memory storage scored zero.
Levels of AGI (Google DeepMind, 2023) Performance against percentiles of skilled adults (Competent = 50th, Expert = 90th, Virtuoso = 99th), with autonomy rated separately Rated the chatbots of the time “Emerging AGI,” the lowest general level
On the Measure of Intelligence (François Chollet, 2019) and the ARC-AGI puzzles How efficiently a system learns a new skill from a few examples, not what it already knows Puzzles most people find easy stayed hard for AI long enough that the benchmark has been revised as models caught up
METR time horizons The length of task, in human working time, an AI agent can finish at a given success rate Doubled roughly every 7 months from 2019; since 2023, about every 131 days

Each of these is thoughtful and worth reading. Together they agree on the big picture: today’s systems are extraordinary in some areas, weak in others, and improving fast. Greg Kamradt has gone further and proposed a verification ladder built around how far we can trust AI output.

That last idea is the one I want to push on. Most people don’t really want to know whether an AI is “general.” They want to know whether they can trust it with a particular job.

How the military decides to trust a system

In the Department of Defense, no system goes on the network until it has an Authorization to Operate, or ATO. Getting one is slow and often painful. But the logic underneath it is sound, and it fits the AGI problem surprisingly well. Here’s the short version.

Draw the boundary. You state exactly what the system is: hardware, software, versions, connections, and who uses it. Anything outside that line isn’t covered.

Test against known requirements. The system is checked against security controls and hardening checklists (STIGs, for the DoD folks), and the results are documented.

Write down what’s still wrong. Known weaknesses go into a Plan of Action and Milestones, or POA&M. Nobody pretends the system is perfect. The flaws are listed, with plans and dates.

A named person accepts the risk. An Authorizing Official signs off, for a specific mission in a specific environment. That signature is personal accountability.

Keep watching. Continuous monitoring runs after approval. The authorization expires, often within three years, and a significant change to the system reopens it.

An ATO never says “this system is secure.” It says “the remaining risk is acceptable for this purpose, and here is who decided that.” That difference is the whole point.

Four lessons for the AGI debate

You authorize a system, not a component. An ATO covers the whole configuration, never a single part. An AI model on its own is like a processor. What people actually use is the model plus its instructions, tools, memory, and whatever human checks its work. The same model can be excellent in one setup and unreliable in another. A claim about “the model” with no setup attached is like approving a server without saying what’s installed on it.

Approval expires. Systems change, and so does the world around them. AI models get updated quietly, tools get added, and behavior shifts. A one-time “it’s AGI” announcement is a snapshot pretending to be a permanent fact. Any capability claim should carry a date and a configuration, and get rechecked when either one changes.

Reliability beats the best result. In operations, nobody cares about your best day. They care about the worst one out of a thousand. Benchmarks usually report what a system can do; operators need to know what it does every time. METR’s own charts make the gap visible: the length of task an AI finishes 80% of the time is far shorter than the length it finishes half the time. For anything that matters, the high-reliability number is the one to watch.

Someone has to sign. “Is it AGI?” has no owner. “Is this AI good enough to patch my servers unattended?” does: the person who answers for it if something breaks. Trust is always scoped to a job, and to a name.

Picture two people looking at the same AI agent, with the same benchmark scores. One runs a small online shop and wants it to answer customer emails overnight. The other runs IT for a hospital and wants it to apply software patches to hundreds of servers.

The shop owner can reasonably say yes after a week of spot-checking replies, because a bad email costs an apology. The hospital administrator should want much more: a fixed list of what the agent may touch, a log of every action, a test environment first, a rollback plan, and a failure rate measured over thousands of runs. A bad patch at 3 a.m. can take down patient records.

Neither of them needs to know whether the agent is “AGI.” Each needs to know whether it’s trustworthy for their job, and each is the one who signs.

For the engineers reading this

I won’t pretend to teach evaluation researchers about psychometrics or benchmark design. But operations people have decades of practice at one narrow problem: deciding when to trust a complex system, and recording that decision so someone else can check it. If I were writing requirements for an AGI-style evaluation, these are the parts I’d borrow.

Publish the system boundary. Model version, system prompt, tools and permissions, memory setup, and how much human help was involved. Without that, a result can’t be reproduced or compared.

Report the worst case, not just the average. Variance across runs, success rates at high reliability thresholds, and every excluded or failed run.

Keep a known-weaknesses list. The AI version of a POA&M: documented failure modes that travel with every capability claim, instead of living in a footnote.

Define re-evaluation triggers. A model update, a new tool, a longer context window, or a new deployment environment should reopen the evaluation, the way a significant change reopens an ATO.

Scope every claim to a mission. “Trusted for unsupervised software maintenance at 99% task success, in this configuration, as of this date” can be checked. “General intelligence” can’t.

Pieces of this already exist. METR reports more than one reliability threshold, and the Hendrycks framework names memory as a specific gap. What’s missing is one record that pulls them together and says what a system is trusted to do, for whom, and until when.

So, are we at AGI?

By some definitions, arguably yes. By others, clearly not. I’ve stopped finding that argument useful.

The better questions are the ones I’d ask about any system before it goes on my network. What exactly is it? What is it approved to do? How often does it fail? Who signed off? When do we check again?

Those questions have answers. “Is it AGI?” mostly has opinions.

Sources