What AI Capability Evaluations Actually Tell Us About Work

The number that sounds more decisive than it is

One of the most discussed measures of recent AI progress comes from METR, a nonprofit that evaluates advanced AI systems.

Its task-completion time horizon asks a deceptively intuitive question: how long are the tasks that an AI agent can complete at a specified probability of success, where task length is measured by how long a human expert needs to complete the same task? METR reports both a 50 percent and an 80 percent horizon.[1]

In a May 2026 frontier assessment, METR estimated that the strongest public models it evaluated during February and March had a 50 percent time horizon of roughly 12 hours and an 80 percent horizon of roughly 1.5 hours. The confidence ranges were wide, and METR explicitly warned that its current suite becomes unreliable above 16 hours.[2]

Read casually, a figure such as "12 hours" can sound like a measure of how many hours of a professional job an AI can perform. It is not.

METR is unusually explicit about this. Human task duration is being used as a measure of task difficulty. It is not the amount of clock time the AI spends working. The tasks are primarily software engineering, machine learning and cybersecurity tasks. They are intentionally self-contained, well-specified and automatically scorable. Human baseliners also begin with much less context than an experienced employee normally has inside a real organization.[1]

That distinction matters.

A senior engineer may spend two hours making a decision only because she has spent three years learning the codebase, knows which customer cannot tolerate a certain failure mode, remembers why an apparently strange architectural choice was made, and can walk across the room to ask somebody what changed yesterday. A benchmark can measure a difficult technical task without reproducing that organizational context.

METR also says directly that an eight-hour time horizon does not mean AI can automate all jobs. Most jobs contain interaction, tacit context, accountability and success criteria that cannot be reduced to a clean automated score.[1]

So the time-horizon results are consequential. They show that the difficulty of tasks frontier systems can sometimes complete autonomously has risen dramatically. But the metric becomes economically meaningful only after we ask several additional questions.

What capability evaluations actually measure

A capability evaluation is a controlled test.

Researchers define tasks, give a model tools and an environment, run repeated attempts, score the results, and estimate how performance changes as tasks become more difficult. A good evaluation can tell us whether the system crosses a capability threshold, how reliably it succeeds, where it fails, and how quickly those results are changing.

That is already valuable.

It is also narrower than a workplace.

METR's own methodology illustrates why. The current Time Horizon 1.1 task suite is dominated by software, machine learning and cybersecurity. The tasks are designed to be coherent and self-contained. They contain less pre-existing organizational context than ordinary work, and METR notes that AI performance falls on "messier" tasks.[1]

Reliability changes the picture too. The gap between a 50 percent success threshold and a more demanding threshold can be large. A system that succeeds half the time on a class of tasks may be useful for exploration, drafting or low-cost experimentation. It may be unacceptable for payroll, legal filings, production infrastructure or a decision whose failure is expensive.

The evaluation setup also matters. An "AI system" in an agent benchmark is not only a model. It includes a scaffold, tools, prompts, token limits, permissions and an interaction loop. Change those components and performance can change.

Finally, benchmark coverage is never complete. METR notes that its public time-horizon page may lag some releases and should not be treated as a complete record of every frontier model.[1]

None of these caveats make the evaluation unimportant. They tell us what kind of evidence it is.

Capability evidence answers:

Can the system perform this kind of task under these conditions?

It does not yet answer:

What happens to work?

For that, we need the rest of the ladder.

The missing ladder between capability and jobs

The most useful framework is simple:

1. Capability

Can the system perform the task at all under the evaluation conditions?

2. Reliability

Can it perform the task consistently enough for the intended use, at an acceptable error rate?

3. Usability

Can the capability be integrated into real work once context, supervision, interfaces, permissions, latency, exception handling and accountability are included?

4. Productivity

Does using the system actually make the worker, team or process faster, cheaper, better, or capable of producing more?

5. Adoption

Do organizations deploy the technology at meaningful scale, redesign workflows around it and continue using it?

6. Labor demand

After substitution, complementarity, lower prices, higher output, new tasks, organizational redesign and changes in demand are taken into account, what happens to the number and kinds of people firms want to employ?

A capability result is evidence relevant to the next step. It does not settle the next step automatically.

This is why forecasts that move directly from a benchmark score to a number of jobs lost should be treated cautiously. They have skipped most of the economic mechanism.

It is also why the opposite argument is too simple. A mixed workplace-productivity result does not prove capability progress is economically irrelevant. A capability can be real before organizations have learned how to use it well.

When capability meets real work

Workplace experiments show why the distinction matters.

In one of the best-known field studies, Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied 5,172 customer-support agents after a generative AI assistant was introduced. Access to the tool increased productivity by about 15 percent on average, measured as issues resolved per hour. Less experienced and lower-skilled workers improved substantially, including about a 30 percent increase in issues resolved per hour, while gains for more experienced and higher-skilled workers were small.[3]

That is not a benchmark result. It is evidence from a real production setting.

It is also not evidence that every knowledge worker becomes 15 percent more productive.

The task environment was customer support. The workflow was measurable. The AI could draw on patterns found in successful interactions. The benefits were heterogeneous across workers.

A second study, now published in Organization Science, makes the unevenness even clearer. Researchers working with Boston Consulting Group assigned 758 consultants realistic knowledge-work tasks. For tasks that fell inside the tested AI system's capability frontier, participants with AI completed 12.2 percent more tasks and worked more than 25 percent faster on average, with higher quality. But on a complex task deliberately chosen to sit outside that frontier, AI-assisted participants were 19 percentage points less likely to reach the correct answer.[4]

The same technology improved performance in one part of the workflow and damaged it in another.

That is the "jagged technological frontier": capability does not advance evenly across all tasks that look similar to a human observer.

A job is not one task. It is a bundle of tasks, relationships, judgments and responsibilities. If AI improves six parts of that bundle and weakens performance on the seventh, the economic outcome depends on which part matters most.

Real work can also contradict expectations

METR's randomized study of experienced open-source developers offers a useful counterweight to simple productivity stories.

Sixteen experienced developers completed 246 tasks in mature repositories they knew well. When early-2025 AI tools were allowed, the developers took 19 percent longer to complete the tasks. Before the study they expected AI to make them faster. Even after participating, they still believed AI had accelerated their work.[5]

The result is striking precisely because it runs against both the capability trend and the participants' own perceptions.

It should not be generalized beyond what the study measured. These were experienced open-source developers working in familiar codebases with a particular generation of tools. METR itself later changed the design of its follow-up productivity experiment because developers increasingly refused to participate when they might be assigned to work without AI, creating a selection problem that made the newer estimate unreliable.[6]

That methodological update is important.

It shows how quickly the measurement problem itself is changing. As AI tools become embedded in workflows, randomized comparisons between "AI" and "no AI" may become harder to conduct because experienced users reorganize their work around the tools.

More recent observational evidence from Anthropic's Claude Code sessions points in another direction again. In roughly 400,000 sessions from October 2025 through April 2026, people made most planning decisions while the agent made most execution decisions. Domain expertise remained positively associated with successful sessions, and more expert users elicited more work per instruction.[7]

These studies are not contradictory in a way that requires choosing a winner. They observe different people, systems, tasks and moments in a rapidly changing transition.

That is the point.

Exposure is not replacement

Another common shortcut is to treat occupational exposure as a forecast of job loss.

Exposure measures usually ask how much of an occupation overlaps with tasks AI may be able to perform or assist. That can identify where change is more plausible. It cannot, by itself, tell us whether the change will be substitution, augmentation, reorganization or some mixture.

The International Labour Organization's refined global index assesses nearly 30,000 occupational tasks. Its 2025 update estimated that one in four workers worldwide is in an occupation with some degree of generative-AI exposure. Only 3.3 percent of global employment was in the highest exposure category. The ILO's conclusion is not that one in four jobs will disappear. It argues that transformation is more likely than widespread redundancy because many exposed occupations still require substantial human input.[8]

In April 2026, the ILO returned to the measurement problem directly, warning that AI exposure indicators should not be interpreted on their own as predictions of job losses or labor-market outcomes.[9]

Anthropic's March 2026 labor-market study tries to move one step closer to real impact by distinguishing theoretical exposure from "observed exposure." Its measure combines estimates of what models could theoretically do with actual Claude usage, giving more weight to automated, work-related use. Anthropic reported that actual coverage remained well below theoretical capability. It found no systematic increase in unemployment among workers in highly exposed occupations, while finding suggestive evidence of slower hiring for younger workers.[10]

That is a more economically meaningful measure than theoretical capability alone, but it still has limits. It is built partly from usage of one company's products and is not a census of AI activity across the economy.

The lesson is the same: each measurement answers a narrower question than the headline debate often asks of it.

Adoption is a separate economic event

Even if a technology is useful, firms still have to adopt it.

The U.S. Census Bureau's 2026 Business Trends and Outlook Survey gives us a nationally representative view of that step. During the November 2025 to January 2026 supplement period, about 18 percent of firms reported using AI in a business function. Because larger firms were more likely to use AI, the employment-weighted rate was about 32 percent. Adoption was especially high in large firms in information, professional services and finance.[11]

Among firms that were using AI, use was still relatively narrow. Fifty-seven percent reported AI in three or fewer business functions.[11]

Later Census releases showed overall business AI use hovering roughly in the high teens through spring 2026, with higher expected use over the following six months.[12]

This is important because the capability frontier and the adoption frontier move at different speeds.

A model can become better overnight. A firm has contracts, software, managers, procurement rules, compliance obligations, customer expectations and employees whose responsibilities were designed around older tools. Even when adoption is fast by historical standards, organizational redesign has friction.

That friction can delay economic effects.

It can also concentrate them. Large knowledge-intensive firms may reorganize around AI earlier than small businesses or sectors where work is physical, interpersonal or difficult to standardize.

So when someone asks whether a capability benchmark has "shown up in the jobs data," the answer partly depends on whether the relevant capability has passed through the adoption layer yet.

Aggregate stability can coexist with concentrated disruption

Labor-market data now provide a good example of why averages are not enough.

In August 2026, researchers at the Stanford Digital Economy Lab released a revised analysis using ADP payroll data covering millions of U.S. workers through June. They reported no evidence of widespread economy-wide job displacement associated with generative AI.[13]

That is the aggregate result.

Inside it, they found a more concentrated pattern. Employment among workers aged 22 to 25 in highly AI-exposed occupations stood 19 percent below the level it would have reached had it followed the trajectory of less-exposed peers. The authors found no comparable gap among experienced workers, and they reported that the difference was driven mainly by reduced hiring of younger workers rather than increased separations.[13]

The researchers are careful about causality. They describe the findings as early descriptive indicators, not causal estimates. Some divergent trends predate generative AI, the effects weaken under some controls, and the ADP sample is not identical to the national workforce.[13]

Those qualifications are not footnotes to be discarded. They define what the evidence can support.

But the result also shows why "no economy-wide displacement" cannot be translated into "no labor-market effect."

A technology can affect entry-level hiring before it produces large layoffs. It can alter one occupation group before another. It can change the composition of teams without reducing total employment. It can slow the creation of positions that, in an earlier technology regime, would have appeared.

Aggregate stability and concentrated disruption are compatible.

Productivity does not mechanically determine employment

Suppose AI makes a task twice as fast. What happens to employment?

There is no universal answer.

If a firm needs the same amount of output, higher productivity may reduce the labor required for that task. If lower costs increase demand enough, the firm may produce much more and employ the same number of people or more. AI may automate some tasks while making complementary human skills more valuable. It may remove routine work while creating new coordination, verification, sales or service tasks. It may change which level of experience is valuable. It may also alter the boundary of the firm itself, changing whether work is done by employees, contractors or software.

Daron Acemoglu's task-based macroeconomic framework is useful here because it forces the analysis to separate the share of tasks affected from the productivity improvement on those tasks, while also accounting for complementarity and the creation of new tasks. Under the assumptions and evidence used in his published analysis, the implied aggregate productivity gains were meaningful but much smaller than some widely cited forecasts.[14]

The exact macroeconomic estimate is not the most important point for this article.

The important point is structural:

Productivity is not an employment coefficient.

A productivity result enters an economic system. The employment outcome depends on substitution, complementarities, demand, prices, wages, new tasks and organizational responses.

That is why a benchmark does not contain a labor-market forecast inside it.

What capability evaluations can actually tell us

After all these qualifications, it would be easy to overcorrect and conclude that capability evaluations tell us little about work.

That would also be wrong.

They tell us where the technical constraint is moving.

If systems can reliably complete only short, tightly specified tasks, many forms of delegation remain impractical. As systems become capable of handling longer sequences, recovering from intermediate failures and operating tools, the set of workflows that can plausibly be reorganized expands.

Capability evaluations can therefore help identify pressure building upstream of economic change.

They are especially informative when combined with other evidence:

• Capability evaluations tell us what systems can do under defined conditions.
• Reliability measures tell us whether that capability survives a more demanding success threshold.
• Workplace experiments test whether people or teams perform differently with the tools.
• Usage data show what people are actually delegating.
• Firm surveys show whether organizations are adopting AI across business functions.
• Labor-market data show whether hiring, employment, wages or hours are changing.

The strongest inference comes from the pattern across these layers, not from any one metric.

As of October 2026, that pattern is neither "nothing is happening" nor "benchmarks have already proved mass replacement."

A more defensible summary is:

AI systems have become capable of substantially longer and more complex tasks in some domains. Real workplace productivity effects can be large, small or negative depending on the task and worker. Firms are adopting AI unevenly but at meaningful scale. Economy-wide displacement is not evident in the strongest current administrative analysis, while concentrated pressure is visible in some highly exposed groups, especially younger workers entering certain occupations.

That is a less dramatic sentence.

It is also more useful.

Ask the more useful question

"Will AI take our jobs?" compresses too much.

A better sequence is:

What can the system do?

How reliably can it do it?

Can that capability be integrated into real work?

Does it improve productivity in that context?

Do organizations adopt it and redesign work around it?

Does it substitute for labor, complement labor, create new tasks, expand demand, or do several of these at once?

Only then do we reach employment.

This distinction also explains why the economically important threshold may arrive before complete job automation. A company does not need an AI system to perform every task in an occupation before it changes hiring, team structure or the mix of work assigned to people. Partial capability can matter if it removes enough bottlenecks, raises the capacity of existing workers, or changes which tasks justify a new position.

That is the boundary Inside 2027 will keep watching.

The novel 2027: A World Without People explores the human consequences of a world in which such decisions accumulate. This essay has a narrower job: to separate what the evidence currently shows from what any future story, forecast or scenario might infer from it.

The distinction is not a technicality.

It is the difference between measuring a capability and understanding an economy.