Below the Ice — The Four Numbers That Tell You If Your AI Is Actually Working

This is the print twin of tonight's Below the Ice — our evening deep-dive, one topic told properly. Prefer it in your ears while you wind down? Listen to today's episode.

Most teams deploying AI right now are flying blind on whether it's working. They can tell you the monthly model bill. They struggle to say how many tasks it actually completed, what it cost per outcome, or whether users trusted it enough to keep coming back. Sarah Friar, OpenAI's CFO, put a practical framework in front of that problem this week. It is four numbers. It sounds deceptively simple — and the devil is entirely in how you define and measure each one. Tonight we go below the scorecard: what these metrics actually capture, where they get fuzzy, and why the honest version of the "return on compute" conversation is more uncomfortable than most roadmaps are ready for.

What it is

The scorecard Friar describes has four metrics:

1. Useful work — the fraction of tasks where the AI produced a genuinely successful outcome, not just any response.
2. Cost per successful task — total spend divided by useful outcomes, not by tokens or API calls.
3. Dependability — reliability over time: does it work Monday the same way it worked Thursday, at scale, not just in the demo?
4. Return on compute — the business value unlocked per dollar of inference spent.

These are not four arbitrary KPIs. They form a chain. Useful work tells you if the system works. Cost per successful task tells you how expensively it works. Dependability tells you whether last week's measurement still applies today. Return on compute is the business question — was it worth it? Each one is deceptively hard to track in practice, and most teams are currently only tracking the bill.

How it actually works

Think of a vending machine. The vendor knows: how many times someone pressed a button, how often the item fell out, the refund rate, and the revenue per machine per day. Now imagine you paid the vendor for button presses, not for items delivered. That is how most teams currently pay for AI: tokens, API calls, model invocations. You pay for the button press regardless of whether anything came out.

Friar's scorecard reframes measurement around the item, not the button press.

Useful work requires you to define, ahead of time, what "success" looks like for a task. This sounds obvious until you try it. A support chatbot that routes queries correctly 94% of the time but resolves them without human escalation only 67% of the time has two very different "useful work" rates depending on which definition you pick — and teams routinely pick the flattering one. The most common mistake is measuring completion (the model gave a response) rather than success (the response was right and something good happened because of it).

Cost per successful task forces you to divide total cost — inference, prompting, retries, human review, the engineering time that keeps the thing running — by the number of outcomes that were actually useful. When you include the fully-loaded cost, the number often surprises teams. This doesn't mean the AI isn't worth it; it means you need the true denominator to make that call honestly.

Dependability is the metric most ignored during pilots. A model is non-deterministic by nature, so dependability isn't about getting the identical answer twice — it's about staying in the same quality band consistently, at scale, without the cliff edges where it quietly collapses on edge cases. Prompt changes, model updates, retrieval pipeline changes — all of these can drift your useful-work rate without any obvious alarm. The practical minimum is a regression harness that runs on every change and tells you if the number moved.

Return on compute is the honest business question, and it is where things get uncomfortable. Value created divided by compute spent. "Value" is the part that's hard to define and harder to attribute. A legal review tool that cuts research time from 12 hours to 40 minutes has a plausible value per task — you can estimate an hourly rate and do the arithmetic. A brainstorming copilot that "helps the team think" is harder to pin down. Most ROI statements in enterprise AI right now are closer to the brainstorming copilot than the legal tool, and teams that cannot make the return-on-compute math work on paper should think carefully before scaling the deployment.

Why it matters now

This scorecard lands at an inflection point. The first wave of enterprise AI deployments was largely about demonstrating motion — getting something in front of users, showing the board the company was moving, running a pilot. That wave is over for most mature organizations. The Cars24 case study published the same week captures what the second wave looks like: more than a million conversation minutes a month, 12% lead recovery from conversations that would otherwise have gone cold, agentic workflows wired into operations teams across the company. That is not a pilot. That is a system where getting the measurement wrong has real consequences — financial, operational, reputational.

When a million conversations per month flow through AI and 12% of the sales recovery pipeline depends on it, you need to know your dependability number. A drift in useful-work rate that a pilot team would have called "interesting variance" is now a business incident. The consistent finding across annual AI adoption research is that the organizations extracting the most value from AI are distinguished primarily by measurement rigor — not by model choice or absolute spend. The scorecard Friar is describing is the operationalization of that rigor.

For builders, the practical ask is narrower than it sounds. Before the next AI feature ships: define what "useful work" means in writing, build a way to measure it at runtime, and put the total cost in the denominator when you run the return-on-compute calculation. Three steps. Most teams are doing zero of them.

What is overhyped

The framing of "return on compute" can create a false sense of precision where none actually exists.

For a legal research tool or a code review agent, the chain from model output to business value is short and attributable — you can do the arithmetic. For a strategic assistant, a brainstorming tool, a writing aid, or anything where the AI's contribution is one input among many into a human decision that produces an outcome downstream, the attribution problem is severe. Correlation — "teams using the AI had better results" — is easy to generate and almost always misleading. The companies that will overclaim on ROI are the ones that draw the attribution line at the convenient point rather than the accurate one.

The second thing to watch: "dependability" in most vendor conversations means uptime — the API was available. That is not the right definition. You want dependability of outcome quality, which requires test sets, ongoing evaluation, and real alerting on quality drift. None of those live in an SLA. Demand the right definition before you trust the number.

What to watch

Three concrete things as this framework takes hold.

1. Eval harnesses moving from differentiator to table stakes. Today, having a regression test suite that tracks your model's useful-work rate across prompt and model changes puts a team in the top tier. In 18 months it will be the floor — the thing a customer, a regulator, or a procurement team asks for before signing. Watch for frameworks that make outcome-tracking the default path, not the advanced option.

2. Outcome-based pricing from model providers. The industry currently bills on tokens. Friar's scorecard implicitly pushes toward a world where cost is measured per successful task, not per API call. Some providers are already experimenting with this framing. If it becomes a commercial norm, the incentive structure changes materially: providers become financially accountable for whether their model completes tasks, not just whether it responds.

3. The ROI conversation shifting from marketing to finance. A CFO publishing this framework is a signal. The first wave of AI ROI claims lived in marketing decks. The second wave will have to survive a finance team with a spreadsheet and a reasonable question about total cost. Watch for the case studies that do the full math — total cost in the denominator, realistic attribution in the numerator — not just the highlight reel.

---

That's tonight's Below the Ice. The full episode — same topic, slower and out loud — is up now: listen to today's episode. More deep-dives at penguinalley.com.

Sources: A Scorecard for the AI Age — Sarah Friar, OpenAI · How Cars24 scales conversations and builds faster with OpenAI · The State of AI — McKinsey Global Institute