schristoph.online

AI · 12 min read

AI Usage Metrics Make a Good Smoke Detector and a Bad Scoreboard

Stefan Christoph
On this page
TL;DR: Measure AI adoption, but don’t rank people by it. I once put a token leaderboard on the wall at a customer hackathon. I knew tokens are a weak metric and did it anyway, because the participants needed a nudge to really use the AI tool. That works for three days and a room of teams. Make it a standing metric with careers attached and it becomes a target, and people optimise the number. Amazon’s recently published engineering memo takes a middle path. Input metrics are deeply flawed, it says, but they still correlate with outcomes, so look at the 10th and 90th percentile and ask why. Read those tails as a smoke detector: a reading means “go and look”, at team level, next to one delivery metric such as successful production changes, and a team that finds AI doesn’t help its work has produced a result worth writing down.

The Lisbon hackathon leaderboard

At a customer AI hackathon in Lisbon in May, I built a leaderboard during a coffee break [1]. It showed, close to real time, how many tokens each team was consuming on Amazon Bedrock, and we projected it on the wall.

I knew while building it that tokens are a weak metric. A team stuck in a retry loop burns tokens without getting anywhere, and I wrote that into the post at the time. I built it anyway, for a reason the post didn’t spell out: the participants were a little shy about really using the AI tool. When one team’s bar flatlined, they noticed, changed their approach and got going again [1]. Sometimes people need a nudge in the right direction, and a visible number is a cheap nudge.

That hackathon lasted three days. The leaderboard was per team, and nobody’s salary depended on it. Take away those three conditions and the same chart does something else.

Why everyone ends up counting tokens

Organisations reach for input metrics because the outcomes they care about are hard to see. Amazon’s engineering leadership said as much in a memo to its software engineers. CNBC reported that it was circulated internally in September; it was posted publicly on 27 October [2] [3]. The memo says it is best “to measure, and be measured by, long-term outcomes,” and then admits that those outcomes “can be time delayed, difficult to measure, and hard to isolate from other trends” [2].

So you count what you can count today, which means tokens, weekly active users of the tool, pull requests and reviews. They arrive daily, they fit on a dashboard, and they go up when people try the tool. The memo calls them “deeply flawed” because “they are disconnected from what matters,” and warns that they increase cost and complexity when they’re used as a target [2]. Goodhart’s Law, in other words, and I’ve covered that one before [11].

Three positions in public view

Three public examples from the past year span the range. I’m calling them positions, since each company was solving a real problem.

The first treats usage as the goal. At Meta, an employee built a leaderboard of the top 250 token users across the company, with titles such as “Token Legend”. It came down two days after it was reported on, with data being shared externally given as the reason, and Meta told Fortune the employee removed it at their own discretion [4]. That leaderboard was a side project, but it shows how naturally a usage number turns into a ranking once it exists.

Shopify treats usage as an expectation. In an April 2025 memo, Shopify’s CEO Tobi Lütke wrote that “reflexive AI usage is now a baseline expectation,” added AI usage questions to performance and peer reviews, and asked teams to show why AI couldn’t do the work before requesting headcount [13]. In an episode of Alex Heath’s Sources podcast published on 28 October 2026 [15], as reported by Search Engine Journal, he said it didn’t matter how people answered the review question. Having it there gave the company “a reputation to live up to” [5]. In September he had described the failure on the other side, AI output handed to a colleague unread, which staff call a “slop grenade” [14].

The third treats usage as a flawed but useful signal. The Amazon memo says the input metrics “correlate well with the outputs that matter” and are expected to keep doing so for some time, and that using them “with care and judgement” is appropriate [2]. Then it gives one concrete instruction:

We recommend paying special attention to the 10th percentile (finding those teams who aren’t embracing AI effectively and understanding why) and the 90th percentile (finding those teams who are heavily using AI and understanding whether they’re seeing delivery increases to match) of AI adoption. [2]

That sentence is the one I’d build on. It looks at the tails of the distribution instead of the average, and it frames both tails as questions.

The smoke detector reading

A smoke detector doesn’t tell you who started a fire, how big it is, or whether there’s a fire at all. It tells you to go and look. Most of the time you find burnt toast. You don’t hand out grades based on how often each room’s detector goes off.

Teams in the bottom tenth of an adoption measure (weekly active users across teams over a quarter, say) raise two questions: why measured use is low, and whether the team gets value anyway. The memo doesn’t fix the measure, cohort or window, so pick them yourself and say which. The answers vary. The tooling may not reach their codebase or their language. Their work may be the kind current tools don’t help much with, say a legacy system with little test coverage, or safety-critical code with a heavy review process, or work that is mostly coordination. They may never have had time to learn the tool. Or they may be worried about what the tool means for their job and keep their distance. Each needs a different fix, and none of them is “use more tokens.”

The 90th percentile asks whether delivery is rising to match the use. A team at the top of the usage chart may be shipping a lot more. It may also be stuck in retry loops, generating large changes that pile up in review, or running agents on problems that don’t matter. Marc Brooker, a vice president and distinguished engineer at AWS, described this to CNBC as a return-on-investment question. Invest “tools, time, tokens, human effort” where there is an important problem to solve, and avoid “wasted effort where we are going and spending those tokens solving problems that don’t matter” [3]. He added that it’s sometimes hard to know in advance, so some appetite for experimentation is needed [3].

Both questions need something next to the usage number. Brooker’s answer is “feature velocity”: “how many launches are we doing, how many successful changes are we making in production” [3]. That is a delivery indicator you can count weekly, at team level, and a natural companion to an adoption chart, as long as quality and stability sit next to it too.

Diagram: one hexagon labelled AI usage number (per team) splits into two lanes. Orange lane, read as a smoke detector: look at p10 and p90 with delivery, ask the team why, fix it or log a finding. Grey dashed lane, read as a scoreboard: rank people, usage turns into a target, number rises and the signal is lost.

The same number, read two ways. Only one of the routes ends with something you learned.

Why the usage number alone misleads

Two studies show the gap between input and outcome. More are in The Bottleneck Moved [12].

Faros, an engineering-analytics vendor, analysed telemetry from more than 10,000 developers on 1,255 teams in mid-2025 [8]. Comparing each team’s lowest- and highest-adoption quarters, developers on high-adoption teams merged 98% more pull requests. Their PR review time went up 91% at the same time, and Faros found no significant correlation between AI adoption and improvements at the company level [8]. It’s vendor research and correlational: higher-adoption quarters came with more merged pull requests and longer reviews, while Faros detected no significant company-level improvement linked to adoption.

DORA’s 2025 report, based on survey responses from nearly 5,000 technology professionals, found AI adoption positively related to software delivery throughput and negatively related to delivery stability, and summed it up as “AI doesn’t fix a team; it amplifies what’s already there” [7] [16]. To see which of the two is happening to a given team, you need an outcome metric next to the usage chart.

Self-reports don’t close the gap either. In METR’s early-2025 trial, 16 experienced open-source developers took 19% longer on tasks where AI was allowed, and believed afterwards that it had sped them up by 20% [9]. METR’s newer data suggests more recent models do speed developers up, and METR now calls this trial design unreliable, partly because developers who expect AI to help decline to take part or keep that work out of the study [10]. So I wouldn’t quote the 19% as today’s number. What lasts is the perception gap the trial showed: how fast people felt was a poor guide to how fast they were.

The objection that percentiles become rankings

The strongest objection to all of this came from the paddo.dev blog, a few days after the memo went public. “Watch the 10th percentile” sounds gentle, he wrote, “but if you are in the bottom 10% of AI adoption, your manager now has a number with your name next to it. That is a ranking.” [6]

He’s right that it can become one. The memo leaves the how to its readers, and it explicitly allows these metrics into thinking about careers, with care and judgement [2]. A percentile turns into a leaderboard the moment it gets attached to a person’s name in a performance conversation. I’d guard against that with four rules.

  • Look at teams, never at individuals. A team-level outlier starts a conversation with the team about their work. An individual outlier puts a person on the defensive before anyone has asked a question. Where teams are very small, group them further, because a two-person team’s number is a person’s number.
  • Never show usage without an outcome next to it. Successful production changes, launches, incidents, review time, whatever the team already trusts. High usage with flat delivery is worth a question. If usage is low and delivery is fine, maybe leave the team alone.
  • Set no usage target. The moment there’s a number to hit, people will hit it, and the signal is gone.
  • Keep the readings out of reviews. The percentile tells a leader where to spend an hour listening. It doesn’t belong in anyone’s rating.

My hackathon leaderboard broke two of those rules. It showed usage with no outcome next to it, and it ranked teams in public. It got away with that because it counted teams rather than people, it ran for three days, and its job was to get shy people started, with nobody being judged. A standing dashboard has none of those protections, so it needs the rules.

Writing down where AI doesn’t help

The line from the memo I’d most like to see copied is this one: “If your work does not benefit from current AI tooling, that is a finding, not a failing.” The memo asks engineers to document why, share it with peers and bring it to their manager, and says that explaining where AI doesn’t yet help “is demonstrating the judgement we are asking for” [2].

That turns the 10th percentile from an embarrassment into a source of information. A team that tried and found the tools don’t help with their work has learned something worth more than another month of forced usage. The platform team can learn what to fix, and other teams what to skip. A short, repeatable write-up is enough. It covers five things:

  • the type of task where AI doesn’t help (yet)
  • what the team tried: the tools, the setup, and for how long
  • the concrete step where it broke, with an example
  • the approach the team used instead
  • what would change the picture, such as missing context, access, test coverage or a tool feature

Give these write-ups a shared place, read them in the same review where you look at the percentiles, and thank the people who write them. The memo also asks leaders to have used the tools on their own production work and to be able to name a task where they helped and one where they didn’t [2]. A leader who can do that will read a “doesn’t help here” note as useful information, and will probably recognise it.

Measuring adoption without ranking people

None of this means hiding the adoption dashboard. When a team rolls out AI tools, a usage number tells you whether anyone is trying, which is worth knowing, and sometimes, as at my hackathon, a visible number is exactly the nudge people need to start. What changes is what you do when the number moves.

If you have an adoption dashboard in front of you, three things are cheap to do this month. Turn it into a team-level outlier review that looks at the bottom and the top, each as a question. Put one delivery metric next to it, such as successful production changes, and watch stability too. And open a place for “AI doesn’t help here” write-ups and treat each one as a result.

Measuring AI adoption is fine. Ranking people by it is not. Read the number the way you’d read a smoke detector, go and look, and talk to the team.

Sources

❤️ Created with the support of AI (Kiro)

Related

AINativeThe Post-Agile Operating Model: How AI Changes How Teams ShipAIThe Bottleneck Moved: What 10 Studies Say About AI Developer ProductivityAIThe AI Investment Paradox — A 1962 Book Explains Why Billions Don't (Yet) DeliverAIYour Agent Needs to Remember What Passed

AIMetricsAdoptionEngineeringLeadershipDeveloperProductivity