Which of Your Work Is Actually Verifiable?

I spent part of an internal conference recently listening to a talk about the future of my own job. One of the talks pointed at a public lecture by the sociologist Zeynep Tufekci, and that pointer sent me to the full ninety-one minutes [1]. I went in expecting an argument about whether AI takes the Solutions Architect job. I came out with a sharper and more uncomfortable question, which is the one I want to hand you.

The wrong benchmark

Tufekci opens with the arrival of the car. Early observers measured it against the thing it seemed to replace. “Is it faster than a horse?” [1] The first car magazines were literally named Horseless. Smart people, thinking hard, fixed on the one benchmark they already had, and so they missed traffic, suburbs, cities designed for cars instead of people, and everything downstream of that. As she puts it, the speed was the wrong benchmark.

She turns the same lens on AI. Asking whether a model reaches some general-intelligence bar is, in her words, “a completely irrelevant, misguided, misleading benchmark” [1]. Notice what she does there. She does not answer the question “will AI take the jobs?” She rejects it as the wrong question, the horse-speed question, and that move is the useful part for anyone trying to reason about their own career.

The line already exists, and two people drew it well

If you replace the jobs question with a better one, you land on a distinction other people have already made carefully, and I want to credit them before I add anything, because the mechanism underneath this post is theirs, not mine.

Jason Wei wrote it down as the verifier’s rule: “All tasks that are possible to solve and easy to verify will be solved by AI” [2]. His reasoning runs through reinforcement learning specifically: anything you can reliably check is something you can turn into a training signal, so a checkable task is one you can build a training environment around. He lists five properties that make a task easy to verify: an objective notion of a good answer, fast checking, checking many answers at once, low noise between the check and real quality, and a continuous signal you can rank on [2]. Five, not “a few.” Meet those and the task is, on his account, on its way to being automated.

Christian Catalini at MIT Sloan arrives at the same structure from economics. In a working paper with Xiang Hui and Jane Wu, he models the transition as a collision of two cost curves: the plunging cost of getting a machine to produce something, and the stubborn, biologically fixed cost of a human verifying it. The widening space between them is what he names the Measurability Gap, and the binding constraint on the whole economy stops being intelligence and becomes human verification bandwidth [3]. His summary of where that leaves us: “We are moving from a world where we were valued for what we could create, to one where we are valued for our ability to steer and stand behind what is created” [4]. He calls the healthy organizational shape a sandwich: human intent on top, machine execution in the middle, human underwriting at the bottom [4].

Two different fields, no evidence they read each other, same shape. Cheap-to-verify work migrates to the machine. Human value concentrates in what stays expensive to check. If you have felt that shift in your own week, this is the theory of it.

What neither of them dwells on

Here is the crack I want to widen, and it is the only genuinely new thing I am claiming.

Both accounts are about the cost of verification once you already know what you are dealing with. Wei’s five properties are a checklist you apply to a task; Catalini’s two curves need a task already placed on the cheap or the expensive axis. Neither of them is careless about this, and neither claims the placing is trivial. They are simply not writing about it. Classification error is not a variable in either model, and that is the gap I want to put something in.

Tufekci’s entire subject is that we are bad at exactly that step. The people benchmarking cars against horses were not lazy; they were confidently wrong about which properties of the car mattered. Put her observation next to Wei’s and Catalini’s and you get the thing this post is actually about: the danger is not only that AI takes the work we can verify. It is that we misjudge which of our work is verifiable, wave the “that part is easy to check” part toward the machine, and discover too late that the checking was the job.

Sort your own artifacts

So do the sort, honestly, on your own output. Here is mine as a Solutions Architect.

The left column is real work, and it is the work I am proudest of on any given day. But look at what each check actually establishes. Infrastructure code that compiles and deploys proves that it is syntactically valid and that the resources came up, and nothing at all about whether they were the right resources. A cost model with validated inputs can be checked for arithmetic consistency, and against a real bill only later, and against the forecast it was really making almost never. A Well-Architected review establishes conformance to written criteria. A reference architecture matched against a documented pattern establishes resemblance, not suitability. Those narrow properties genuinely are cheap to check, they score well on Wei’s five, and that is exactly why assistants are already good at them. The trap is the slide from “the check passed” to “the artifact is right,” because only the first of those is what got verified.

The right column is the uncomfortable one. Is this design right for where the organization actually wants to be in five years? Is the constraint the customer stated the real constraint, or the one that was safe to say in the room? Will this particular team, with its actual staffing and its actual on-call culture, operate this thing, or quietly route around it? Is the trade-off acceptable given internal politics that nobody will write down? Run these against Wei’s checklist and they score badly, though not uniformly and not completely. Some have partial signals: you can load-test an operability worry, and you can rank two designs against a strategy someone has written down. What they lack is an answer everyone accepts, a fast one, and one that transfers to the next organization. Verifiability here is a gradient rather than a wall, and the gradient has been steep enough that the work has stayed with people.

The estimate depends on a line somebody has to draw

The reason I trust this framing is that the same lecture lands on a case where the estimate turns entirely on a distinction somebody has to make by hand.

Tufekci reaches for a fit she considers reasonable for language models: routine legal drafting. Wills, estate planning, some kinds of insurance are, she says, “fairly prescribed” and “fairly standard documents”; law is “very formalized”, so you can “build some harnesses around it like some actual deterministic code” and let a trained specialist handle the final step [1]. She is careful about it, and in the same breath she names the condition the whole design rests on: the output “can’t 100% be true unless you check somehow” [1].

Take her at her word, because I think she is right about the documents she is describing. What the same profession also reports is how quickly that check stops being cheap as you move along the spectrum. At the complex end, by a practitioner account published as sponsored content by a legal-software vendor, lawyers who cannot tell which parts the model got right end up reading every line, and the hours saved on drafting come back as hours spent hunting for hallucinated clauses [5]. Discount that source as you like; the weight-bearing admission is hers, and she has already made it. Somebody has to check. Same domain, same formality, and the cost of checking swings from delegable to punitive depending on which document landed on the desk.

So the classification is not a property you can read off the artifact. It rests on a routine-versus-complex judgement that someone has to make, document by document, before any of the automation reasoning applies at all. Law has centuries of practice at making that particular call. I have a landing zone, a customer who described their constraints in one meeting, and my own estimate of which parts are “standard”. That estimate is the weakest link in the chain, and it is the one nobody audits.

The part that stays hard

There is a floor under all of this. Wei notes that some tasks are actually harder to verify than to produce: it can take longer to fact-check an essay than to write it, and a novel scientific hypothesis is cheap to state and expensive to confirm [2]. Much of the highest-value architecture work lives here. Proposing a direction is a morning. Being sure it was the right one can take a year of the system running in production. That year is not thinking time and it is not compute; it is waiting for reality to produce the evidence. Model progress can shrink a great deal of what surrounds that wait, with better simulation, better proxy signals, faster interim evaluation. What it cannot do is deliver an outcome that only the future is in a position to supply.

The move that compounds

None of this is a reason to guard the right column like a moat. Wei points at the opposite, and it is the most useful thing an SA can take from any of this. Verification asymmetry can be improved. You front-load the work that turns an expensive check into a cheap one: an answer key for a math problem, test cases with good coverage for code. “Indeed, this is what Leetcode does” [2].

That is a job description hiding in plain sight. The compounding SA move is to convert expensive judgment into cheap, explicit criteria before the machine gets there. Turn “is this secure enough” into written threat models and automated policy checks. Turn “will this scale” into load tests with named thresholds. Turn “is this the right pattern” into a decision record with the trade-offs made explicit, so the next review is a lookup instead of an argument. Every time you do that, you move a slice of work from the expensive column to the cheap one, on purpose, and you make the checking that remains faster for everyone who comes after you.

There is an honest objection to this, and it comes from the same paper. Catalini and his co-authors name a failure mode they call the Codifier’s Curse, in which experts codify their obsolescence [3]. Write down everything you know and you have handed over the only thing that was hard to replace. The objection lands, and the answer is that the alternative is worse: becoming the human verifier puts you directly under the biological ceiling their own model describes, where you are the bottleneck rather than the value. Codifying is only a curse if you stop there. Writing down this year’s judgment and then going to acquire next year’s compounds. Writing it down and standing still is how you automate yourself. What separates the two is direction of travel, not the act of codifying.

It is worth being honest that this is a strategy and not a guarantee. Codifying reliably makes the organization more valuable. Whether it keeps you valuable depends on there being new scarce judgment to move on to, and on you being the one who gets to move. That has held in this craft so far, through several waves of exactly this kind of automation. I would not bet the argument on it holding forever.

The camouflage is gone

The reframe that stuck with me is almost consoling. The expensive-verification work was always the actual job. Producing the artifact, the diagram, the document, the pile of infrastructure code, was partly camouflage: it looked like the work, it filled the day, and it let us postpone the harder question of whether any of it was right for the situation in front of us. AI is stripping that camouflage away. What it exposes is not that the Solutions Architect has less to do, but that the part we can hand off was never the part that mattered most.

So the honest career risk is not automation. It is misclassification. Sort your own artifacts by how expensive they are to be sure about, distrust the ones that feel obviously easy, and then spend your effort building the verifier rather than becoming it.

Which of your artifacts have you been filing under “easy to check” without actually checking that they are?

Sources

• [1] Zeynep Tufekci — Are We Having the Wrong Nightmares About AI? (The Policy Institute, King’s College London; the horse-and-car benchmark, the AGI-benchmark critique, and the legal-drafting example, all quoted from the lecture).
• [2] Jason Wei — Asymmetry of verification and verifier’s rule (the verifier’s rule, the five criteria, improving asymmetry, and tasks harder to verify than to produce).
• [3] Catalini, Hui & Wu — Some Simple Economics of AGI (working paper: two racing cost curves, verification bandwidth as the binding constraint, measurability-biased technical change).
• [4] MIT Sloan — In tomorrow’s AI economy, what’s the role of humans? (press summary of Christian Catalini’s work: the Measurability Gap, the sandwich topology, “from Doing to Underwriting”).
• [5] Above the Law — On The Most Complex Matters, Probably Right Is Not Good Enough (sponsored content, written by a director at a legal-software vendor; cited as an illustrative practitioner account of the verification burden, not as independent reporting).

About the Author

Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.

This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.

Learn more →

• AI
• SolutionsArchitect
• Verification
• AgenticAI
• CareerDevelopment