The Matryoshka of Humans in the Loop

I design agent systems for a living, and where a human has to sit in them comes up in almost every design review. The history and the law below are cited at the bottom. The wall-box story is a thought experiment. It did not happen.

The cabinet closes

Amazon Mechanical Turk is gone. The official help page states that “Amazon Mechanical Turk permanently closed on September 30, 2026” [1]. The same page notes that the closure “also applies to SageMaker Ground Truth and Amazon Augmented AI,” where the Mechanical Turk worker type is no longer available for labeling jobs or human review workflows [1].

The reaction on LinkedIn was a mix of nostalgia and irony. Eli Aleyner, whose first team at Amazon was Mechanical Turk, wrote about the shutdown in a post that Micah Walter reshared, and he compressed twenty years into two sentences [2]:

“The original Turk hid a person inside a machine. Ours ended up hiding a machine inside the person.”

That line made me think of a matryoshka, the Russian nesting doll. Open one and there’s another inside. I’ll use it like this: each doll is an interface that hides who or what does the work behind it, and the dolls mostly alternate between people and machines.

Take one small case of that newest layer. A car’s assistant, tasked with cutting its owner’s commute and charging costs, decides a home wall box would pay for itself. It finds an electrician through a home-services marketplace, one who is listed in the grid operator’s installer directory. The owner taps confirm on the quote and the agent pays. One morning the electrician rings the doorbell of the house the owner rents, the landlord turns up, and nobody has asked the landlord. Every API call in that chain succeeded.

There are three ways to look at the dolls: from history, from the engineering present, and from the governance question of where the nesting has to stop.

The history of hiding people

A person inside a machine

The service took its name from Wolfgang von Kempelen’s chess automaton. In the official telling, von Kempelen built it in 1769, it toured Europe, and to persuade skeptical audiences he slid open the cabinet doors to show the gears. “What they did not know was the secret behind the Mechanical Turk: a chess master cleverly concealed inside” [1]. Histories differ on which year you’ll see quoted (built in 1769, exhibited from 1770), which suits a machine built on misdirection.

Humans as a callable resource

Long before 2005, people were already doing the part computers couldn’t, with software wrapped around them. NASA’s Clickworkers project had public volunteers mark craters on Mars images, and its results were written up in 2001 [3]. Google Answers let you post a question and say how much you would pay, and a paid researcher went and found the answer. “You will only be charged for your question if and when an answer is posted to it” [4]. In 2004, Luis von Ahn and Laura Dabbish turned image labeling into the ESP Game, where players labeled web images as a side effect of playing [5]. Each one wrapped the human work in an incentive: a game, a bounty, a volunteer project.

Artificial artificial intelligence

Mechanical Turk, in 2005, made this general. Its “About” page describes the reversal itself. Normally the human asks and the computer answers, so: “What if this process was reversed and a computer program could ask a human being to perform a task and return the results?” [1] Requesters could “integrate Artificial Artificial Intelligence directly into their applications,” and the transaction “looks very much like any remote procedure call” [1].

That function call fed one of the most influential datasets in machine learning. The ImageNet paper describes “the data collection scheme with Amazon Mechanical Turk” [6], and the ImageNet challenge, run every year from 2010, became a central benchmark for computer vision [22].

Then the loop became a product. Amazon SageMaker Ground Truth, launched at re:Invent 2018, turned dataset labeling into a managed service [7]. Amazon Augmented AI (A2I) did the same for review. When a configured condition was met (a low-confidence prediction, for example) or the application asked for one, a human loop sent the item to a work team drawn from the Mechanical Turk workforce, a vendor, or the customer’s own private workforce [8]. The human was now a configurable component.

A machine inside the person

In 2023, Veselovsky, Horta Ribeiro and West re-ran an abstract-summarization task on Mechanical Turk and estimated that 33 to 46% of the workers had used a large language model to do it [9]. That’s one text task, and the authors say it’s unclear how far it generalizes to tasks that suit an LLM less well [9]. Still, the doll had turned inside out. Requesters paid for human judgment and sometimes got a model’s output passed through a human.

The engineering present

In 2005 an application called a human to do a task. In 2026 an agent calls one to fill in what it doesn’t know, or to confirm or approve an action, and MCP, one of the protocols agents speak, now has a standard way to make that call.

The Model Context Protocol (MCP) defines elicitation as “a standardized way for servers to request additional information from users through the client during interactions” [10]. A tool can stop mid-call and ask the person for input or confirmation, or send them to an external URL to authenticate. The spec forbids collecting passwords or payment credentials through the form itself [10]. On Amazon Bedrock Agents, you can enable user confirmation for an action, so that “the user is presented with an option to CONFIRM or DENY the action” before the action-group function is invoked [11]. AgentCore Gateway forwards MCP elicitation requests “when a tool needs user confirmation, authentication, or additional input to proceed” [12]. The call no longer needs a user to start the conversation, either. AWS’s reference design for ambient agents on AgentCore turns an S3 upload or a schedule into a job, lets the agent pause through a single ask_human tool, and by default holds each new job until a person runs it, but leaves the choice of who answers to the application built on it [24].

Payments are moving the same way. With x402, a server answers a request that arrives without payment with HTTP 402 Payment Required, and the client pays and retries [19]. No person needs to be in that exchange, so the question of who authorized it moves elsewhere.

Where an agent is about to commit something (money, a booking, a change to someone’s account), designers reach for mechanisms such as a checkpoint that asks a person at that moment, or a policy limit that encodes what someone authorized in advance. Confirmation and elicitation are product features in the systems cited here. Multi-party sign-off, role-based permissions and after-the-fact review are broader patterns that usually need their own identity, workflow and audit pieces, and none of these mechanisms says on its own whose authority the action needs, or whether the right party set a configured limit.

Where the nesting stops

The recursion has to bottom out at whoever holds authority over the action, and that is often more than one party.

Could it be machines all the way down?

The obvious candidate for the next doll is more machines. The Agent2Agent (A2A) protocol is an open standard for agents to discover each other and delegate tasks to one another [14], and I’ve written about collectives of agents that start to behave like organizations [21]. The plumbing for agents tasking agents tasking agents now exists, although every hop adds inference, state and failure handling.

Whether such a stack can run on its own is less clear. Shumailov and colleagues showed in Nature that models trained indiscriminately on model-generated data develop irreversible defects, with the tails of the original distribution disappearing [15]. That result is about training: it applies when generated output re-enters training data without enough access to the original distribution. The paper does not show that delegation between agents degrades anything at runtime. So my own design rule is that agents checking each other need evidence from outside the stack, whether from tools, tests, authoritative data, the physical world or people.

That’s also where the doll metaphor strains. A real matryoshka nests the same kind of doll, each smaller than the last. Here the kinds mostly alternate between human and machine, and the layers do not shrink; they scale up.

Back to the doorbell

So which human is the innermost doll? Back to the wall box. I’m using German rules because they make the layers unusually explicit. Details differ by country, and this isn’t legal advice.

Under BGB §554, a tenant can demand that the landlord permit structural changes that serve EV charging. That is a right to ask for permission, not to skip it, and the landlord can refuse where the change cannot reasonably be expected of them, with the tenant’s interests weighed too [16].

On the grid side, NAV §13 allows work on the installation behind the house fuse only by the grid operator or by an installer registered in a grid operator’s installer directory [18], which the agent got right when it picked an installer from that directory. The same section makes the connection owner responsible to the grid operator for the installation, and that responsibility stays where it is if the property is rented out [18]. The connection owner isn’t necessarily the person who ordered the work. NAV §19 then requires charging equipment to be notified to the grid operator before it is commissioned, and above 12 kVA total rated power the operator’s prior consent is needed [17].

The workflow, as described, recorded the tenant’s confirmation but modeled neither the landlord’s authorization nor any evidence that the grid notification (and, above 12 kVA, the operator’s consent) had been completed. Nothing in the stack represented those parties, so no confirmation dialog could reach them. A different design could require both records before the booking goes through, with an authorization boundary for each of those parties. The same shape turns up elsewhere, for example in a procurement agent that books an on-site technician at a client’s building, where the company is only a tenant and the building operator has its own contractor rules.

Every earlier doll hid a worker inside an interface. Here the electrician, who did the work, is not the innermost doll, and the landlord sits outside the stack entirely. As built, the nesting stopped at the tenant’s tap, when it should have reached the party whose authority the work needed.

What is the next doll?

The loop Mechanical Turk made famous now lives in protocols, approval steps and the payment layer. More machine layers will come, because the protocols for delegation between agents exist, and technically it can go on. Operationally it stops in two places. Claims need evidence from outside the stack, which tools, tests and data can supply. Consequential actions need an authority boundary, and that necessarily comes down to a person or organization entitled to authorize the action.

The next doll, and the one we have to design on purpose, is not a worker and not a reviewer clicking through a queue. It is the authority holder: the account owner, the landlord, the company that owns the building, the approver of record. When the holder is an organization, the human in the loop is whoever is entitled to act for it. The authority may be exercised in a live approval or delegated in advance as a mandate or policy limit, and the system has to know whose it is. So every agent that can spend money or change the physical world needs an explicit answer to whose permission the action actually needs, and to whether the system asks that holder or holds a mandate from them. The answer is rarely more confirmation dialogs. It is a better-placed question.

What is the innermost doll in the agents you are building, and is it the right one?

If you’re running this on AWS

The primitives for putting the right human in the loop exist today, so the remaining work is deciding where each one goes.

• In Amazon Bedrock Agents, enable user confirmation on actions that commit something. The agent returns control with the proposed action, your application sends the user’s CONFIRM or DENY back as confirmationState in the function result, and the action-group function is invoked only after CONFIRM [11].
• For a question in the middle of a tool call, AgentCore Gateway forwards MCP elicitation requests from MCP server targets, including form mode for confirmations and URL mode for authentication [12]. Use it when a tool needs input it shouldn’t guess.
• AGENTSEC04-BP02 in the Well-Architected Agentic AI Lens warns that routing every action through review “produces rubber-stamp approvals” and recommends deterministic risk classification, approval tiers, timeouts with escalation, and logging of “reviewer identities” [13]. Its suggested mechanisms include Step Functions callbacks, Bedrock Agents user confirmation and AgentCore Runtime async tasks.
• Policy in AgentCore records the actions in a session, and its temporal policies (session-aware rules in the Dogwood policy language, alongside stateless Cedar policies) evaluate that history. A sensitive action can be permitted only after an approval step earlier in the same session, or forbidden once its summed value within a time window reaches a cap. The documentation uses a funds transfer for both examples [20]. Policy evaluates tool calls that pass through AgentCore Gateway before it allows them [23], so if a payment is exposed to the agent as a gateway tool, the policy can permit or deny each call on earlier approvals or configured limits. That lets you block the call, but it can’t show that the approver had authority over the payment, and the payment itself still runs in the tool behind it.
• In the ambient-agent reference for AgentCore Runtime, autoExecute is off by default, so a signal-triggered job waits until a person runs it. Mid-run, the agent asks through ask_human, including Review turns answered with APPROVE, REJECT or MODIFY [24]. The question goes to the job’s owner. If that isn’t the authority holder, you route it yourself.
• None of these services can do the mapping for you. Map each high-risk action to the party that holds authority over it, and to someone entitled to act for that party. That may not be the user typing to the agent, however convenient the nearest user is. And when the system can’t tell who the holder is, the safe default is to stop and escalate.

Mechanical Turk’s reversal, “a computer program could ask a human being to perform a task” [1], never had to say which human.

Sources

• [1] Amazon Mechanical Turk FAQs (closure FAQ and “About Amazon Mechanical Turk”) · closure on 30 Sep 2026, effect on Ground Truth and A2I, von Kempelen’s Turk, “Artificial Artificial Intelligence.”

• [2] Micah Walter on LinkedIn, resharing Eli Aleyner’s post on the Mechanical Turk shutdown · the reshare includes Eli Aleyner’s original post and the quoted lines.

• [3] NASA Ames Research Center, “Results of Click Workers Mars Crater Marking and Crater Classification,” NASA NTRS (2001) · crater-marking results; the archived project site calls it “NASA Ames’s experiment in volunteer science.”

• [4] Google Answers FAQ (archived) · priced questions answered by paid Researchers.

• [5] von Ahn and Dabbish, “Labeling images with a computer game,” CHI 2004 · the ESP Game.

• [6] Deng et al., “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR 2009 · data collection with Amazon Mechanical Turk.

• [7] AWS News Blog: Amazon SageMaker Ground Truth keeps simplifying labeling workflows · launched at AWS re:Invent 2018.

• [8] Amazon Augmented AI: core components · human review loops started by configured conditions or by the application.

• [9] Veselovsky, Horta Ribeiro and West, “Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks,” arXiv:2306.07899 (2023) · 33 to 46% estimate on one summarization task.

• [10] Model Context Protocol specification: Elicitation · servers requesting information from users mid-interaction.

• [11] Amazon Bedrock: Get user confirmation before invoking action group function · CONFIRM / DENY and confirmationState.

• [12] Amazon Bedrock AgentCore: Use elicitation with your AgentCore gateway · elicitation modes and flow.

• [13] AWS Well-Architected Agentic AI Lens: AGENTSEC04-BP02 Human-in-the-loop for critical decisions · risk-tiered approval, timeouts, audit logging.

• [14] Agent2Agent (A2A) Protocol specification · agent-to-agent discovery and task delegation.

• [15] Shumailov et al., “AI models collapse when trained on recursively generated data,” Nature 631 (2024) · model collapse.

• [16] BGB §554, Barrierereduzierung, E-Mobilität, Einbruchsschutz und Steckersolargeräte · tenant’s right to demand permission for EV-charging changes.

• [17] NAV §19, Betrieb von Anlagen und Verbrauchsgeräten · notify before commissioning; consent above 12 kVA.

• [18] NAV §13, Elektrische Anlage · registered installers; connection owner’s responsibility.

• [19] x402: an open standard for internet-native payments · HTTP 402 Payment Required for programmatic payment.

• [20] Amazon Bedrock AgentCore: authoring temporal policies · session-aware temporal policies for approval-gated and capped actions.

• [21] When the Model Becomes an Org · my earlier post on agent collectives that self-organize.

• [22] Russakovsky et al., “ImageNet Large Scale Visual Recognition Challenge,” arXiv:1409.0575 · the ImageNet benchmark, run annually from 2010.

• [23] Policy in Amazon Bedrock AgentCore: Control Agent Interactions · Policy intercepts agent traffic through AgentCore Gateway and evaluates each request before allowing tool access.

• [24] Building ambient agents with Amazon Bedrock AgentCore: From event-driven signals to human-in-the-loop workflows · event-triggered jobs, review-first autoExecute default, the ask_human interrupt.