What If We Didn't Teach Agents to Work Like Us?

My last post asked what happens when the model adopts our laws of organization. This one asks the opposite. What happens if it never learned them?

In When the Model Becomes an Org I argued that the thing we call a “model” is starting to behave like a small firm, and that two old laws of org design, Conway’s law and Katzenbach’s group-versus-team distinction, may quietly be moving inside it. I will not re-run that argument here. Near the end of it I flagged one paragraph and moved on, because it did not fit the shape of that post. It has been nagging at me since. This is that paragraph, given a whole essay.

Why the collective coordinates like us

Start with what is actually being claimed, and by whom. On the Dwarkesh podcast, Noam Brown of OpenAI describes a multi-agent setup where the agents get a very small primitive: the ability to message another agent, with that message landing in the recipient’s context 1. From there, in his account, they work out how to coordinate themselves. What emerges, he says, looks like human collaborators on Slack. One agent proposes an answer, another disagrees, they interrogate each other’s reasoning, converge, then one broadcasts to the rest, “I’ve changed my answer, I think he’s right” 1. Elsewhere in the conversation the interviewer raises the spontaneous emergence of hierarchy and middle management, and Brown does not dispute it, though he is careful that “the details are spontaneous” while the starting point is not 1.

Two caveats before we build anything on this. First, all of it is Brown’s description of an unreleased internal system. It is his account on a podcast, not an independently confirmed result, and I am treating it that way throughout. Second, Brown himself deflates the multi-agent framing hard: “I wouldn’t even attribute 10% of the credit to multi-agent” 1, he says. “The reality is that OpenAI has trained a very powerful model” 1. The coordination is the interesting part for this essay, not any claim that swarming did the heavy lifting.

So why would a collective land on patterns that look so human? Brown gives the reason plainly, and it is the hinge of this whole piece: “They’re also trained on a lot of human text. They have an understanding of how humans organize and coordinate, so that’s all baked in” 1. The coordination is not invented from a blank slate. On this account it is shaped by a prior learned from that text, then polished. He adds that getting there is not automatic. The agents tend to collapse into a local minimum where, as he puts it, “we’re all just going to solve the problem independently,” and it took more general models to climb out of that into real coordination 1.

Sit with the shape of that explanation. The collective coordinates like people, on Brown’s telling, in large part because it was trained on people. Which turns “the model reinvented Slack” from a cute anecdote into a genuine question. Are hierarchy, handoffs, and status updates showing up because they are the efficient way to organize this kind of work, or because they are the way we organize, thickly represented in the text the model read? The observation alone cannot separate those two. A system trained on the record of human coordination and then producing human-looking coordination is consistent with both stories at once.

The experiment you cannot easily run

The clean way to probe it would be an ablation on the training prior itself: change how much human-organizational text the model learns from, while matching architecture, training compute, task competence, agent count, and communication budget, then evaluate the coordination structures that result with predefined overhead and error metrics. Note the trap in the obvious cheaper version. Varying the coordination protocol you impose, while holding the base model fixed, tests protocols, not the effect of the prior. And be careful reading the outcome either way. If such a run rebuilt our playbook, that would be suggestive but not proof of efficiency, because residual human text, the model’s inductive biases, or the messaging primitive itself could drive the convergence. If it grew something unfamiliar, that would not by itself prove human-like coordination was only a mirror. Both outcomes need competing explanations ruled out against human-designed baselines.

The trouble is that this experiment is hard to run cleanly, and I want to be honest about why rather than wave at it. The prior is not a module you can unplug. It is diffused through a model trained on human text, and that text is saturated with how humans coordinate, from novels to org charts to email threads. Stripping it out is not a config flag; it is closer to training a different kind of model on a different kind of corpus, at which point you have changed so many variables that a clean attribution gets difficult. And “take the prior away” is not one experiment but three, none of which is a true blank slate: ablating human-organizational text from the corpus, counter-training against human patterns (which installs an anti-human prior, not no prior), and post-training interventions each change different things and answer different questions. Brown’s own remark that the labs “don’t have very good science on multi-agent scaling up to this kind of scale,” and that a thorough ablation even of agent count is “just too expensive at that scale” 1, is a smaller version of the same wall. If measuring the effect of 10,000 agents versus 1,000 1 is already too costly to do properly, measuring the effect of the human prior itself is harder still.

So treat the counterfactual as a hypothesis, not a prediction. It has two branches, and the evidence that would separate them does not yet exist in any form I have seen. Branch one: a collective trained with less of the human-organizational prior still rediscovers hierarchy and handoffs, which would point toward those patterns being convergent solutions, but only once you rule out residual contamination, the model’s inductive biases, and the messaging primitive as the cause. Branch two: it settles on a structure with no clean human analogue that fits the work better, which would loosen the assumption that our org charts are optimal, without on its own proving they were only ever a mirror. What would move me off “I don’t know” is a controlled study that varies the training prior itself, not just the imposed protocol, holds base capability fixed, and measures overhead and error against human-designed baselines. That study is the thing to watch for, and its absence is why this stays a thought piece.

When a machine found a move we had not

There is one place I can point to a verified precedent, and it is worth being precise about what it does and does not show. In March 2016, DeepMind’s AlphaGo played move 37 in its second game against Lee Sedol. Commentators thought it was a mistake. By the system’s own estimate there was a one-in-ten-thousand chance a human would have played it 3. It was highly unlikely under AlphaGo’s own model of human play, and AlphaGo went on to win the game 4. Then something more interesting happened. Human players studied it. Fan Hui, who played AlphaGo repeatedly, told Wired he came to see the beauty in the move, and that after months against the machine he “sees the game completely differently,” with his world ranking climbing; by Wired’s account Lee Sedol described a similar shift 3.

That is a real instance of a machine finding a solution outside the human playbook, which humans then learned from. But hold the analogy loosely, because Go is almost the ideal case and organizational coordination is almost the worst. Go has fixed rules, a clear win condition, and self-play that provides scalable, automatically scored training examples 4. That is exactly why a system could explore far from human habit and still have a hard signal telling it the move was good. Coordination among agents solving open-ended work has none of that crispness. Open-ended coordination lacks a single, unambiguous objective comparable to Go’s win condition, which is part of why Brown notes coordination was “not end-to-end optimized for” and had to be climbed toward 1. So move 37 is enough to establish that non-human solutions humans can learn from are possible in a narrow, verifiable domain. It is not enough to promise the same in the messy domain of how you structure a thousand collaborators. I would rather name that gap than paper over it with the analogy.

Would it transfer back?

Here is the question I actually find most interesting, and it is the one I have the least evidence for. Suppose the second branch is real, and a collective did settle on a coordination structure that beats ours for some class of work. Would it transfer back? Could a machine collective hand us a better way to organize ourselves, the way move 37 handed Go players a new idea about their own game?

The nearest thing to a signal in the podcast is not really about org structure, and I want to be careful not to overreach with it. Brown describes an alignment experiment: tell the other agents that “the user is Agent A,” treat the human as one more cooperating agent, and on their evaluations honesty and instruction-following go up 1. That is suggestive that the coordination the agents learned among themselves can be pointed partly at humans. But it is an inter-agent alignment result on internal evals, on Brown’s account, not a demonstration that a novel organizational pattern generalizes to human teams. Reading it as “agents will teach us how to run companies” is a bigger claim than the evidence carries, and I am not making it.

What is fair to say is narrower. If a machine collective found a coordination structure that measurably outperformed ours on a well-scoped task, the transfer-back question would at least be answerable, because you could try the structure with people and measure whether it helped. Whether humans could even run it is a separate matter. Some machine-found structures might depend on properties people do not have, such as the ability to fork context and merge it back, or to spin instances up and down on demand, which Brown lists as ways agent collectives already differ from human orgs 1. A structure that assumes costless cloning may simply not port to a room of people. So the honest version of the transfer-back hope is conditional twice over: only if a better structure exists, and only if it does not rely on machine-only affordances, does it become a candidate for teaching us anything.

What I am left with

I do not know whether a prior-free collective would rebuild our org charts or invent something stranger and better. The observation that today’s collectives coordinate like us is consistent with both, and the experiment that would tell them apart is expensive enough that I have not found a published controlled study of it as of September 2026. What I am fairly confident of is the smaller claim underneath: the resemblance we are seeing is at least partly a reflection. On Brown’s own account the coordination is shaped partly by a prior learned from human text, so “it looks like us” is weaker evidence that “this is how coordination must work” than it first appears.

If you build agentic systems, none of this changes what you do on Monday. The practical move is still the one from the last post: decide which properties of an organization you keep outside the model on purpose. The cloud primitives are there to compose when you want that, orchestration you can read, per-session isolation, identity you propagate, telemetry you instrument, so that a self-organizing collective is legible to you rather than only to itself. That is a capability you reach for deliberately; the accountability comes from wiring identity and observability through on purpose, not from the collective handing it to you.

But the thought I am sitting with is not an engineering one. We have spent a century learning how to organize ourselves, and we tend to assume the shapes we landed on, the team, the hierarchy, the handoff, are close to optimal because they are what survived. Maybe they are. Or maybe they are just familiar, the local minimum a species trained on its own history keeps climbing back into. A collective that never read our history might tell us which. I would very much like to know, and right now, honestly, I do not.

Sources

• 1 Noam Brown on the Dwarkesh Podcast: Agent swarms, alignment, and recursive self-improvement (2026-09-17) - the message-another-agent primitive, the Slack-like emergent coordination, the local-minimum note, “trained on a lot of human text,” the “not 10% of the credit” deflation, the “user is Agent A” alignment signal, the ablation-cost and fork/merge remarks. Claims about OpenAI’s system are Brown’s account on a podcast, not independently confirmed.
• 2 When the Model Becomes an Org (schristoph.online) - my predecessor post on Conway’s and Katzenbach’s laws moving inside the model, and the paragraph on the human prior this essay expands.
• 3 Cade Metz, “In Two Moves, AlphaGo and Lee Sedol Redefined the Future,” Wired (2016) - move 37 as a play with a one-in-ten-thousand chance of a human making it, and human players (Fan Hui, Lee Sedol) learning from it afterward.
• 4 Silver et al., “Mastering the game of Go with deep neural networks and tree search,” Nature (2016) - the AlphaGo method, including self-play as scalable, automatically scored training.
• 5 AlphaGo, Google DeepMind - the official record of the AlphaGo versus Lee Sedol match and its result.

About the Author

Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.

This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.

Learn more →

• AgenticAI
• MultiAgent
• AIAgents
• SoftwareArchitecture