
Evaluation · 15 min read
The 6x Math Jump Looks Like a Loop Fix
On this page
This autumn I wrote two posts about reading eval results before trusting them, one on benchmarks that run for days [7] and one on evaluators that need a reference answer [8]. Both came down to the same habit: before you trust a number, find out what produced it. A result published on 2 November is a good test of that habit, because the explanation sits in the same table as the headline.
What the MIT Team Reported
Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab of MIT CSAIL published “Sparks of Weight-Space Programming” as a blog post with eight experiments, in which OpenAI’s GPT-6 Astra reads or edits the weights of other networks [1] [5]. The headline is Experiment 8. Astra got a Python session over the weights of Qwen3.5-0.8B, practice problems from the GSM8K and MATH training sets, and a budget of 10, 25 or 50 forward passes. Gradients and training were not allowed [1]. The evaluation ran in thinking mode with up to 32,768 tokens per answer, on test sets Astra never saw [1]. The final edit changed 131,072 of the model’s 0.8 billion weights, which is 0.016% [1].
| Qwen3.5-0.8B | MATH-500 | GSM8K | IFEval | Hit token limit |
|---|---|---|---|---|
| Base | 7.9% | 13.8% | 21.0% | 93.0% |
| Edited, 10 passes | 12.3% | 25.7% | 49.0% | 87.2% |
| Edited, 25 passes | 14.1% | 25.1% | 39.9% | 86.4% |
| Edited, 50 passes | 48.1% | 67.1% | 62.3% | 33.1% |
Experiment 8 results from the MIT post [1]. Each edited row is a separate run with a different edit [5].
Read only the first three columns and you see a small model that got six times better at math and three times better at following instructions. Those are the numbers that made the headlines.
The Token-Limit Column Tracks the Jump
The last column is the share of MATH-500 answers that ran to the 32,768-token limit without finishing. For the base model it is 93.0%. After the 50-pass edit it is 33.1% [1]. Turned around, the share of answers that finished went from about 7% to about 67%. The two columns are separate measurements and overlap a little (in the base row they add up to 100.9%), but the direction is clear.
So what happened? The write-up says it plainly. The base model loops: it repeats the same text and rarely reaches an answer, and 85 of its first 96 practice answers ran to the limit [1]. Qwen’s model card warns about exactly this. In thinking mode, the card says, the 0.8B model “is more prone to entering thinking loops compared to other Qwen3.5 models, which may prevent it from terminating generation properly” [3]. The model runs in non-thinking mode by default, and the experiment switched thinking on [5], so the baseline is the model in the mode its card warns about. For math that’s a reasonable choice, but it isn’t the default configuration. Astra traced the looping to one direction written by the MLPs in layers 15 and 19 and subtracted that direction from 64 output columns in each of the two layers [1]. Those are the 131,072 weights.
The simplest reading of the table, then: the model now finishes answers it used to abandon. My best guess is that the loop used to cut off some solutions the model could already reach, and the edit let them through. The table itself doesn’t split correctness by finished and unfinished answers, so it can’t say how many, and it can’t rule out that part of the gain comes from better reasoning as well. The aggregate numbers point strongly at the loop reading, but they can’t measure how much of the gain it accounts for. The authors lean the same way: “Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked.” [1]
The largest score rise is in the 50-pass run, which is also the only run with a large drop in token-limit hits. Gemma rarely hits its limit, and the same procedure barely moved its score.
Why Instruction Following Jumped Too
IFEval measures instruction following, and it rose from 21.0% to 62.3% [1]. If the edit had taught the model math, an IFEval gain that size would be a surprise. If the edit stopped the loops, you’d expect it, because a model that loops fails any task that needs it to stop and answer. The authors make the same point: the loops aren’t specific to math, so performance beyond math improved too [1]. When unrelated benchmarks jump together after one small change, I’d look for something general first, and the finishing rate is the obvious candidate here.
Gemma 3 270M as a Comparison
The authors also ran a comparison that bears directly on this reading. Gemma 3 270M is an instruction-tuned model about a third the size of Qwen3.5-0.8B, and it rarely loops: 8.5% of its answers reach its token limit, against 93% for Qwen [1]. Astra worked on it with budgets from 5 to 50 forward passes. MATH-500 went from 6.8% to at most 7.7%, a gain Astra itself reported as not established [1]. Reading Gemma’s failures, Astra found ordinary reasoning errors, such as wrong arithmetic and invalid algebra steps, with no single component behind them to switch off [1].
Gemma works as an imperfect negative control: a model without the failure mode, run through the same procedure. Its score didn’t move. That supports the loop explanation without settling it: Gemma is a different and smaller model, sampled with its own model card’s settings and up to 8,192 tokens [1], and a method can work on one model and not another for reasons that have nothing to do with looping, such as architecture, baseline ability or how hard the search is.
Open Questions and What Checks Out
Some parts of the result rest on very little.
One Edit Search per Budget
There was one Astra run per budget [4], and the budget curves are uneven. “One run” here means one edit search. The resulting models were each scored over several samples per problem, four on MATH-500 [1], which reduces sampling noise for each checkpoint. The post reports no confidence intervals, though, and the larger uncertainty sits in the search. MATH-500 reached 12.3% with 10 passes, 14.1% with 25 and 48.1% with 50, so it rises at every budget but jumps only at the last one. IFEval is non-monotonic: it went from 49.0% at 10 passes down to 39.9% at 25 [1]. The token-limit share barely moved until the 50-pass run (87.2% and 86.4%) [1]. To me that looks less like a smooth budget curve and more like one run that happened to find the loop direction. How often a second run would find it is unknown. The logs also leave out trial runs made with earlier prompt versions [4], so we can’t see whether other attempts were made or how they went.
No Cheaper Baseline
The post doesn’t compare the edit against other decoding settings or against a standard direction ablation found with existing interpretability tools [4]. Either might match the result or fall well short. The post doesn’t run either comparison.
Sampling Settings
The evaluation used temperature 0.6, top-p 0.95, top-k 20 and presence penalty 1.5, for the base and edited model alike [1]. The model card recommends temperature 1.0 with the same top-p, top-k and presence penalty for thinking mode on text tasks, and lists temperature 0.6 (with presence penalty 0) for vision and precise coding [3]. The card also says the loops show up “when using the recommended sampling parameters” [3]. So the eval’s settings differ from the card’s text recommendation, but the card doesn’t suggest its own settings would stop the looping. I’d treat that as a question for a re-run. It isn’t evidence that the base score was held down.
Other parts hold up well.
An Independent Evaluation of the Checkpoint
Hugging Face’s akhaliq evaluated the released edited checkpoint on all 500 MATH-500 problems, 2,000 generations in total, and got 51.0%, with 31.6% of answers hitting the token limit [5]. Both the score and the token-limit share reproduced. Keep in mind what that covers: the released checkpoint. Astra’s edit search itself wasn’t repeated.
Open Logs
For every reported run, the repository includes the prompts, the replies with their reasoning summaries, the code the model ran with its output, and the evaluation behind each number. All 1,125 model turns were audited as tool-free [2]. The Experiment 8 edit is included as an exact diff against the public base weights, with a script that applies it [2]. The trial runs with earlier prompt versions are outside that record [4]. With these files anyone can inspect the mechanism and evaluate the released checkpoint independently [4].
None of this makes the edit less real. Finding the one direction tied to a small model’s looping, removing it, and getting a measured improvement that an independent evaluation reproduced is a useful debugging technique. Sean Goedecke argues it follows a playbook a capable ML engineer would recognize, steering and direction ablation [6]. For a debugging tool, that counts in its favour. Where I’d push back is the framing. “Better at math” describes the benchmark row. “Stopped looping” is the better-supported description of what changed in the model.
Five Checks for the Next Big Jump
When a jump like this shows up in a paper, a vendor announcement or your own eval dashboard, five questions cover most of the ground, and none of them costs much to ask.
- Look at the truncation share before and after. Find the share of answers that hit the token or time limit. If it moved about as much as the score, you’re looking at a finishing-rate change first. If the report doesn’t include that column, ask for it.
- Compare the sampling settings with the model card. Put the eval’s temperature, top-p and penalties next to what the card recommends for that mode, and read the card’s warnings. A baseline run under poor settings is easy to beat.
- Count the runs per condition and look at the shape of the curve. One run per condition tells you what happened once. A curve that jumps at one budget and dips at another needs repeats before you read a trend into it. Ask about runs that weren’t reported, too.
- Look for a negative control. Run the same method on a model that doesn’t have the failure mode. If that model doesn’t move, the likeliest story is that the method fixes the failure and adds no ability. If it does move, you may have found something bigger.
- Write down what tools and feedback the method had. Here that was a Python session over the weights, practice problems from the training sets and up to 50 forward passes [1]. That tells you which simpler methods to compare it with.
The five checks applied to Experiment 8. Orange cards change how you read the headline, the blue card supports the loop reading, and the grey cards are context or still open.
Your Agent Evals Have a Token-Limit Column Too
The same checks apply to evals you run yourself, and agent evals have their own version of the token limit. An agent that times out, hits a step limit or runs out of budget is in the same position as an answer that ran to 32,768 tokens. In “When Your Benchmark Runs for 88 Hours” I suggested treating runs that time out without converging as censored instead of counting them as converged, and summing cost per solved task over the failed runs too [7]. Both need the eval to record why each run ended.
So when a new model or prompt raises your pass rate, check the timeout share alongside correctness and the other reasons runs ended. A change that stops an agent from stalling is worth shipping. It’s still a different change from one that makes the agent reason better, and it needs different follow-up.
“Wrong, Not Broken” covered the other half: an evaluator is only as good as the reference it compares against [8]. An answer that finishes still has to be checked against ground truth. A useful report shows finishing and correctness as two columns, side by side.
Running These Checks in Amazon Bedrock Evaluations
Amazon Bedrock Evaluations can run these checks on your own prompts instead of only on public benchmarks [9].
Your Own Prompt Dataset
A model evaluation job that uses a judge model reads a custom prompt dataset from Amazon S3 in JSON Lines format, with up to 1,000 prompts per job. Each line has a prompt, and optionally a referenceResponse (your ground truth) and a category [10]. Scores are reported per category, so math, instruction following and your own task types each get their own number [10].
Comparing Models and Configurations
An LLM-as-a-judge job evaluates one model, and you run several jobs on the same dataset to compare models [14]. Generators can be Amazon Bedrock foundation models, Marketplace models, customized or imported models, prompt routers, or models with Provisioned Throughput [12]. You can also bring inference responses you generated yourself. In that case Bedrock skips the model call and evaluates the responses you supply [10] [12].
A “Reached a Final Answer” Metric
Custom metrics are judge prompts you write, up to 10 per job [11]. They can use the input variables {{prompt}}, {{prediction}} (the generator’s response) and {{ground_truth}} (from referenceResponse) [11] [13]. Add a judge metric for whether the response reached a final answer, and you have a completion proxy to read next to a correctness metric:
A custom metric definition in the JSON format the console accepts, with a two-value rating scale [11]:
{
"customMetricDefinition": {
"metricName": "reached_final_answer",
"instructions": "You are checking whether a model response is complete.\nYou are given a prompt and the response a model produced for it. Decide whether the response ends with a clear final answer to the prompt. A response that stops in the middle of its reasoning, or repeats the same text without concluding, has not reached a final answer. Do not judge whether the answer is correct.\n\nRate the response as follows:\n- Final answer given: the response ends with a clear final answer.\n- No final answer: the response stops mid-reasoning or repeats itself without concluding.\n\nHere is the actual task:\nPrompt: {{prompt}}\nResponse: {{prediction}}",
"ratingScale": [
{ "definition": "Final answer given", "value": { "floatValue": 1 } },
{ "definition": "No final answer", "value": { "floatValue": 0 } }
]
}
}
The judge only sees the response text, so this metric measures apparent completion. It can’t tell whether generation stopped at the max-token limit: a response can contain a final answer and still hit the limit, or stop unfinished for another reason. The actual truncation column has to come from the generation metadata your inference harness records, such as the stop reason for each response. If you generate the responses in your own harness, record that before you upload them and keep it as a separate measurement next to the judge’s score. Writing it into the category field is one way to get scores split by finished and truncated responses in the report.
Before you plan a job, check the user guide for which judge models support custom metrics and which models and Regions support evaluation. The judge-model list for custom metrics is separate from the one for built-in metrics [9] [12].
What’s the token-limit column in your own eval reports, and who looks at it when the score jumps?
Sources
- [1] Rikiya Takehi, Jayden Teoh, Alex Zhang, Omar Khattab (MIT CSAIL), “Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights,” 2 November 2026. https://rikiyat.github.io/blog/weight-space-programming/ · Experiment 8 setup, sampling settings, samples per problem and results table, the loop mechanism, the Gemma 3 270M follow-up and the “obstacle” quote.
- [2] RikiyaT/lm_understand-logs on GitHub. https://github.com/RikiyaT/lm_understand-logs · complete logs of every reported experiment, the 1,125-turn tool-free audit, and the Experiment 8 edit as an exact diff.
- [3] Qwen3.5-0.8B model card, Hugging Face. https://huggingface.co/Qwen/Qwen3.5-0.8B · recommended sampling parameters per mode and the thinking-loop warning.
- [4] Andreas Maier, “Weight-Space Programming: What Is Measured and What Is Anecdote,” Andreas’ AI Morning Read. https://akmaier.substack.com/p/weight-space-programming-what-is · one Astra run per budget, the non-monotonic budgets, the missing cheaper baselines and the trial runs left out of the logs.
- [5] R&D World, “MIT team says GPT-6 Astra can improve other AI models by editing their weights. Critics say its biggest result relied on standard ML engineering.” https://www.rdworldonline.com/mit-team-says-gpt-6-astra-can-fix-an-ai-model-by-editing-its-weights-critics-say-the-method-was-standard-ml-engineering/ · authors and publication date, thinking mode switched on against the model’s default, each edited row as a separate run, and the Hugging Face evaluation of the released checkpoint (51.0%, 31.6% token-limit hits).
- [6] Sean Goedecke, “LLMs are not superhuman at reading and writing neural network weights.” https://www.seangoedecke.com/llms-are-not-superhuman-at-reading-and-writing-neural-network-weights/ · the Qwen fix described as a steering and direction-ablation playbook.
- [7] Stefan Christoph, “When Your Benchmark Runs for 88 Hours,” 30 September 2026. https://schristoph.online/blog/the-eval-horizon-problem/
- [8] Stefan Christoph, “Wrong, Not Broken: An Agent Evaluator Is Only as Good as Its Reference,” 30 October 2026. https://schristoph.online/blog/wrong-not-broken/
- [9] “Evaluate the performance of Amazon Bedrock resources,” Amazon Bedrock User Guide. https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html
- [10] “Create a prompt dataset for a model evaluation job that uses a model as judge,” Amazon Bedrock User Guide. https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-prompt-datasets-judge.html
- [11] “Create a prompt for a custom metric,” Amazon Bedrock User Guide. https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-custom-metrics-prompt-formats.html
- [12] “Evaluate model performance using another LLM as a judge,” Amazon Bedrock User Guide. https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-judge.html
- [13] “Use custom metrics to evaluate your generative AI application with Amazon Bedrock,” AWS Machine Learning Blog. https://aws.amazon.com/blogs/machine-learning/use-custom-metrics-to-evaluate-your-generative-ai-application-with-amazon-bedrock/
- [14] “Evaluate models or RAG systems using Amazon Bedrock Evaluations – Now generally available,” AWS Machine Learning Blog. https://aws.amazon.com/blogs/machine-learning/evaluate-models-or-rag-systems-using-amazon-bedrock-evaluations-now-generally-available/
❤ Created with the support of AI (Kiro)