
AIAgents ยท 14 min read
Many Agent Steps Are Classification in Disguise
On this page
Look at the steps of an agent and count how many of them end in a choice the code already knows the shape of. Which skill handles this request. Does a person need to confirm before the agent sends the message. Is this urgent today, this week, or not at all. In many agents, including ones I have built, each of those steps sends a prompt to a large language model and parses a word out of the reply.
That works, and it’s the fastest way to get started. But parsing the reply gives you no number to act on. The model says “calendar”, and you can’t tell whether it meant “calendar, certainly” or “calendar, I suppose”. So there’s nothing to put a threshold on, and no way to count how often the hesitant answers are wrong.
In October 2026 the AWS Machine Learning Blog put the problem in one sentence: applications that route requests, select agent actions, or assign policy outcomes often need scores over a fixed set of choices rather than free-form text [1]. This post is part 1 of two. It goes from the data through the training run to the evaluation. Part 2 takes the trained model to a SageMaker AI endpoint with llama.cpp, following that AWS post.
What a decision model returns
A decision model reads three things: the state (the request, the ticket, the agent trace), the question, and the options you declare for that question. It returns a probability for every option instead of generating text.
Unsloth’s FastDecisionModel builds one from an ordinary language model. It puts the input, every question and every option into one prompt, lets the model read it once, and adds a small scoring head that rates each option from the model’s internal state [2] [3]. You train LoRA adapters on the language model and the new head together.
The question types are small and cover the fixed-choice steps discussed here [4]:
| Type | Use it for | You get back |
|---|---|---|
noul | yes or no | the probability of yes |
choice | one of N options | a probability per option |
score | an ordered scale | a probability per level |
One request can carry several questions about the same input. A support ticket can be routed to a team, checked for a refund request and rated for urgency in one pass.
You can get part of the way without training. Read the probabilities a large model assigns to each option and skip the writing. That’s roughly what the SemIf adapter in the AWS post does at serving time: it reads the final logits over declared answer slots without generating an answer token [1]. Those raw scores stay unvalidated until you evaluate them on labelled, held-out outcomes from your own decisions. Fine-tuning adds a head trained on your options. Calibration (Unsloth fits a temperature per question type on held-out rows) would work on raw scores too.
Shaping the data
The format Unsloth trains on comes from the typed-decisions benchmark on Hugging Face [5]. Each row has a state, a questions object with the type, instructions and a written description of every option, and a gold object with a full probability distribution per question. The descriptions are part of the input, so writing them well is part of the work.
The benchmark has four workflows with 300 training and 100 test cases each: customer service, invoice processing, security incidents, and one that agent builders will recognise, deciding whether an agent run needs human review and how urgently [5]. It’s synthetic, generated from sampled latent factors and then labelled.
For my own step I wanted a router: given a request, which skill handles it, does a human need to confirm, and how urgent is it. I wrote a small generator that builds such rows from templates, with five skills (research, calendar, writing, expenses, other), and checked its field layout against real rows from the benchmark. A generated row looks like this:
One synthetic routing row (shortened). The questions and gold fields are JSON strings, as in the benchmark.
{"id": "routing-train-0000",
"state": "{\"request\": \"Is there an official guide for migrating to object storage? This is blocking a customer demo today.\"}",
"questions": "{\"skill\": {\"type\": \"choice\", \"instructions\": \"Which skill should handle this request?\", \"criteria\": {\"research\": \"Answer a question by reading documentation or the web.\", \"calendar\": \"...\"}}, \"needs_human\": {\"type\": \"noul\", ...}, \"urgency\": {\"type\": \"score\", ...}}",
"gold": "{\"skill\": {\"type\": \"choice\", \"label\": \"research\", \"probabilities\": {\"research\": 0.87, ...}}, ...}"}
Template data is a smoke test. It proves the pipeline runs and the format is right, and it is far easier than real traffic, because every request was written to belong to one skill, and real requests mix intents (“find a slot and draft the invite”). Start synthetic, then add labelled requests from logs you are allowed to train on, and keep a share of those for testing only.
Two constraints shape the questions. Keep the option list short. Unsloth’s serving docs note that options in a choice share a fixed token budget, so past about 20 described options the labels get trimmed [4]. If your router has 60 tools, decide the tool family first and the tool second. Keep authorization and deterministic business rules outside the model, as the AWS post recommends [1].
The training run
The recipe is short. This is the core of Unsloth’s published notebook, with its defaults [2]:
Fine-tuning Qwen3.5-4B into a decision model (from Unsloth’s notebook, 4-bit base, LoRA rank 16).
from unsloth import FastDecisionModel, DecisionTrainer, is_bfloat16_supported
from transformers import TrainingArguments
model, tokenizer = FastDecisionModel.from_pretrained(
model_name="unsloth/Qwen3.5-4B", max_seq_length=2048, load_in_4bit=True)
model = FastDecisionModel.get_peft_model(
model, r=16, lora_alpha=16, lora_dropout=0,
use_gradient_checkpointing="unsloth", random_state=3407)
items, report = FastDecisionModel.build_dataset(train_rows, tokenizer, model)
train_items, eval_items = FastDecisionModel.split_holdout(items, seed=3407)
trainer = DecisionTrainer(
model=model, processing_class=tokenizer,
train_dataset=train_items, eval_dataset=eval_items,
args=TrainingArguments(
per_device_train_batch_size=8, gradient_accumulation_steps=4,
warmup_steps=10, max_steps=60, learning_rate=2e-4,
lr_scheduler_type="cosine", weight_decay=0.01,
bf16=is_bfloat16_supported(), fp16=not is_bfloat16_supported(),
output_dir="outputs", report_to="none", seed=3407))
trainer.train()
FastDecisionModel.calibrate(model, tokenizer, eval_items) # on held-out rows
model.save_pretrained("qwen-decisions") # adapters + head + calibration
Unsloth reports these results for that notebook. The new head starts at about 30 percent accuracy before training, because it is new and guesses, and reaches 75 to 78 percent on the benchmark’s test split after 60 steps. On the held-out rows, which are a different set, the notebook reports 80 to 84 percent accuracy and an expected calibration error of 0.02 to 0.06 after calibration [2]. Temperature scaling doesn’t change which option ranks first, so calibration moves the error, not the accuracy. Those are from Unsloth’s runs, so treat them as a target to reproduce on your own data.
The hardware note in the notebook is worth reading twice. On a free T4 the 60 steps take about two hours, because the T4 has no bfloat16, Qwen3.5 produces NaNs in pure float16, and Unsloth falls back to float32 [2]. Check for bfloat16 support before you pick a GPU. Unsloth’s guide reports 76 percent after the same 60 steps in 10 minutes on an L4 [3].
On AWS the L4 is the GPU of the G6 instance family. The G5 family carries the A10G, with 24 GB per GPU [7] [8]. One A10G was enough for my run below; I didn’t record peak memory. For orientation, Unsloth lists 3.5 GB as the minimum for a 3B model and 5 GB for 7B in 4-bit [6].
The training run. Calibration uses the held-out rows; the test split is only for the final measurement.
My companion code wraps this in a script that also writes per-case predictions for the evaluator and saves both the adapter and a merged 16-bit model. The calls are the ones in Unsloth’s notebook and guide. I ran it once on an EC2 g5.xlarge: 60 steps took 17.6 minutes, held-out accuracy was 0.800 both before and after calibration, and the calibrated model scored 0.785 on the test split.
What the accuracy number measures
The typed-decisions gold labels are the mean of three samples from a teacher model of roughly 4B capability [5]. A score on this benchmark is agreement with that teacher. When the dataset authors drew a fresh sample from the teacher and compared it with the gold built from the others, it agreed 73.5 percent of the time, and the card reads scores well above that as a sign that a model is learning the teacher’s quirks [5]. That is the card’s interpretation, not a hard limit: a student can beat a single fresh sample by averaging out the teacher’s noise. The fresh-sample teacher-agreement rate also varies by question, from 0.560 for the urgency of an agent trace to 0.937 for the category of a customer conversation [5].
So the notebook’s 75 to 78 percent says the fine-tuned 4B student agrees with the teacher about as well as the teacher agrees with itself. That’s a good result, but it’s the most this benchmark can show, since it says nothing about whether the teacher was right. The card’s own example is an invoice the teacher missed as a duplicate even though its ID matched an earlier one [5]. Your gold labels will have the same property, whoever writes them. If a large model labels your routing data, that model’s self-consistency is a reference point for your router’s accuracy, not proof it is right.
Model size still matters once you leave the training distribution. The AWS post benchmarked Qwen3.5 GGUF models served with its SemIf adapter on 231 tasks from a different public benchmark. The 4B model reached 77.5 percent on CPU and 77.9 percent on GPU. The 2B and 0.8B models reached 66.7 and 58.4 percent [1]. Unsloth, for its part, reports that fine-tuning on a mix of 13 sources, typed-decisions among them, moves 0.8B and 2B models from 36 and 33 percent to 73 and 78 percent on typed-decisions [3]. The two numbers come from different benchmarks and different setups, so I wouldn’t subtract one from the other. Read together, they suggest that a small model trained on your own decisions can do well on those decisions, and that you should test it on inputs that look like next month’s traffic as well as last month’s.
Checking the evaluator
Accuracy only looks at the top option. The reason to use a decision model at all is the probabilities, so the evaluation has to look at them too. The typed-decisions card reports three such measures next to accuracy: Brier score (squared distance between the predicted and the gold distribution), KL divergence from the gold distribution, and expected calibration error (ECE, the gap between stated confidence and observed accuracy) [5].
I wrote a small evaluator for these, and before pointing it at any model I ran it on the two baselines the card publishes: a uniform guess, and a prior that ignores the input and always answers each question’s average training distribution. My first version disagreed with the card. I had averaged the Brier score over options where the card sums it, and I had broken ties by option order. After fixing both, the uniform baseline matches the card’s Brier score and KL divergence to three decimals (Brier summed over options, KL from gold, ECE over 10 equal-width confidence bins; revision c806f6c). The prior row is close, not exact, and I haven’t found why:
| Baseline (test split, 2,000 decisions, my run vs card) | Accuracy | Brier | KL from gold | ECE |
|---|---|---|---|---|
| Uniform, my evaluator | 0.318 | 0.238 | 0.444 | 0.000 |
| Uniform, dataset card [5] | 0.308 | 0.238 | 0.444 | 0.169 |
| Prior, my evaluator | 0.479 | 0.181 | 0.327 | 0.034 |
| Prior, dataset card [5] | 0.470 | 0.189 | 0.347 | 0.088 |
The accuracy gap on the uniform row is tie handling (my evaluator splits the credit across tied options). The ECE column doesn’t match at all, and the uniform row shows why that metric needs company. Under my definition a uniform guess is perfectly calibrated, because its stated confidence always equals its expected accuracy. It is also useless. The card’s prior shows the same thing. It has the best ECE of all its rows while knowing nothing about the input, which is why the card tells readers to look at KL and Brier [5].
Then I scored my trained model with the same evaluator. Its Brier score was 0.125, better than both baselines, but its KL divergence from gold was 0.497, worse than both. Unsloth’s calibration fits against the hard gold label, not the soft gold distribution, so it sharpens the probabilities: on the held-out rows ECE fell from 0.18 to 0.03 while the soft cross-entropy rose from 0.85 to 1.13. A fix on one metric can cost you on another.
I wrote about the same trap from a different angle in The Gate That Couldn’t See the Defect [9]: a check that cannot separate a good result from a bad one gives you confidence you have not earned. So run every metric on the published baselines first. For a decision model those cases are free, because the benchmark publishes them. If your numbers differ from published ones, find out why before the model’s numbers mean anything. Mine match on Brier and KL for the uniform row, the prior row is still slightly off, and my ECE uses a different definition, so I don’t compare it with the card.
The escalation rule
Once the probabilities have been checked on held-out outcomes, the agent can put a threshold on them. If the top option’s probability is at or above it, the small model’s answer stands. Below it, the step goes to a larger model or to a person. This is a policy you measure, not a guarantee: it won’t catch a confident error or an input unlike anything in training.
Escalation on probability. The threshold comes from a sweep on held-out data, and verified or adjudicated escalated outcomes can become labelled examples.
Pick the threshold from data. My evaluator sweeps it: for each threshold it reports the share of decisions answered locally, their accuracy, and the share escalated. Read that table with whoever owns the step. In my run I swept the test split itself, so this describes the model rather than selecting a threshold: at 0.7, 72.7 percent of test decisions were answered locally at 86.6 percent accuracy. The sweep only counts correct answers, so the prior mismatch doesn’t affect it. In production, choose the threshold on held-out data and report it once on an untouched test split. A routing step can live with a lower bar than a “send to the whole company” check.
Three details decide whether this works in practice. The threshold belongs on the option’s probability, and a separate confidence field is the wrong place for it. Unsloth’s docs recommend this, because confidence formulas differ between implementations and thresholds don’t transfer [4]. Calibrate on held-out data from your own traffic, and again when the input mix changes, because a model calibrated on last quarter’s requests has no reason to stay calibrated on a new product’s. Log every escalated case with its final answer, and once someone has checked that answer, it is a labelled example for your next training run.
This is the kind of control I argued for in From Chaos to Control [10]: keep the flexibility of a language model where you need it, and put the predictable, testable part where you can. A decision model with a threshold is a small, measurable component in a place that used to be a prompt, as long as you keep measuring it on new traffic.
What part 2 does
Part 2 serves the model. The AWS post shows two paths on SageMaker AI with the AWS Deep Learning Container for llama.cpp: OpenAI-compatible chat completions, and a child image with the SemIf adapter that returns probabilities over declared options without decoding an answer token, on CPU or GPU [1]. I will take the decision-model path from this post to an endpoint, measure latency and cost on CPU and GPU, and tear it down. This post makes no cost claim; those numbers come from that run.
Which step in your agent would you replace first, and what would you set the threshold to?
The companion code for this post (data generator, baselines, evaluator and training script) is in my blog samples repository; the data and evaluation parts run on a laptop CPU.
Sources
- [1] Deploy decision models with llama.cpp on Amazon SageMaker AI โ AWS Machine Learning Blog, Jyothirmai Kottu, Dmitry Soldatkin, Daniel Quang and Daniel Wirjo, 12 October 2026
- [2] Qwen3.5 (4B) Decision notebook โ Unsloth, LGPL-3.0
- [3] Train your own Decision Model โ Unsloth documentation
- [4] Run and Serve Decision Models Locally with Unsloth: Laya + Jev API โ Unsloth documentation
- [5] Typed Decisions dataset card โ Hugging Face, Apache-2.0
- [6] Unsloth Requirements โ Unsloth documentation
- [7] Amazon EC2 G6 instances โ AWS
- [8] Amazon EC2 G5 instances โ AWS
- [9] The Gate That Couldn’t See the Defect โ Stefan Christoph
- [10] From Chaos to Control: Building Predictable AI Agents That Get Smarter Over Time โ Stefan Christoph
๐ฌ Also available as a blog walkthrough video on YouTube
โค Created with the support of AI (Kiro)