schristoph.online

SageMaker ยท 14 min read

Serving a Decision Model on SageMaker AI, and Measuring What You Serve

On this page
TL;DR: Part 1 fine-tuned Qwen3.5-4B into a decision model that returns probabilities over declared options. Serving it with llama.cpp needs llama.cpp b11443 or newer, which Unsloth’s exporter source names as the first compatible release, with the converter and the /v1/systemone endpoint this model type needs. The AWS Deep Learning Container for llama.cpp is built from an older llama.cpp and routes /invocations to chat completions. So I built a small child image of it with one changed nginx line and served the 2.7 GB Q4_K_M file from a CPU endpoint. On ml.c5.9xlarge a case with five questions took 4.7 seconds on average. With the instance kept busy, that works out to about $2.70 per 1,000 cases at list price. The served model kept the same top answer as part 1’s numbers on 88 percent of 2,000 decisions. Its test accuracy was 0.764 against 0.787. On the first 100 test cases, the largest change in top answers came before llama.cpp, between the training-time model and the merged 16-bit checkpoint, though I didn’t isolate why. Evaluate the file you deploy, and pick thresholds from its numbers.

When part 1 ended I had a calibrated decision model, Qwen3.5-4B with LoRA adapters and a small scoring head, trained with Unsloth’s notebook, plus an evaluation and an escalation threshold [12]. What I didn’t have was anything an agent could call.

In October the AWS Machine Learning Blog published the serving half, decision models with llama.cpp on Amazon SageMaker AI, on CPU or GPU [1]. This post walks that path with the model from part 1. The trained model needs a newer llama.cpp than the container ships with (about twenty lines of Dockerfile fix that). And the model I deployed doesn’t return the same probabilities as the one I evaluated in part 1. You only see that if you evaluate a second time.

Two ways to get option probabilities out of llama.cpp

The AWS post shows two paths from one packaged GGUF file [1]. The first uses the AWS Deep Learning Container (DLC) for llama.cpp as published [3]: SageMaker’s /invocations goes to llama-server’s OpenAI-compatible chat completions. The second builds a child image with SemIf, an adapter that reads the final-position logits of a language model and restricts them to the answer slots of the declared options. It returns a probability per option without generating an answer token, and it works with a standard GGUF of a language model. The sample uses a community conversion of Qwen3.5-4B [1] [2].

Part 1’s model is a third case. Its probabilities come from a scoring head trained on my decisions, which sits next to the language model. SemIf reads the language model’s logits and wouldn’t use that head. To use the head, llama.cpp has to know the model type, which it does since release b11443, published on 6 October 2026. Unsloth’s exporter names it as the first release with a converter for Unsloth’s decision layout (called Clef) and with the /v1/systemone endpoint that serves it [6] [8]. The server answers typed questions about a state (the typed-decisions questions went in unchanged in my runs), and Clef reads all questions of a request in one prompt and decides them jointly [7].

The DLC’s changelog lists its llama.cpp source as b10433 [4]. The SageMaker CPU image I pulled on 13 October reported build 10964. Both are older than b11443. On top of that, the image’s nginx maps /invocations to /v1/chat/completions and proxies every other path unchanged [5]. SageMaker’s runtime only calls /invocations, so a newer llama.cpp also needs that route changed.

Converting the model

The merged checkpoint from part 1 converts with llama.cpp’s own script. The Clef converter writes the graph and the scoring head, and llama-server recognises what comes out, so /v1/models lists the model with the output modality decisions [7].

The converter drops the calibration temperatures. Unsloth’s calibration step fits one per question type, and Unsloth’s GGUF exporter says in its source that the Clef converter leaves them out, so the exporter writes them back into the file afterwards [8]. My converted file had no temperature keys at all. So I wrote a short script that copies them from the checkpoint’s unsloth_decision_config.json, with the same key names and the same clamp as Unsloth’s exporter. In this model they sit between 0.97 and 1.03, so I expect the effect to be small here, though I didn’t measure it.

Converting part 1’s merged checkpoint (llama.cpp b11443).

python llama.cpp-b11443/convert_hf_to_gguf.py merged --outtype bf16 --outfile model-BF16.gguf
llama-quantize model-BF16.gguf model-Q8_0.gguf Q8_0
llama-quantize model-BF16.gguf model-Q4_K_M.gguf Q4_K_M
for q in Q8_0 Q4_K_M; do
  python set_decision_temperatures.py model-$q.gguf merged/unsloth_decision_config.json
done

The files came out at 8.5 GB for BF16, 4.5 GB for Q8_0 and 2.7 GB for Q4_K_M. The AWS post uses Q4_K_M as its default [1], so that’s what went to the endpoint. Unsloth’s exporter also warns that k-quants of decision models can move answer probabilities and flip close answers, and that Q8_0 stays closest to PyTorch [8].

Six numbered steps in two rows: 1 merged checkpoint from part 1, 2 convert to GGUF with llama.cpp b11443, 3 write back the calibration temperatures, 4 quantize to Q8_0 or Q4_K_M, 5 upload to an S3 prefix with no model.tar.gz, 6 real-time endpoint where /invocations goes to /v1/systemone.

From part 1’s checkpoint to an endpoint. Step 3 exists because the converter drops the calibration temperatures.

A child image with one changed route

The AWS sample already shows how to extend the DLC. Its SemIf image starts FROM the published DLC and keeps the DLC’s compiled libraries [2]. I followed the same pattern with two changes. A build stage on Amazon Linux 2023, the same base as the DLC, compiles llama.cpp b11443 with one CPU backend per x86 generation, the way the DLC does [3]. The final stage copies those binaries over the DLC’s and swaps the nginx template so that /invocations goes to /v1/systemone. Everything else stays: the entrypoint that turns SM_LLAMA_CPP_* variables into llama-server flags, /ping to /health, and nginx in front.

The final stage of the child image (the build stage compiles llama.cpp b11443 on Amazon Linux 2023).

FROM public.ecr.aws/deep-learning-containers/llama-cpp:server-sagemaker-cpu-v1
RUN rm -rf /opt/llama.cpp/bin && mkdir -p /opt/llama.cpp/bin
COPY --from=build /out/ /opt/llama.cpp/bin/
# Same template as the DLC's, with /invocations -> /v1/systemone
COPY nginx-systemone.conf.template /etc/nginx/llama_cpp_sagemaker.conf.template

The image built in under three minutes on an 8-core machine, and after a test with the GGUF mounted at /opt/ml/model I pushed it to a private Amazon ECR repository for SageMaker to pull from.

Deploying it

The GGUF doesn’t need to go into a model.tar.gz, which saves packing and unpacking a 2.7 GB archive. With S3DataType set to S3Prefix and CompressionType set to None, SageMaker copies the objects under the prefix into /opt/ml/model as they are [9], and the DLC entrypoint picks up the first .gguf it finds there [5].

Clef evaluates the whole prompt in one batch, so the prompt has to fit into llama-server’s --ubatch-size [7]. A typed-decisions case with five questions came to 760 to 1,250 tokens, so I set context, batch and ubatch to 4,096 through SM_LLAMA_CPP_CTX_SIZE, SM_LLAMA_CPP_BATCH_SIZE and SM_LLAMA_CPP_UBATCH_SIZE.

I turned on network isolation, as the AWS sample does [2]. The container never has to call out, because the model arrives through the model data source. SageMaker expects a real-time response within 60 seconds [10], far above anything I measured, but keep it in mind for long states on a small CPU instance.

Instance choice came down to quota. My account had no endpoint quota for the ml.c6i instances the AWS post uses, and it did have some for ml.c5, so I deployed the same image and the same file three times, on ml.c5.2xlarge, ml.c5.4xlarge and ml.c5.9xlarge. Each endpoint was InService after about three minutes.

Latency and cost on CPU

I sent the first 100 test cases, in file order, to each endpoint, one request at a time. I didn’t test concurrent requests. The AWS sample scales by adding instances, because one llama.cpp context serves one request at a time there [2]. Each request is one case with five questions, and all 100 come from the same workflow (agent-trace review), with a mean of 763 prompt tokens. The cost column uses the mean latency and assumes the instance is kept busy with requests back to back, at the on-demand hosting price in Frankfurt on 13 October [11]:

InstancevCPUThreadsp50p95Price per hourCost per 1,000 cases
ml.c5.2xlarge8815.7 s16.4 s$0.466$2.03
ml.c5.4xlarge1688.7 s9.2 s$0.931$2.26
ml.c5.9xlarge36184.7 s5.2 s$2.095$2.73

Latency falls as cores are added, a little less than linearly at the top, and the cost per case stays in the same range. The larger instance buys speed, at a small premium per case. On the full 400 cases, which include longer states from the other workflows, ml.c5.9xlarge took 5.7 seconds at the median and 6.8 at p95, with a maximum of 8.6.

The probabilities were identical on all three instance types for the cases they shared, so once the served file is evaluated, the instance type changes the speed and leaves the answers alone.

A case here is five decisions, so divide the last column by five for the cost per decision. An endpoint costs its hourly price whether it is busy or not, and for sparse traffic the per-hour price matters more than the per-case one. These numbers also don’t compare with the benchmark in the AWS post. That was SemIf on a different benchmark with different prompts, measured on ml.c6i.8xlarge (1.7 seconds at the median on CPU) and on a GPU instance (0.35 seconds) [1]. I didn’t build a GPU image for the decision model. If seconds per case is too slow for your step, the AWS post’s GPU numbers suggest where to look next. Its advice is CPU for background or offline decisions and GPU when the decision sits in an interactive path [1]. I only tested the CPU side.

How far the served answers moved from part 1

Then I ran the endpoint over the full typed-decisions test split, 400 cases and 2,000 decisions [14], and scored its answers with part 1’s evaluator, unchanged:

AccuracyBrierKL from goldECE
Part 1, training-time predictions0.7870.1250.4980.030
Served Q4_K_M, SageMaker AI endpoint0.7640.1310.4420.040

Accuracy dropped by 2.3 points (95 percent bootstrap interval over cases: 0.8 to 4.0), and Brier rose by 0.006 (0.001 to 0.012). The calibration error change, +0.010 (โˆ’0.005 to +0.025), is within that noise.

KL from the gold distribution improved, because part 1’s predictions were rounded to four decimals, which turns small probabilities into exact zeros, and KL(gold โ€– model), clipped at 1e-12, punishes those. The server returns full precision; rounded the same way, its KL rises from 0.442 to 0.493.

For 87.9 percent of the 2,000 decisions the top answer stayed the same. Looking at the whole distribution instead of the top answer, 38 percent moved by more than 0.1 in total-variation distance. Score questions, the ordered scales such as urgency, flipped most often, 140 of 800 (17.5 percent, against 9 and 8 percent for the other two types). This is a paired comparison on the same decisions. And the served model returned identical probabilities on different instance types, so those 243 changed answers aren’t noise.

That matters for the escalation rule from part 1. At a threshold of 0.7, part 1 kept 72.7 percent of decisions local at 86.6 percent accuracy. The served model keeps 67.5 percent at 85.9 percent, so about five percentage points more traffic goes to the larger model (5.1 points, interval 3.0 to 7.2), and the local accuracy difference is within noise (โˆ’0.6 points, interval โˆ’2.2 to +0.9).

To find where the numbers moved, I scored the first 100 cases with every variant I had. The PyTorch reference is the merged checkpoint, run with the small joint_schema_model.py that Unsloth saves next to it, in bfloat16 on CPU and without Unsloth.

On the left, the training-time model (4-bit base plus LoRA, part 1 predictions), with an arrow labelled 82% same top answer to the merged 16-bit PyTorch model in the middle. From there two arrows fork to the right: 88% to GGUF Q4_K_M in llama.cpp and 98% to GGUF Q8_0 in llama.cpp.

Share of decisions with the same top answer at each step, on the first 100 test cases (500 decisions).

Variant (first 100 cases)AccuracySame top answer as merged PyTorch
Part 1, training-time predictions0.72681.8%
Merged checkpoint, PyTorch bf160.682(reference)
GGUF BF16, llama.cpp0.68699.6%
GGUF Q8_0, llama.cpp0.69298.0%
GGUF Q4_K_M, llama.cpp0.70087.8%

On these 100 cases the largest shift happens before llama.cpp gets involved at all, between part 1’s training-time predictions and the merged checkpoint. llama.cpp itself stays close. At BF16 it agrees with the PyTorch reference on 99.6 percent of decisions, at a mean total-variation distance of 0.009. Q4_K_M moves about one answer in eight, which matches Unsloth’s warning. It also lands closer to the training-time accuracy here, but that’s one workflow and 500 decisions, so I wouldn’t build on it.

I can’t say for certain why. Part 1 trained the adapters on a 4-bit base, as Unsloth’s notebook does, and the merged checkpoint runs them on 16-bit weights, so the served model isn’t the exact network that was trained and calibrated. The prediction code differs as well: part 1 used Unsloth’s predict, the reference uses the saved systemone() function, and I didn’t check whether the two build the prompt identically. Either way, the result is the same for anyone deploying: the numbers from the training run describe a model you are not serving.

This is the gate problem I wrote about in The Gate That Couldn’t See the Defect [13], one step further along: part 1’s evaluation was sound, and it measured a different artifact from the one I deployed. According to the llama.cpp server documentation, probabilities are scaled with the temperatures stored in the file, and they aren’t guaranteed to be calibrated for your data [7].

What I would do differently next time

Evaluate the endpoint as part of the deployment. Keep a held-out set and run it through the endpoint after every conversion, quantization or image change, then compare against the previous served version with the same evaluator. Because you look at it after every change, treat it as a deployment-validation set, not as your final test. Here that took 38 minutes on ml.c5.9xlarge for 400 cases. Part 1’s temperatures were fit on the training-time model. With the served model’s probabilities from a calibration split kept apart from the test split, the temperature fit can be repeated and written into the GGUF with the same script. Pick the threshold from the served model’s sweep on a separate validation split, keep an untouched test split for the numbers you report, and redo both when the file changes.

Try Q8_0 where close decisions matter. It is 4.5 GB (the Q4_K_M file is 2.7 GB) and stays closer to the merged model. On CPU a larger file means a bit more memory, and I didn’t benchmark Q8_0 on an endpoint.

Consider training on a 16-bit base if you know you’ll merge and quantize. That removes one of the two candidate causes above, at the cost of more GPU memory during training. I haven’t done the temperature refit or the 16-bit training yet.

Cleaning up

The endpoints, endpoint configurations and models went first, as soon as each run finished. Then the ECR repository, the execution role, the endpoint log groups, the GGUF in S3, and the S3 bucket with part 1’s checkpoints. The AWS post’s cleanup list makes a good checklist, because its script deliberately deletes only the three SageMaker resources [1]. By my estimate, all three endpoints together cost about $2.30 at list price for the time they were up: roughly 0.55, 0.35 and 0.8 hours including provisioning, at $0.466, $0.931 and $2.095 per hour, which gives $0.26, $0.33 and $1.68, plus a few cents for storage and logs. That is a list-price estimate, not the bill.

Which step in your agent would you serve this way, and would you rather pay for CPU or GPU instance hours for it?

The companion code for this post (temperature script, child image, deploy and cleanup scripts, the scorer and the comparison) will be in my blog samples repository.

Sources

๐ŸŽฌ Also available as a blog walkthrough video on YouTube

โค Created with the support of AI (Kiro)

SageMakerInferenceAIAgentsEvaluationCostOptimizationHandsOn