Your AI Models Have an Expiry Date — A Practical Guide to Model Lifecycle Management

Introduction, The Promise I Made

In my previous article [1], I explored the maintenance trap in IT, how software systems are more like plants than stones, requiring constant care. I ended with a cliffhanger: “What is open from the article is how to specifically test and evaluate models, something to be picked up in the next article.”

This is that article.

Since publishing the first piece, something happened that made this topic very real for many of my customers. Anthropic announced the deprecation of the Claude 3.5 Sonnet models (claude-3-5-sonnet-20240620 and -20241022) on its own API, with an October 28, 2025 retirement date [21], Amazon Bedrock and Google Cloud set their own, separate retirement schedules for the same models. Claude 3.5 Sonnet had become the backbone of countless production applications, and teams that had built their systems around a specific model version suddenly faced a hard deadline to migrate. Of the teams I spoke with, some were prepared. Most were not.

The reactions I observed ranged from mild annoyance to genuine panic. And this is just the beginning. As AI models become critical infrastructure components, the question is no longer if your model will be deprecated, but when, and whether you’ll be ready.

The Lifecycle Compression Problem

Let’s put numbers on this. Traditional enterprise software operates on generous support timelines. Microsoft provides five years of mainstream support followed by five years of extended support. Cisco gives six months advance notice before End of Sale dates. Server hardware follows five to seven-year lifecycle patterns.

AI models? Completely different.

Amazon Bedrock guarantees a minimum of 12 months from launch to End-of-Life. Azure OpenAI provides 12 months of Generally Available support plus six months of extended access. Google Cloud sets retirement dates with new access blocked one month before. OpenAI retired its first multimodal model in February 2026 with minimal notice.

We went from a decade to a year. A 6-10x compression in lifecycle duration.

And it’s not just one model. Modern applications typically use three to five different AI models [2], each on its own deprecation timeline. The compounding overhead of managing multiple model lifecycles simultaneously, one deprecation clock per model, is something most organizations haven’t even begun to think about.

Understanding the Lifecycle, Bedrock as an Example

Amazon Bedrock provides a structured lifecycle that is a good reference model for how managed AI services handle deprecation. Every model on Bedrock exists in one of three states:

Active, The model provider is actively working on this version. Inference is available, and so are fine-tuning and provisioned throughput where the specific model and Region support them.

Legacy, The model will be retired. It remains available for at least six months before the EOL date. You can continue using existing deployments, but you cannot create new fine-tuning jobs or new provisioned throughput endpoints.

End-of-Life, The model is gone. API calls return errors.

The key policy: once a model launches on Bedrock, it will remain available for at least 12 months before the EOL date [3]. This is your planning horizon.

What makes this manageable is that Bedrock exposes the lifecycle programmatically. The GetFoundationModel and ListFoundationModels APIs return a modelLifecycle field with the current state. You can, and should, build automated monitoring around this: a scheduled EventBridge rule that invokes a small function to poll those APIs, compare against the last known state, and alert your team when a model you depend on transitions to Legacy gives you a six-month head start on migration planning.

The Architecture That Saves You, Abstraction Layers

The single most important architectural decision for model lifecycle management is one you need to make before you deploy your first model to production: decouple your business logic from specific models.

AWS Prescriptive Guidance puts it clearly: “Use model selection abstraction layers, decouple business logic from specific models to enable dynamic routing, fallback, or cost-performance tuning over time.” [4]

In practice, this means three things:

1. Use the Converse API

If you’re on Amazon Bedrock, the Converse API [5] provides a consistent interface across the models that support it. Your application code talks to one API. Switching models starts with changing a model ID string rather than rewriting your integration layer, though supported parameters, tool use, modalities, and response behavior still differ per model, which is exactly why the evaluation workflow below exists. This is the minimum viable abstraction.

2. Build Model Fallback Chains

Model fallback chains are backup models that activate automatically when primary models fail or become unavailable. The mechanism detects errors, rate limits, or outages and routes requests to pre-configured alternatives [6]. Think of it as a load balancer for intelligence, if Model A returns a 404 because it’s been retired, Model B takes over automatically. The routing is the easy part, though: Model B answering is not the same as Model B answering equivalently. Treat every fallback pair like a mini-migration, evaluate capability compatibility and output quality up front, and be careful where non-idempotent tool calls get retried.

Salesforce’s Agentforce platform demonstrates what mature failover looks like: gateway-level failovers, soft failovers, and circuit breakers in a multi-layered design [7]. You don’t need to be Salesforce to implement this pattern, but you do need to think about it before your primary model disappears.

3. Use Intelligent Prompt Routing

Amazon Bedrock provides a native routing capability that dynamically routes requests between models within the same family based on query complexity [8]. It predicts response quality per request and routes to the best model, optimizing for both quality and cost. The service can reduce costs by up to 30% without compromising accuracy.

The lifecycle benefit? The router gives you a single control point for model transitions: when a new model version becomes available, you add it to the router configuration and re-evaluate; when an old version approaches EOL, you shift traffic away in one place. Adding or removing models remains an explicit, evaluated change, but it’s a routing-layer change, not application code scattered across your services.

The Evaluation Imperative, Going Beyond Vibes

Here’s where most teams fall short. They have an abstraction layer. They know a model is being deprecated. They even have a candidate replacement model identified. But they don’t have a systematic way to verify that the replacement actually works for their use case.

When Andrej Karpathy coined the term “vibe coding” in February 2025, describing a workflow where you “fully give in to the vibes and forget that the code even exists”, he captured something real about how we interact with AI. The AWS Public Sector team applied the same concept to evaluation, calling it “vibe testing”, validating AI solutions based on emotional reactions to model outputs instead of quantitative criteria [9]. “It seems to work fine” is not an evaluation strategy.

The fix is to establish evaluation baselines before you need to migrate. When a model transitions to Legacy, you should already have the data to compare against.

Building Your Evaluation Harness

Amazon Bedrock Evaluations provides three complementary approaches:

Programmatic evaluations compute scores using built-in metrics (accuracy, robustness, toxicity). Run the same evaluation dataset [20] against your current model and the candidate replacement. Compare scores, against acceptance thresholds you defined up front, and with repeated runs where sampling variance could swamp the difference. This is your first gate.

LLM-as-a-Judge uses a second model to score the generator model’s responses on metrics like correctness, completeness, helpfulness, and responsible AI criteria. This delivers human-like evaluation quality with up to 98% cost savings compared to human evaluation [10]. You can define custom metrics for your specific business requirements, brand voice alignment, domain accuracy, regulatory compliance (as a screening signal: an actual compliance determination still needs rule-based checks and qualified human review).

Human evaluations bring subject-matter experts into the loop for high-stakes use cases where automated metrics aren’t sufficient. Healthcare, legal, financial services, anywhere the cost of a wrong answer exceeds the cost of human review.

The critical feature for lifecycle management: Bring Your Own Inference (BYOI). You can evaluate models running anywhere, on Bedrock, other clouds, or on-premises [11]. This means the models under test aren’t locked to a single provider’s ecosystem, the evaluation service itself still runs on AWS, but it can judge inference outputs from anywhere.

The Evaluation Workflow for Model Migration

Based on the patterns I’ve seen work in practice, here’s the workflow:

1. Baseline: Run your evaluation suite against the current production model. Programmatic metrics, LLM-as-a-Judge, and domain-specific custom metrics. Store the results. This is your reference point.

2. Candidate: Run the exact same suite against the replacement model. Same dataset, same metrics, same guardrail configuration. No shortcuts.

3. Compare: Side-by-side analysis. Bedrock Evaluations lets you view multiple evaluation jobs together. Look for regressions, improvements, and edge cases.

4. Shadow: Deploy the candidate in shadow mode against production traffic. Log responses without serving them to users. Compare quality, latency, and cost in real-world conditions.

5. Canary: Route a small percentage of traffic to the candidate. Monitor for regressions that didn’t surface in evaluation.

6. Rollout: Full migration with rollback capability. Keep the old model available as a fallback during the transition period.

This isn’t theoretical. The AWS blog on migrating from Claude 3.5 Sonnet to Claude 4 Sonnet [12] walks through exactly this pattern, including prompt engineering adjustments, new capability enablement, and phased deployment strategies.

Evaluating Agents, A Harder Problem

For agentic AI systems, evaluation gets more complex. You’re not just testing a model’s response to a prompt. You’re testing the observable trajectory, intermediate outputs, tool invocations, and multi-step workflows, not the model’s hidden reasoning itself.

Two approaches stand out:

Ragas with LLM-as-a-Judge [13] provides an open-source framework for evaluating Bedrock Agents on RAG, text-to-SQL, and chain-of-thought capabilities. It evaluates the full agent trajectory, not just the final output.

Lightweight evals without a framework: Gunnar Grosch published an excellent Builder article [14] with practical evaluation patterns that don’t require a full eval framework. It comes with a demo repository you can fork and adapt. The key insight: you don’t need a massive evaluation infrastructure to start. You need a curated set of test cases and a systematic way to run them.

SageMaker, When You Own the Model

Everything above applies to foundation models consumed via APIs. But many organizations also train, fine-tune, and deploy their own models. This is where Amazon SageMaker’s lifecycle management tooling comes in.

SageMaker Model Registry provides a central catalog for model versions, think of it as a package manager for ML models. Each model group contains versioned model packages with metadata, approval status, and lineage tracking. The staging construct lets you define custom lifecycle stages (Development → Testing → Production) with IAM-enforced permissions at each transition [15].

The power is in the automation. EventBridge integration means you can trigger deployment pipelines when a model moves to “Approved” status. Model Monitor watches production traffic for data drift and quality degradation, alerting you when a model’s performance is drifting from its baseline. Whether a newer model version would perform better is a separate question, that’s a benchmark you run against a candidate, not an alert you receive.

For organizations using both Bedrock and SageMaker (increasingly common), the SageMaker Model Registry can track which Bedrock foundation model version is used in each deployment, a mapping you build yourself with the registry’s custom metadata, not a built-in integration, creating a unified lifecycle view across provider-owned and customer-owned models.

The Governance Dimension

Model lifecycle management doesn’t exist in a vacuum. It sits within a broader governance framework that’s becoming a regulatory requirement.

The EU AI Act entered into force on 1 August 2024, with obligations phasing in over 36 months. The timeline matters for model lifecycle management specifically:

• February 2025: Prohibited AI practices banned (social scoring, manipulative AI). Violations: up to €35 million or 7% of global annual revenue.
• August 2025: General-purpose AI (GPAI) model provider obligations apply: technical documentation, training data summaries, copyright compliance. Violations: up to €15 million or 3% of global turnover.
• August 2026: High-risk AI system obligations fully apply: risk management, data governance, human oversight, and critically: automatic event logging (Article 12). Violations: up to €15 million or 3% of global turnover.

Article 12 is the one that directly connects to model lifecycle management. It requires that high-risk AI systems technically allow the automatic recording of events (logs) over their lifetime, so that system behavior stays traceable. Retention is a separate duty: under Article 19, providers must keep those logs for a period appropriate to the system’s intended purpose and for at least six months, unless other Union or national law, data protection law in particular, provides otherwise [16]. Note what that is not: a flat number you can hardcode. The floor is six months, the actual obligation depends on the system’s purpose and on whatever other law applies to you. In practice, building toward these obligations means knowing which model version produced which output, and a defensible governance trail adds when that version was evaluated and who approved the transition. The logging capability isn’t a best practice. It’s a legal requirement for high-risk systems operating in the EU.

Beyond the EU, the Colorado AI Act takes effect in July 2025, and ISO/IEC 42001 provides the first certifiable AI management system standard [17], certification applies to an organization’s management system, not to an AI model or product, giving organizations a structured framework for responsible AI governance.

Despite this regulatory momentum, only about 24% of organizations have AI governance programs in place [18]. The gap between adoption and governance is staggering, and it’s a gap that model lifecycle management directly helps close, because the evaluation baselines, migration records, and version tracking you build for operational reasons double as the audit trail regulators will ask for.

SageMaker Model Cards provide immutable records of intended model uses, risk ratings, training details, and evaluation results. Bedrock’s logging and monitoring capabilities feed into CloudWatch and CloudTrail. Together, they create the documentation trail that serves operations and supplies evidence for compliance, mapping that evidence to your specific obligations, and judging whether it’s sufficient, remains work for your governance and legal teams.

Hands-On: Try It Yourself

Theory is great, but builders want to get their hands dirty. Here are three concrete paths you can follow today, from the simplest to the most thorough.

Path 1: Compare Two Models in 15 Minutes (Bedrock Console)

This is the fastest way to experience model evaluation. No code required.

Step 1: Prepare your dataset. Create a JSONL file with prompts that represent your use case. Each line is a JSON object with a prompt key. You can have up to 1,000 prompts per evaluation job. Start with 20-50 that cover your critical scenarios.

{"prompt": "Summarize the key risks of deploying AI in healthcare."}
{"prompt": "Write a Python function that validates an email address."}
{"prompt": "Explain the difference between RAG and fine-tuning to a non-technical stakeholder."}

If you’re evaluating for a model migration, use real prompts from your production system, not synthetic ones.

Step 2: Upload to S3. Put your .jsonl file in an S3 bucket in the same region as your Bedrock models. Enable CORS on the bucket (required for console-based evaluations) [19].

Step 3: Create the evaluation job. Open the Bedrock console → Evaluations → Create evaluation job. Select “LLM-as-a-Judge” as the evaluation type. Pick your current production model as the generator. Select an evaluator model (Amazon Nova Pro or Claude work well). Choose metrics, start with correctness, completeness, and helpfulness.

Step 4: Run a second job with the candidate model. Same dataset, same evaluator, same metrics, just swap the generator model to the one you’re considering migrating to.

Step 5: Compare. Use the Bedrock Evaluations compare feature to view both jobs side by side. Look at the histogram of scores and read the evaluator’s explanations for the first few prompts. You’ll immediately see where the candidate model improves, matches, or regresses.

That’s it. You now have a quantitative comparison between two models on your actual use case. No evaluation framework to build, no application code to write, just the prerequisites the job itself needs: Bedrock model access, an S3 bucket with CORS enabled, an IAM role for the evaluation job, and the job’s inference charges.

Path 2: Automate Evaluation with Python SDK

When you need to run evaluations programmatically, for example, as part of a CI/CD pipeline that triggers when a model transitions to Legacy, use the Bedrock Python SDK.

import boto3
import json

bedrock = boto3.client("bedrock", region_name="us-east-1")

# Create a programmatic evaluation job (built-in automated metrics).
# An LLM-as-a-Judge job additionally configures an evaluator model.
response = bedrock.create_evaluation_job(
    jobName="claude-migration-eval-2026-03",
    roleArn="arn:aws:iam::123456789012:role/BedrockEvalRole",
    evaluationConfig={
        "automated": {
            "datasetMetricConfigs": [
                {
                    "taskType": "General",
                    "dataset": {
                        "name": "my-production-prompts",
                        "datasetLocation": {
                            "s3Uri": "s3://my-eval-bucket/prompts.jsonl"
                        }
                    },
                    "metricNames": [
                        "Builtin.Accuracy",
                        "Builtin.Robustness"
                    ]
                }
            ]
        }
    },
    inferenceConfig={
        "models": [
            {
                "bedrockModel": {
                    "modelIdentifier": "anthropic.claude-sonnet-4-5-20250929-v1:0",
                    "inferenceParams": "{\"inferenceConfig\":{\"maxTokens\":512,\"temperature\":0.0}}"
                }
            }
        ]
    },
    outputDataConfig={
        "s3Uri": "s3://my-eval-bucket/results/"
    }
)

eval_job_arn = response["jobArn"]
print(f"Evaluation job started: {eval_job_arn}")

Run this for both your current and candidate model, then compare the results in S3. You can parse the output JSONL to compute aggregate scores and flag regressions automatically.

Tip: Run this on a schedule: an EventBridge rule that invokes a Lambda function which polls GetFoundationModel, keeps the previously seen state, and kicks off the evaluation when a model you depend on newly reports modelLifecycle: LEGACY. Fully automated migration readiness assessment.

Path 3: Full Lifecycle with SageMaker Model Registry

For organizations that train or fine-tune their own models, SageMaker provides the complete lifecycle management infrastructure.

Step 1: Create a Model Group. This is your container for all versions of a model solving a specific problem.

import boto3

sm = boto3.client("sagemaker")

response = sm.create_model_package_group(
    ModelPackageGroupName="my-text-classifier",
    ModelPackageGroupDescription="Production text classifier — all versions"
)

Step 2: Register model versions. Each trained model becomes a versioned package in the group.

response = sm.create_model_package(
    ModelPackageGroupName="my-text-classifier",
    ModelPackageDescription="v3 — retrained on Q1 2026 data",
    InferenceSpecification={
        "Containers": [{
            "Image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-model:v3",
            "ModelDataUrl": "s3://my-models/v3/model.tar.gz"
        }],
        "SupportedContentTypes": ["application/json"],
        "SupportedResponseMIMETypes": ["application/json"]
    },
    ModelApprovalStatus="PendingManualApproval"
)

Step 3: Set up the staging construct. Define lifecycle stages with permissions, only ML engineers can move models to Testing, only the ML lead can approve for Production.

response = sm.update_model_package(
    ModelPackageArn=model_package_arn,
    ModelLifeCycle={
        "Stage": "Development",
        "StageStatus": "InProgress",
        "StageDescription": "Initial training complete, pending evaluation"
    }
)

Step 4: Automate with EventBridge. When a model’s stage changes, trigger your evaluation pipeline, deployment workflow, or notification.

# EventBridge rule pattern for lifecycle transitions
{
    "source": ["aws.sagemaker"],
    "detail-type": ["SageMaker Model Package State Change"],
    "detail": {
        "ModelLifeCycle": {
            "Stage": ["Production"],
            "StageStatus": ["Approved"]
        }
    }
}

Step 5: Monitor in production. SageMaker Model Monitor detects data drift and quality degradation. When it fires an alert, that’s your signal to run a separate benchmark and evaluate whether a newer model version or a different foundation model would perform better.

Which Path Should You Start With?

The key insight: start with Path 1 today, even if you think you need Path 3 eventually. Having any evaluation baseline is infinitely better than having none when the deprecation email arrives.

A Practical Checklist

If you take away one thing from this article, let it be this checklist:

• Use the Converse API (or equivalent abstraction). Never hardcode model-specific integrations
• Monitor model lifecycle states. Automate alerts when models you depend on transition to Legacy
• Build evaluation baselines now. Don’t wait for a deprecation announcement
• Implement fallback chains. At minimum, configure a secondary model for every primary model
• Version-control your prompts. Prompts are code, treat them accordingly
• Run shadow tests before migrating. Production traffic reveals what evaluation datasets miss
• Document everything. Model versions, evaluation results, migration decisions. Your future auditor will thank you.

Conclusion, The New Normal

AI model lifecycle management is not a one-time project. It’s a continuous discipline. The provider-managed models you deploy today will be deprecated, and even the model artifacts you own will eventually need reevaluation, migration, or retirement. The question is whether you’ve built systems that can evolve gracefully or systems that will break suddenly.

The good news: the tooling is maturing fast. Bedrock’s lifecycle states, Converse API, and Intelligent Prompt Routing provide the infrastructure layer. SageMaker’s Model Registry and staging constructs provide the governance layer. Bedrock Evaluations provides the quality assurance layer. The pieces are there.

The remaining challenge is as much organizational as technical: wiring these pieces into your stack is real engineering work, and it requires treating model lifecycle management as a first-class concern from day one, not as an afterthought when the deprecation email arrives.

As I wrote in the first article: software systems are more like plants than stones. AI models? They’re more like cut flowers. Beautiful, powerful, and temporary.

What’s your approach to model lifecycle management? Are you already running automated evaluations, or still in “vibe testing” mode? I’d love to hear your experiences. Drop a comment or reach out directly.

Sources

[1] IT System Maintenance in the Age of AI: https://schristoph.online/blog/it-system-maintenance-in-the-age-of-ai/

[2] Advanced AI Cost Optimization Strategies: https://www.aicosts.ai/blog/advanced-ai-cost-optimization-strategies-2025-enterprise-guide

[3] Amazon Bedrock Model Lifecycle: https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle.html

[4] AWS Prescriptive Guidance, Prompt, Agent, and Model Lifecycle Management: https://docs.aws.amazon.com/prescriptive-guidance/latest/agentic-ai-serverless/prompt-agent-and-model.html

[5] Amazon Bedrock Converse API: https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html

[6] Model Fallback Chains: https://www.operion.io/learn/component/model-fallback-chains

[7] Salesforce Agentforce Failover Design: https://www.salesforce.com/blog/failover-design/

[8] Amazon Bedrock Intelligent Prompt Routing: https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-routing.html

[9] Going Beyond Vibes, Evaluating Bedrock Workloads for Production: https://aws.amazon.com/blogs/publicsector/going-beyond-vibes-evaluating-your-amazon-bedrock-workloads-for-production/

[10] LLM-as-a-Judge on Amazon Bedrock: https://aws.amazon.com/blogs/machine-learning/llm-as-a-judge-on-amazon-bedrock-model-evaluation/

[11] Bedrock Evaluations GA with BYOI: https://aws.amazon.com/blogs/machine-learning/evaluate-models-or-rag-systems-using-amazon-bedrock-evaluations-now-generally-available/

[12] Migrate from Claude 3.5 Sonnet to Claude 4 Sonnet on Amazon Bedrock: https://aws.amazon.com/blogs/machine-learning/migrate-from-anthropics-claude-3-5-sonnet-to-claude-4-sonnet-on-amazon-bedrock/

[13] Evaluate Amazon Bedrock Agents with Ragas and LLM-as-a-Judge: https://aws.amazon.com/blogs/machine-learning/evaluate-amazon-bedrock-agents-with-ragas-and-llm-as-a-judge/

[14] Evaluating Agent Output Quality, Lightweight Evals Without a Framework: https://builder.aws.com/content/3ARUgrl7zf386B8vucsOyASQRty/evaluating-agent-output-quality-lightweight-evals-without-a-framework

[15] SageMaker Model Registry Staging Construct: https://docs.aws.amazon.com/sagemaker/latest/dg/model-registry-staging-construct.html

[16] EU AI Act, Regulation (EU) 2024/1689, Article 12 (record-keeping) and Article 19 (automatically generated logs): https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-19

[17] ISO/IEC 42001:2023 Information technology, Artificial intelligence, Management system (ISO): https://www.iso.org/standard/42001

[18] AI Governance Regulations 2025-2026: https://learn.g2.com/ai-regulations

[19] Required CORS Permissions for Bedrock Evaluations: https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-security-cors.html

[20] Bedrock Evaluation Prompt Datasets: https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-prompt-datasets.html

[21] Anthropic, Model deprecations (Claude Sonnet 3.5 deprecation notice, August 13, 2025): https://platform.claude.com/docs/en/about-claude/model-deprecations

About the Author

Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.

This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.

Learn more →

• AI
• ModelLifecycle
• Evaluation
• AmazonBedrock
• SageMaker
• Architecture