What happens in Las Vegas ... Nah - let's have a look. All things (Gen) AI.

written by Stefan Christoph

December 3, 2024 - 20 minutes read

What happens in Las Vegas … Nah - let’s have a look. All things (Gen) AI.

![](assets/reinvent-las-vegas-vltke-cover.jpg)

What happens in Las Vegas … Nah - let’s have a look. All things (Gen) AI.

Created on 2024-12-03 06:38

Published on 2024-12-03 07:46

At the time of starting this article, the first day of AWS re:Invent is over. AWS re:invent is the cloud computing conference, hosted annually by Amazon Web Services (AWS) in Las Vegas, Nevada. The 13th annual event takes place December 2-6, 2024. This year I’m not on-site, but still curious what it is happening there. Plan for the week is to update this article on a daily basis with the new things announced there. Let’s see how this goes.

I turned out to be a very challenging but good learning experience. Although is it pretty hard to follow all those updates - even just on one single topic like GenAI. For me it turned out to be useful way of absorbing the information, I hope you as a ready get some benefit from this as well.

***Disclaimer: I’m writing this over my morning coffee and didn’t had the time to dive in every detail of the new announced features and service. I’m just taking you with me on me personal discovery of the new announcements and hope this turns out useful. ***

Prompt Engineering

Prompting Foundation models is an art and engineering at the same time. The process of designing and optimising inputs to generate specific responses from AI models, particularly Large Language Models (LLMs) turns out to be cumbersome. It’s a crucial skill for effectively communicating with and getting the best results from AI systems. Hence the try to apply engineering practices to create reliable results. The so called prompt engineering. It’s one of the most important skills which help customers to apply Gen AI successfully in production. I’ve been talking about this a number of times during this year, including a recorded session at AWS Summit Stockholm 2024: “Prompt engineering best practices for LLMs on Amazon Bedrock” (https://www.youtube.com/watch?v=W8jd50-_8oQ). While this is all valid, the journey of improving the way we prompt models continued. Back in May I gave a talk at the first edition of the Generative AI Deep Dive Days in Munich on “From Prompt Engineering to Auto- Prompt Optimisation” exploring the use of LLMs to optimise prompts. Sounds like an awkward circular dependency, but turns out to be very powerful. In the end prompts are just text - something which LLMs are good in producing.

Prompt Optimisation

*Now in Las Vegas, AWS launched a Prompt Optimisation Feature within the Amazon Bedrock service. In https://aws.amazon.com/blogs/machine-learning/improve-the-performance-of-your-generative-ai-applications-with-prompt-optimization-on-amazon-bedrock/ you can find an extensive write-up. Exciting to have Prompt Optimisation at you fingertips as part of a managed service. *

Prompt Optimisation in Action right in the Console. Maybe hard to read, but the optimise generates a more detailed prompt and more reliable and parsable results.

Retrieval Augmented Generation

Retrieval Augmented Generation (RAG) is a technique that enhances generative AI models by supplementing their responses with information from external data sources. It combines a retrieval model that searches databases or knowledge bases with a generation model (typically a Large Language Model) to produce more accurate and context-aware responses.

RAG turned out to be the most successful architecture style being adopted in production use cases. By grounding the model in the provided data, customers can minimise hallucinations without the need to invest in training their own models. Building RAG systems is a fairly complex. Good insights in the complexity provides the article “Süddeutsche Zeitung’s AI-powered review of the year with retrieval-augmented generation (RAG)”(https://medium.com/s%C3%BCddeutsche-zeitung-digitale-medien/qa-roy-5e018b7f08e5), which showcased how they successfully build a RAG system from the ground up. Amazon Bedrock later offered a managed RAG solution called Amazon Bedrock Knowledge Bases, which eases the effort to build RAG systems dramatically. One of the exemplary adoption of the service is showcased in another, later, project of Süddeutsche Zeitung in “An AI Assistant for EU Election Q&A”(https://medium.com/s%C3%BCddeutsche-zeitung-digitale-medien/ai-assistant-for-eu-election-q-a-e8e4b62c2751*). *

Amazon Bedrock Knowledge Bases now offers automatic knowledge base evaluation for RAG applications. This feature uses LLMs to compute evaluation metrics, allows comparison of different configurations and enables optimisation of settings for specific use cases. This will further ease the creation of RAG based applications. You can finde an extensive write-up at https://aws.amazon.com/blogs/aws/new-rag-evaluation-and-llm-as-a-judge-capabilities-in-amazon-bedrock/. This feature is right now in preview.

Additional new features in Amazon Bedrock Knowledge Bases are:

Amazon Bedrock Knowledge Bases has launched automatically-generated query filters that improve retrieval accuracy by intelligently filtering search results based on document metadata without requiring manual filter construction: https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-bedrock-knowledge-bases-auto-generated-query-filters-improved-retrieval

Amazon Bedrock Knowledge Bases has introduced support for custom connectors and streaming data ingestion, enabling direct API-based updates to knowledge bases without requiring full syncs or intermediary storage https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-bedrock-knowledge-bases-connectors-ingestion-data/

Amazon Bedrock Knowledge Bases has introduced the RetrieveAndGenerateStream API, enabling streaming responses from LLMs during RAG workflows to reduce latency and provide a more responsive user experience: https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-bedrock-knowledge-bases-streaming-retrieveandgeneratestream-api/

Amazon Bedrock has introduced support for reranker models through the Rerank API, which improves RAG application accuracy by reordering retrieved documents based on their relevance to user queries before sending them to foundation models: https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-bedrock-rerank-api-accuracy-rag-applications/

Structured-data-Retrieval: Amazon Bedrock Knowledge Bases has introduced natural language querying capabilities that can convert user questions into SQL queries, enabling direct structured data retrieval from sources like Amazon Redshift and SageMaker Lakehouse without data preprocessing: https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-bedrock-knowledge-bases-structured-data-retrieval/. In other words so far your carefully structured data has been fairly inaccessible for your RAG systems. This new feature translate the user’s natural language query to SQL and hence unlocks the valuable data being stored in your databases.* *

Wait, not all my nicely structured data actually best fits in to a sort of relation database. What about data which can be naturally structured in Graphs. Another announcement got you covered! Amazon Bedrock Knowledge Bases now supports GraphRAGModel Selection & Optimisation. Amazon Bedrock Knowledge Bases now supports GraphRAG in preview, combining RAG techniques with graph data to provide more comprehensive and relevant responses by leveraging relationships between entities across documents, enabling complex multi-step queries like finding family-friendly beach destinations with specific amenities. More details can be found in https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-bedrock-knowledge-bases-graphrag-preview/.

What about non-structured data? Preparing valuable data, which is hidden in some documents like PDF or images and videos, to be accessible for a RAG application requires a lot of effort and engineering. Here **Amazon Bedrock Data Automation **can be very useful. Amazon Bedrock Data Automation, now available in preview, enables developers to automatically generate insights from unstructured multimodal content, streamlining the creation of GenAI applications with features like video summaries, image content analysis, and document processing, while offering high accuracy, cost-effectiveness, and integration with Bedrock Knowledge Bases: https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-bedrock-data-automation-available-preview/. While all of this has been possible before, the new feature will likely take away a lot of undifferentiated heavy lifting. Give it a try but note again that his is currently in preview.

If you want to build a successful Gen AI application, you need to make sure that you select the right model for the use case. It needs to be powerful enough for what you intend to build, but should be also be cost efficient to ensure that you really can put your idea into production. Lately we discussed a lot about the greatest and latest frontier models (we call those now also state-of-the-art (SOTA)) and forgot another crucial aspect: A Generative AI Application is so much mode than just the model. I wrote back in June about all of this in more length in a medium article “How do you choose the foundation model for your Generative AI App — like your car?” (https://medium.com/p/18f8a299678c).

So we need a good menu of different models and the ability to understand which of those is the best one to use for the use case at hand. Let’s dive deeper into the relevant announcements.

Model Evaluation

Stefan, which model should I use for my use case?" is one of the most frequently asked questions by my customers. Obviously I can - and actually need to - answer with the good old consultant standard: “It depends”. But is really does depend on the individually use case and needs to be evaluated case by case. Especially if you want to optimise your system for performance and cost.

Similar to the topic of prompting, LLMs can provide a good help to evaluate models using LLMs as a judge. In the talk “From Prompt Engineering to Auto- Prompt Optimisation” we also touched on this topic.

Amazon Bedrock Model Evaluation supports you now in evaluating models utilising the LLM-as-a-judge approach:https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-bedrock-model-evaluation-llm-as-a-judge-preview. This feature is now in preview.

A more extensive write-up can be found at https://aws.amazon.com/blogs/aws/new-rag-evaluation-and-llm-as-a-judge-capabilities-in-amazon-bedrock/ and you will notice that the same concept is used in Amazon Bedrock Knowledge Bases.

Additional Models

Amazon Nova model family

New Foundation Model Amazon Nova available in Amazon Bedrock available as the latest addition to the extensive selection of 1st party, so provided by Amazon, and 3rd party models provided by leading model providers. Overall the Amazon Nova Model Family comes in 6 different flavors spread in two categories:

The second category is Amazon Nova content generation Models with two models: Amazon Nova Canvas and Amazon Nova Reel are creative content generation models that accept text and image inputs and produce image or video outputs. They are designed to deliver customizable high-quality images and videos for visual content generation. Content Generation models will be super useful for many customers. Loads of use cases for Media&Entertainment and Advertising&Marketing Customers come to mind. Can’t way to experiment more with those models and work with my customers to build on top.

Are those models any good? The benchmark results published on the product page(https://aws.amazon.com/ai/generative-ai/nova/) look very impressive. Like always you need to evaluate against your individual use case, but for sure worth a deeper look.

In the meantime the Launch presentation on Amazon nova has been published on YouTube. A good start to dive deeper: (AIM398-NEW) Practical generative AI using Amazon Nova

While writing this, I took a brief look into the updated Amazon Bedrock Console. You can get first experiences right in the Image/Video playground.

First test of Amazon Nova Reel in Console

It delivers good results. Note that I was lazy and hence the input picture is not rightly scaled. Like always you need to take good care of the input to create great output. Quick first Reel generated by Amazon Nova Reel

Amazon Bedrock Marketplace

Having the powerful and large new model family as 1st party models available in Amazon Bedrock is already a game changer, but as mentioned earlier, flexibility of choice of the right model for your use case is key. And well, the choice of models just dramatically increased: Amazon Bedrock Marketplace.

Amazon Bedrock Marketplace now offers over 100 foundation models through unified APIs, allowing developers to deploy specialized models on SageMaker endpoints and integrate them with Bedrock’s tools like Agents, Knowledge Bases, and Guardrails across 14 global regions: https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-bedrock-marketplace-100-models-bedrock/.

A marketplace is a proven mechanism which facilitates transactions between multiple buyers and sellers, providing automated processing, payment handling, and product distribution services in a centralized e-commerce environment. With Amazon Bedrock Marketplace you get access to a broad range of 3rd party models. Those models are available via the same Amazon Bedrock API which facilitates an easy change from non-marketplace models to models from the marketplace and allows you to benefit from the strong benefits the Amazon Bedrock Gen AI Developer Platform provides. Neat. Models from the Marketplace are deployed in your own cloud infrastructure. This is very beneficial for customers who navigate complex data privacy requirements. It also implies a different cost model as those are not server-less offerings.

High highly recommend to head over to the detailed AWS News Blog(https://aws.amazon.com/blogs/aws/amazon-bedrock-marketplace-access-over-100-foundation-models-in-one-place/) for all the details and start experimenting with those models.

What is next?

I’m personally generally not a big fan of things being announced, which are not yet available to fiddle around with as it breaks my natural curiosity. Nevertheless I’m excited with the upcoming availability of** Luma AI Models** in Amazon Bedrock. Again diversity is key and allows not just M&E customers to use even more video generating capabilities. Read more details in https://press.aboutamazon.com/aws/2024/12/luma-ai-announces-new-video-model-ray-2-will-soon-be-available-to-consumers-professionals-and-developers.

Model Optimisation

As discussed above, using the right model for your use case is crucial for performance of your application, the cost to run the application and last but not least to optimize the resource consumption of your application. So it makes sense to optimize this for sure. But what if you want to optimize beyond selection from the list of supported Foundation Models and Sizes? Model Distillation to the rescue!

Model distillation is a machine learning technique that transfers knowledge from a large, complex model (teacher) to a smaller, more efficient model (student). This process enables the deployment of powerful AI capabilities while reducing computational demands and costs.

With **Amazon Bedrock Model Distillation **which is now available in preview you can now distill your own student models without managing all the nitty gritty details of the required process and instead have this done be the managed service.

You find all the details and a run-through in the Blog article(https://aws.amazon.com/blogs/aws/build-faster-more-cost-efficient-highly-accurate-models-with-amazon-bedrock-model-distillation-preview/).

Agents

Gen AI agents can be characterized as autonomous software systems that combine large language models (LLMs) with additional capabilities to perform complex tasks and achieve specific goals without constant human intervention. 2024 has been forecasted as the year of GenAI agents and the forecast holds for 2025. While we can discuss - and maybe should do in another article about agents versus agentic behavior versus intelligence - what exactly agents are, they turn out to be super helpful to compose more helpful and complex Gen AI systems. Amazon Bedrock Agents is a service that enables the creation of AI-powered applications capable of executing complex, multi-step tasks across company systems and data sources.

New released feature:Amazon Bedrock Agents now offers custom orchestration capabilities that allow developers to use AWS Lambda to define and control how agents handle multi step tasks, make decisions, and execute complex workflows using strategies like Plan and Solve, Tree of Thought, and Standard Operating Procedures https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-bedrock-agents-custom-orchestration/.

What if one agent is not enough? This could be because one single agent is not capable to solve the task, the agent becomes to complex to manage or you want to decompose your agent for reuse. There is another new feature in Amazon Bedrock Agents: multi-agent collaboration. Amazon Bedrock multi-agent collaboration enables developers to build, deploy, and manage multiple specialized agents working together seamlessly to address increasingly complex business workflows. Each agent focuses on specific tasks under the coordination of a supervisor agent, which breaks down intricate processes into manageable steps to ensure precision and reliability. By automating these complex operational processes, businesses can free their teams from operational burdens, allowing them to focus on innovation and deliver real business value. In the blog article(https://aws.amazon.com/blogs/aws/introducing-multi-agent-collaboration-capability-for-amazon-bedrock/) you find a good run-through this new feature on an example of a Social Media Campaign Manager agent. Have a look and think about your own agentic workflows.

Security

Automatic Reasoning Checks feature of Amazon Bedrock Guardrails

One of the crucial aspects of security of your Gen AI application is that you take control of what data is used as a input to the models your are using and also to control what kind of output you provide back to your users. This can be a fairly complex tasks, especially if you want to implement the control across different models and have the ability to change the model being used in your application without big re-implementation effort.

Amazon Bedrock Guardrails provides configurable safeguards to help safely build generative AI applications at scale. With a consistent and standard approach used across all supported foundation models (FMs). What I really like that I can use this feature across any model I might be using. But still - how do I formulate my guardrails so that nothing can slip through? This can be very difficult due to the nature of the problem.

A possible solution is the new Automatic Reasoning Checks feature of Amazon Bedrock Guardrails.Automated Reasoning checks help prevent factual errors from hallucinations using sound mathematical, logic-based algorithmic verification and reasoning processes to verify the information generated by a model, so outputs align with known facts and aren’t based on fabricated or inconsistent data.

Have a deep dive in the blog post (https://aws.amazon.com/blogs/aws/prevent-factual-errors-from-llm-hallucinations-with-mathematically-sound-automated-reasoning-checks-preview/) and improve the security posture of you own Gen AI application!

Amazon Bedrock Guardrails supports multimodal toxicity detection

*Another common concern, not only with Media and Entertainment customers, is to ensure that Gen AI generated images or videos are not harmful to their customers. *

*Amazon Bedrock Guardrails now offers multimodal toxicity detection for image content, enabling organizations to automatically filter harmful or undesirable content across text and images using configurable thresholds in a single solution. This service is in preview at the moment and supports pictures only: *https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-bedrock-guardrails-multimodal-toxicity-detection-image-content-preview/

In the AWS News Blog(https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-bedrock-guardrails-multimodal-toxicity-detection-image-content-preview/*) you find a walk-through of the feature. *

Performance Optimisation for your GenAI Applications

For many use cases it is key to get fast responses from the models who drive your applications. The latency. It might be a user directly waiting for a response or even the feasibility of a batch use case.

Another aspect of performance is** cost**. A to high cost can prohibit the implementation of a cool use case.

Prompt Caching

Amazon Bedrock has introduced prompt caching in preview, a new capability that reduces costs by up to 90% and latency by up to 85% by caching frequently used prompts across multiple API calls for supported models like Claude 3.5 and Nova: https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-bedrock-preview-prompt-caching/ More detailed discussion of this feature can be found below.

Intelligent Prompt Routing

Why always using a very powerful LLM to answer user’s request? Maybe some of the request could easily be answered by a less powerful and likely less costly LLM. Good idea!

Amazon Bedrock’s new Intelligent Prompt Routing feature automatically directs requests between different models within the same family (like Claude 3.5 Sonnet and Claude 3 Haiku) based on prompt complexity, optimizing for both quality and cost with up to 30% cost savings: https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-bedrock-intelligent-prompt-routing-preview/

Both the Amazon Bedrock Intelligent Prompt Routing (wow quite a mouthful) and Amazon Bedrock prompt caching are discussed in great detail in the News Blog Post: https://aws.amazon.com/blogs/aws/reduce-costs-and-latency-with-amazon-bedrock-intelligent-prompt-routing-and-prompt-caching-preview/

Applying those features can truly make a difference for your Gen AI application. Have a ready and try it for your own use case, but be aware of the limitation to a subset of models at this point in time, highlighted in the blog post.

Building Models

After all someone also needs to build those amazing models we all love to apply to our Gen AI applications. While managed services like Amazon Bedrock provide great flexibility to customize existing models for your purpose, building your own models requires a different set of tools. Let’s look into the announcements covering this, quite some of them are within Amazon Sagemaker. Amazon SageMaker is a fully managed cloud-based machine learning platform that enables developers and data scientists to build, train, and deploy ML models at scale. Initially launched on November 29, 2017, it has become one of AWS’s fastest-growing services.

Amazon SageMaker Partner AI Apps

Amazon SageMaker has launched Partner AI Apps, enabling customers to discover, deploy, and use specialized ML and GenAI development applications from partners like Comet, Deepchecks, Fiddler, and Lakera directly within their secure SageMaker environment, eliminating integration complexities and data security concerns(https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-sagemaker-partner-ai-apps/). Building your own model is - still - a fairly complex task. Hence it is crucial to have access to the right set of tools in a single place. This announcement is an important step into this directions.

Further enhancements of SageMaker HyperPod

Amazon SageMaker HyperPod is a specialized service for developing and training large machine learning models that removes infrastructure management complexity while providing resilient, scalable computing environments. Training those models requires large clusters of specialised hardware and various engineering skills. Not an easy task, but SageMaker HyperPod offer a lof of the required technology as a managed service.

Amazon SageMaker HyperPod has introduced** task governance capabilities **that provide centralized control over AI development tasks, allowing administrators to set priorities and resource limits while automatically managing compute resources, reducing model development costs by up to 40% through improved resource utilization(https://aws.amazon.com/about-aws/whats-new/2024/12/task-governance-amazon-sagemaker-hyperpod/).

Amazon SageMaker HyperPod now offers flexible training plans that allow users to schedule and budget AI model training with automated resource management, optimizing costs while automatically handling infrastructure provisioning and workload transitions between compute blocks (https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-sagemaker-hyperpod-flexible-training-plans/).

Amazon SageMaker HyperPod has introduced recipes that enable quick training and fine-tuning of popular foundation models like Llama 3.1 405B and Mixtral 8x22B, providing pre-tested configurations that reduce setup time from weeks to minutes while delivering optimal performance across GPU and AWS Trainium instances(https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-sagemaker-hyperpod-recipes/).

Next Generation of Amazon SageMaker

AWS compiles a number of significant updates and extension of Amazon SageMaker as the next generation(https://aws.amazon.com/about-aws/whats-new/2024/12/next-generation-amazon-sagemaker/). Let’s break this into pieces to better understand what we get from this.

Amazon SageMaker Unified Studio: Amazon SageMaker Unified Studio is a next-generation integrated development environment that combines data and AI tools in a single platform. This is currently in preview.

Amazon SageMaker Lakehouse: Amazon SageMaker Lakehouse is a unified data platform that simplifies analytics and AI development by combining data from multiple sources into a single, accessible environment.

Amazon SageMaker Data and AI Governance: Amazon SageMaker Data and AI Governance is a comprehensive capability that streamlines data and AI asset management across an organization’s lakehouse, models, and applications.

So summarizing this is a evolution step of Amazon SageMaker, which makes model building even more efficient as it provides additional capabilities and integrations into relevant data services. Having the right data in the right format is key to be successful with your model creation. With Amazon Bedrock IDE being part of the Amazon SageMaker Unified Studio, SageMaker Studio becomes also more attractive for Builders building on top of Amazon Bedrock. Having that said, it still holds true that AWS services provide their functionality via APIs, which are conveniently accessible in software development kits(SDKs). Hence you can build and run your application in any development environment of you choice. Again, flexibility is key.

GenAI powered Assistance for Developers

GenAI has emerged as the basis of powerful tools, which can support developers in more and more aspects of their work. Amazon Q Developer is a generative AI-powered assistant that helps developers build, operate, and transform software across the entire development lifecycle.

During this re:invent alone there have been so many updates being announced that this would deserve an entire article. I’m listing some of them here:

· Amazon Q Developer can now guide SageMaker Canvas users through ML development

· Amazon Q Developer now provides transformation capabilities for .NET porting (Preview)

· Amazon Q Developer announces automatic unit test generation to accelerate feature development

· Amazon Q Developer can now automate code reviews

· Amazon Q Developer adds operational investigation capability (Preview)

· Announcing GitLab Duo with Amazon Q (Preview)

· Amazon Q Developer can now generate documentation within your source code

· Amazon Q Developer transformation capabilities for mainframe modernization are now available (Preview)

· Announcing Amazon Q Developer transformation capabilities for VMware (Preview)

· Amazon Q Developer launches Java upgrade transformation CLI (Public Preview)

I will need some more time to try this out, but things like coder reviews and documentation generation sound very tempting.

Sources

Note that all of the above announcements are available via the “What’s New with AWS?” -Blog : https://aws.amazon.com/new/?whats-new-content-all.sort-by=item.additionalFields.postDateTime&whats-new-content-all.sort-order=desc&awsf.whats-new-categories=marketing-marchitecture%23artificial-intelligence So if missed a thing, or omitted on purpose, you can always find the latest status there. Let me know if I missed something important from your point of view.

Final Thoughts

This article got longer than initially anticipated. So I take a chance to remind you of my initial disclaimer:

*** I’m writing this over my morning coffee and didn’t had the time to dive in every detail of the new announced features and service. I’m just taking you with me on me personal discovery of the new announcements and hope this turns out useful.***

My head is now full of things I want to dive deeper into. Still catching up on Werner Vogels keynote and all the interesting stuff outside of the Gen AI bubble.

And now time for a second coffee and dive deeper :)

Have a good one!

📝 Last updated: August 14, 2026 — Restored the article body, which was lost in the LinkedIn migration