The ones who joined Philipp and me in our session at the #AWSSUMMIT in Berlin last week already got

The ones who joined Philipp and me in our session at the #AWSSUMMIT in Berlin last week already got a preview of the blog post as we run it as a demo. Nice that it is now published and you all can get hands on it. Kudos!

AWS Inferentia2 is a great way to optimize the inference part of your (gen)AI workloads on AWS for price-performance — cost per inference in particular — and the blog post helps you to dive straight into deploying a LLM (in this case Meta’s Llama 3) model. But it is not “just” Llama 3. From Hugging Face recent blog post: “Enabling over 100,000 models on AWS Inferentia2 with Amazon SageMaker” - https://huggingface.co/blog/inferentia-inference-endpoints. So there is a good chance that your model is supported on AWS Inferentia 2 today — whether it also wins on price-performance for your workload is something to benchmark :)

As we also discussed in out talk, inference is just one side of the coin. If you after training foundation models on AWS, have a look into AWS Trainium.