RAG - just a poor engineering workaround?

RAG - just a poor engineering workaround?

My week kicked off nicely with some inspiring talks on an internal conference. In one of the talks Johannes Langer dived deep on how to build production-ready RAG systems. I answered his opening questions to the audience - “What is RAG?” - with “𝗞𝗼𝗳𝗳𝗲𝗿𝗸𝗹𝗮𝘂𝘀𝘂𝗿”, which translates to 𝗼𝗽𝗲𝗻 𝗯𝗼𝗼𝗸 𝘁𝗲𝘀𝘁 in my head.

Thinking more about this analogy, I find it is helpful to approach the question if RAG is just a workaround to overcome limitations of our current foundation models or is here to stay, one a more conceptual level. The German wikipedia article on “𝗞𝗼𝗳𝗳𝗲𝗿𝗸𝗹𝗮𝘂𝘀𝘂𝗿” talks about some of the motivations for this kind of test: huge efforts for students on memorising independent facts are eliminated, the test scope can be wider and the test is focussing more on the ability to creatively think and find new solutions approaches. In other words this approach is frugal with students resources and incentives creation of new solutions.

From my perspective this is very analogous to foundation models, which we are using today in 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 AI. Similar to students those need some knowledge to be able to generate answers, but there is no reason to limit those to just their internal knowledge if there are other, more efficient, system component to provide them with additional knowledge. I really think that RAG is enabling a reasonable logical decomposition into using, primarily, Foundation models to their strengths and knowledge retrieval systems as their support. The famous Swiss Army knife, while a very handy tool in some situations, is usually beaten by special purpose tools. Hence the focus in today’s system design to find the best fitting components for a use case instead of using just multi-purpose tools.

Therefore I’m really excited to see ongoing innovations in retrieval systems to improve their accuracy. Recently Anthropic released their Contextual Retrieval, which improves RAG architecture by adding additional context information which can be used to retrieve better matches to the actual queries. If you haven’t seen it yet, I recommend to have a look!

Contextual Retrieval: https://www.anthropic.com/news/contextual-retrieval “Kofferklausur” on wikipedia: https://de.wikipedia.org/wiki/Kofferklausur#:~:text=Diese%20Form%20der%20Pr%C3%BCfung%20kann,oder%20zusammenhanglosen%20Fakten%20entf%C3%A4llt%20damit.

• AI
• Architecture
• SoftwareEngineering