At the 2025 Banking Transformation Summit, I explored a pivotal shift in how financial institutions can harness AI in delivering client advice. While the first wave of AI adoption brought us chatbots and copilots, the next phase demands more precision, trust, and enterprise alignment.

This is where Retrieval-Augmented Generation (RAG) comes in. It’s a core architectural building block for LLM-based applications that bridges the gap between the specific needs of financial services, particularly when dealing with client-related information.

In this article, I’ll share the key takeaways from my talk at the Summit and explain why RAG is emerging as a foundational approach for AI in financial services.

Public Large Language Models (LLMs) like ChatGPT, Copilot, and Gemini have demonstrated impressive capabilities in natural language understanding and generation. However, their limitations become starkly apparent in regulated, data-sensitive environments like banking and insurance:

• Accuracy: LLMs often hallucinate, generating plausible but incorrect answers due to a lack of context or training data quality.
• Confidentiality: Sensitive client data cannot be used to train public models, limiting their relevance in real-world scenarios.
• Currency: Updating LLMs with new information is computationally expensive and infrequent, making them ill-suited for fast-moving domains.
• Provenance: These models are black boxes, offering no transparency into how answers are derived.
• Numeracy: Handling of tabular, numeric, or historic data remains a weak spot of large language models.
• Localisation: LLMs are trained on English-language documents from outside the regions where financial institutions are based. This leads to poor responses in non-English languages or when cultural context is important.

To truly transform client advice functions, we must move beyond generic tools and build AI systems that are context-aware, secure, and explainable.

RAG: A smarter way to create value in client advisory services

RAG is a hybrid AI architecture that combines the generative power of LLMs with the precision of enterprise data. Here’s how it works:

1. Internal databases are embedded with documents, such as research notes, policy terms, client records, and regulatory texts.
2. Semantic search retrieves the most relevant content based on meaning, not just keywords.
3. Prompt engineering and model grounding ensure responses are accurate, relevant, and traceable.

This approach enables AI agents to answer complex client queries with confidence, citing the exact clause or document that supports the answer.

A compelling example of RAG in action is Zühlke’s collaboration with UNIQA, Austria‘s second largest insurer. By piloting an award-winning AI chatbot to support insurance sales, we helped empower customer service teams to do more with less.

To sum up, the evolution of AI in client advisory services isn’t just about efficiency, it’s also about customer experience, accuracy, and compliance.  Achieving this level of performance requires more than prompt-engineering and LLMs alone.

From our experience, RAG-based systems can achieve accuracy of up to 95%, though they still require a human in the loop. If higher accuracy is required, it’s necessary to train dedicated models.

The road ahead

As financial institutions continue to digitise and personalise services, the ability to deliver accurate, timely, and trusted information becomes a competitive differentiator. RAG delivers a scalable, secure, and explainable AI architecture that aligns with both regulatory expectations and customer demands.

At Zühlke, we believe the future of client advice lies in AI systems that are not just smart, but strategic. Systems that empower people, enhance trust, and unlock new value across the customer journey.