Foundation Models, Retrieval and Grounding

This chapter explains the architectural choices that surround foundation models: whether to build, buy, or adapt; which obligations attach to general-purpose AI models under the AI Act; how retrieval-augmented generation (RAG) keeps a model's output tied to the organisation's own sources and where it fails; and how chunking, embeddings, vector indexes, prompts, and context windows become configuration that is versioned, evaluated, and costed.

Bommasani and colleagues introduced the term foundation model for a model trained on broad data at scale that can be adapted to a wide range of downstream tasks, and identified two architecturally significant properties: emergence, in which capabilities appear that were not explicitly designed, and homogenisation, in which many downstream systems depend on a few upstream models (46, Bommasani et al., 2021). Both are risks as much as opportunities, because emergent capabilities have not been evaluated and a defect in one model propagates to every system built on it.

The large language models (LLMs) of current practice are transformers, in which self-attention lets every token attend to every other token (45, Vaswani et al., 2017). Two consequences reach the cost model. Attention cost grows with the square of the sequence length, so long contexts are expensive. And the model has no memory between requests, so the organisation's own documents must either be trained into the weights or supplied in the context at request time. Most large organisations end up with the second, the retrieval path, for reasons as much regulatory as technical.

Primary topics explored

1. What a foundation model is and why the architect cares
2. Build, buy, or adapt
3. General-purpose AI model obligations under the AI Act
4. Retrieval-augmented generation
5. Grounding, citations, and hallucination management
6. Chunking, embeddings, and vector indexes