This study proposes a fine-tuning recipe for retrieval-augmented generation (RAG) models that combine pre-trained parametric and non-parametric memory for language generation.

By using a pre-trained neural retriever to access a dense vector index of Wikipedia, RAG models outperform parametric seq2seq models and task-specific architectures on knowledge-intensive NLP tasks, including open-domain QA.

RAG models also demonstrate the ability to generate more specific, diverse, and factual language compared to parametric-only seq2seq models.

Powerpoint for this talk

Reference Paper

• Dense Passage Retrieval for Open-Domain Question Answering
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
• Active Retrieval Augmented Generation