Most existing retrieval-augmented LMs employ a retrieve-and-generate setup that only retrieves information once based on the input.

This is limiting, however, in more general scenarios involving generation of long texts, where continually gathering information throughout the generation process is essential.

This paper propose Forward-Looking Active REtrieval augmented generation (FLARE), a generic retrieval-augmented generation method which iteratively uses a prediction of the upcoming sentence to anticipate future content, which is then utilized as a query to retrieve relevant documents to regenerate the sentence if it contains low-confidence tokens.

Powerpoint for this talk

Reference Paper

• Measuring and Narrowing the Compositionality Gap in Language Models
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
• Training language models to follow instructions with human feedback
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks