This paper proposes HyperPrompt, a novel architecture for prompt-based task-conditioning of self-attention in Transformers.

HyperPrompt allows the network to learn task-specific feature maps where the hyper-prompts serve as task global memories for the queries to attend to, at the same time enabling flexible information sharing among tasks.

HyperPrompt is competitive against strong multi-task learning baselines with as few as 0.14% of additional task-conditioning parameters, achieving great parameter and computational efficiency.

Powerpoint for this talk

Powerpoint for this talk

Reference Paper

• HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning
• Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks
• Parameter-Efficient Transfer Learning for NLP