Training Language Models to Follow Instructions with Human Feedback

Abstract

Making language models bigger does not inherently make them better at following a user's intent. We show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback (RLHF). Our resulting model, InstructGPT, produces outputs preferred by humans over GPT-3 despite being 100x smaller in parameter count.

Key Findings

• 1Demonstrated RLHF as an effective method for aligning LLMs with human preferences
• 2Showed that a 1.3B InstructGPT model was preferred over the 175B GPT-3
• 3Used three-step process: supervised fine-tuning, reward model training, PPO optimization
• 4Reduced toxic and untruthful outputs significantly compared to base models
• 5Established the paradigm used to train ChatGPT and subsequent assistants

Impact & Significance