The Bitterest Lesson

Doing the right task, then data, then compute, then algorithms — extending Sutton's compute-beats-algorithms lesson. The example is InstructGPT, where GPT-2-sized models over 100x smaller than GPT-3 beat it once trained on the right task.

Roughly GPT-7 level to beat that baseline and GPT-9 to beat InstructGPT on GPT-3, per the essay, which reads the comparison from the original paper's figure 31. It is an argument, not new experimental work.

Its TL;DR is that compute drives progress in AI, but what good is progress if you are not doing the right task. Sutton's bitter lesson says compute beats algorithms: researchers want to encode clever ideas about intelligence, yet the approaches that win are the general ones that leverage more computation, as search beat hand-built chess knowledge and neural networks beat hand-built vision features.

Games differ from the real world in two ways that matter. The right task is obvious, since you follow the rules to win or maximize score, and data can be generated endlessly through self-play, which is itself compute. That is why compute becomes the next most important thing: machine learning makes reward go up or loss go down, but someone still has to decide the objective, which means understanding the external system the model will operate in.

Without the right task everything can work perfectly, with the most beautiful loss and scaling curves, and the model may still be useless. The essay says ML research tends to attack these in the opposite order: researchers love inventing algorithms, have more recently learned to love scaling curves, and meanwhile data is messy.

No, and the essay says so directly. Once the task and data are right, scale is incredible; it is an argument against treating scale as the be-all and end-all. Its practical instruction is that choosing the right task often requires leaving the ML problem entirely to study users, products, organizations, or whatever part of the world is supposed to benefit, and that the most important part of ML is not ML at all.

It is the essay's worked case, learned at OpenAI while making InstructGPT and RLHF. GPT-3 was an incredible model trained to predict the next token on internet text, but people wanted something that followed instructions more than a super-powered autocomplete, so GPT-2-sized models over 100x smaller, trained on the right task and with barely any compute, beat it. The comparison is annotated from figure 31 of the InstructGPT paper.