The Bitter Lesson
Why injecting human knowledge into AI systems ultimately limits them, and why raw compute always wins in the long run.
In 2019, researcher Rich Sutton published an essay titled "The Bitter Lesson." Its premise was simple but controversial: the history of AI research shows that leveraging massive computation is vastly more effective than attempting to build human knowledge and heuristics into algorithms.
Compute Over Cleverness
Early computer chess programs tried to code human chess strategies (e.g., "control the center," "knight outposts"). They were eventually crushed by Deep Blue, which relied primarily on massive search. Similarly, in computer vision, hand-crafted edge detectors (like SIFT) were obliterated by Deep Learning models that learned their own features directly from pixels.
The most powerful AI systems do not think like humans; they leverage architectures (like Transformers or MCTS) that scale efficiently with compute to discover their own alien logic.
| Era | Approach | Result |
|---|---|---|
| 1980s-1990s | Expert Systems (Hand-coded logic) | Brittle, failed to scale |
| 2010s | Deep Learning (Learned features) | Surpassed human vision/speech |
| 2020s | Foundation Models (Unsupervised scale) | Emergent zero-shot reasoning |