Scaling Laws
The terrifying predictability of intelligence: just add more parameters and data.
One of the most profound discoveries in modern AI is that the performance of neural networks follows smooth, predictable power laws based on compute, parameter count, and dataset size. If you know how a 1-billion parameter model performs, you can mathematically predict the loss of a 100-billion parameter model.
Emergence from the Curve
While the overall loss curve is smooth, specific capabilities (like arithmetic, or translating languages) emerge suddenly and sharply at specific compute thresholds. This is known as the phase transition of capabilities.
We are not engineering intelligence; we are excavating it by applying massive amounts of thermodynamic energy to silicon.
| Variable | Impact on Loss |
|---|---|
| Compute (FLOPs) | Power law decrease |
| Dataset Size | Power law decrease |
| Parameters | Power law decrease |
FAQ: Scaling
Will scaling ever stop working?
Eventually, we will run out of high-quality human text data, forcing a shift towards synthetic data or alternative architectures.