GPT-4: Sparks of AGI
Is it merely predicting the next token, or is a world model coalescing in the weights? The paper that forced a re-evaluation of stochastic parrots.
In March 2023, Microsoft Research published a paper titled "Sparks of Artificial General Intelligence: Early experiments with GPT-4". Unlike previous analyses of large language models, this paper focused on zero-shot reasoning—tasks the model was never explicitly trained to perform, yet could execute flawlessly on the first attempt.
The "Move 37" moment for GPT-4 wasn't a single output, but a pervasive, unprompted ability to map concepts across entirely different domains.
When asked to write a proof of the infinite primes theorem using a poetic style wherein every line must start with a specific letter, GPT-4 did not just pattern-match text; it balanced the rigorous logic of mathematical proofs against strict syntactical constraints simultaneously.
Emergence vs. Memorization
The primary debate surrounding LLMs is whether their capabilities are truly emergent or simply the result of memorizing vast swaths of the internet.
To test this, researchers created novel tasks that could not possibly exist in the training data. For example: asking the model to write code in a made-up language (defined only in the prompt), or asking it to draw a unicorn in TikZ (a LaTeX drawing package) while manipulating the spatial coordinates blindly.
| Capability | GPT-3.5 (Baseline) | GPT-4 (Emergent) |
|---|---|---|
| Spatial Reasoning (TikZ) | Fails entirely; disjointed shapes | Generates a recognizable unicorn |
| Theory of Mind | Fails complex nested scenarios | Passes 90%+ of false-belief tasks |
| Tool Use | Requires explicit prompting | Spontaneously invents API calls |
The World Model Hypothesis
To predict the next token accurately across the entire internet, an objective function must eventually stop relying on statistical n-grams. The most efficient way to compress the data of the internet is to learn the underlying rules of reality that generated that text. This is the World Model Hypothesis.
If the model understands physics, logic, and human psychology, it isn't memorizing text; it is running a simulation of the world inside its weights to determine what word comes next.