Nicolas Zucchet – nzucchet@stanford.edu
MLSS Tübingen Tutorial – September 8th 2026
The early successes of large language models relied on two main observations:
Scaling laws are the science behind 1., are what justified the huge investments in compute and are key to the pretraining of current models.
They are interesting from an engineering perspective (useful tools to design models), and from a scientific standpoint (suggest some universal principles underlying learning to be discovered!).
Build intuition on why scaling laws are useful and why they hold.
In modern large language models, there are many training stages; we focus on pretraining.
Context in, next-token distribution out.
At inference: sample, append, repeat.
We review some of the results from (Kaplan et al., 2020) and (Hoffmann et al., 2022) in this part and provide some pointers to extensions.
The variables we care about are:
N, D and C are actually coupled: we will now show that C = 6ND.
A linear layer does y = Wx, where W\in\mathbb{R}^{m\times n} has N=mn parameters.
Learning matrix weights for each new token in the context requires the same number of flops.
For Transformers, matrix multiplications dominate the FLOPs cost (but not memory!), so C = 6ND approximately holds; see (Kaplan et al., 2020) for details.
Side note. This holds for dense Transformers; only the number of activated parameters matters in Mixture of Experts, they increase the number of parameters while keeping FLOPs constant.
How does the loss L evolve as a function of D, N, and C?
A larger model reaches any given loss after fewer tokens.
Performance improves as we scale models up, but this does not look as smooth as what we were promised…
How does the loss L evolve as a function of D, N, and C?
The same runs against compute. The envelope (best loss for a given budget) decreases smoothly with compute.
A compute-optimal model is undertrained: optimal training stops with loss still ~8\% above convergence level.
In traditional deep learning
Problem: we can only train our large language model once; how should we pick the hyperparameters?
We can leverage scaling laws to do that!
We now know what the compute-optimal loss looks like, but not how to achieve it (i.e. how to scale N and D).
Different strategies exist:
All should end up with the same result, but they manipulate the data in different ways; it is great to build intuition and will be useful for Part III, so we review them next following (Hoffmann et al., 2022).
Important note. Learning rates are tailored to each model size, and are close to optimal.
For each model size, compare loss against compute (exactly as we did before, but this time with other data).
All 43 model sizes of (Hoffmann et al., 2022) present in the reconstruction of its Figure 4 by (Besiroglu et al., 2024); the legend names six of them.
At fixed compute budgets, sweep the model size.
One \log L parabola in \log N fitted per compute budget, over all nine budgets of (Hoffmann et al., 2022).
One law for all the runs:
E=1.81, A=476, a=0.35, B=1799, b=0.36
Note. Only this construction models the irreducible loss E, the entropy of the data.
Optimizing the loss under the constraint C=6ND gives N^*\propto C^{0.51} and D^*\propto C^{0.49}.
Sanity check: the three methods gave very similar exponents.
Data from (Hoffmann et al., 2022), fitting from (Besiroglu et al., 2024) (all data points except ones that are too overparametrized).
Look at the Chinchilla law more closely:
As a result, the ratio between the compute-optimal model and dataset size remains roughly constant.
Rule of thumb: 20 tokens per parameter. Way more than what (Kaplan et al., 2020) advocated for before (Besiroglu et al., 2024), and the current trend is towards a larger number for other reasons, e.g. inference cost (Gadre et al., 2025) (Sardana et al., 2024).
Exercise. Derive the optimal model size and dataset size from the parametric fit and justify the claim that we have made here.
So far, we have only discussed two parameters N and D that we can play with.
What about more specific architecture details? e.g., number of layers, width of the model, the number of attention heads…
And optimization details? e.g., batch size, sequence length…
Too expensive to retune everything every time, so there are some choices that everyone is making (e.g., what expansion ratio should an MLP have, ratio width / depth of the network, what learning rates should be etc…)
As a result, scaling laws are recipe-specific (on top of being data specific)!
That said, there are some principles that one could use: compare recipe-specific scaling laws.
How to develop a recipe (model / optimizer / data…) when we only have limited compute? What matters is how good it will be at large scale, but we can only afford small runs.
A single model size cannot tell us: compare scaling laws.
PSA: Only fully trust methods that use baselines with properly tuned learning rates!!
In next-token prediction language modeling, performance (and other observables) increases smoothly as a function of compute.
Scaling laws provide us with a framework to design the best large model possible when we cannot afford to train it multiple times.
This paradigm applies elsewhere, as long as
Such conditions hold in other domains, like vision (Zhai et al., 2022) or time series (Edwards et al., 2024).
We have only scratched the surface of scaling laws…
As large language models are becoming more and more complex and used in different practical scenarios, many questions arise:
To see scaling laws in action: (Hall et al., 2025) has fully public LLM development.
In Part I, we have discussed what scaling laws are and how they are useful in practice.
There are many open questions though:
Goals of this part:
Assumption 1. Language modeling as context to next-token memories
Next-token prediction requires learning a mapping from contexts to next-token probability distributions
The model explicitly represents next-token probability distributions, while the data only provides samples from an implicit target distribution.
Assumption. No structure shortcuts the learning; nothing generalizes from one context to another, so the model has no choice but to memorize the mapping, context by context.
Assumption 1. Language modeling as context to next-token memories
Assumption 2. Sequence models as random embedding generation and prediction
We assume that the embeddings are iid distributed uniformly on the unit sphere.
Assumption 1. Language modeling as context to next-token memories
Assumption 2. Sequence models as random embedding generation and prediction
Assumption 3. Contexts follow a power law distribution
Power laws appear in many places (Newman, 2005). Zipf’s law for word frequencies is the famous example for language: frequency is approximately inverse to rank (Zipf, 1949).
We assume that contexts follow a power law, that is
with i the index of the context.
Assumption 1. Language modeling as context to next-token memories
Assumption 2. Sequence models as random embedding generation and prediction
Assumption 3. Contexts follow a power law distribution
Assumption 4. There is only one likely next token in the data
Let’s keep our life simple, at least for now :)
Assumption 1. Language modeling as context to next-token memories
Assumption 2. Sequence models as random embedding generation and prediction
Assumption 3. Contexts follow a power law distribution
Assumption 4. There is only one likely next token in the data
More formally, we have
Model. p(\cdot | i) = \mathrm{softmax}(We_i), with p(i) \propto i^{-\alpha}. Embedding dimension h, d possible next tokens.
Can we guess (without running experiments!)?
In the following:
Model. p(\cdot | i) = \mathrm{softmax}(We_i), with p(i) \propto i^{-\alpha}. Embedding dimension h, d possible next tokens.
We can get some intuition of the storage capacity with the following Hebbian model
Corresponds to taking one gradient descent step, starting from W=0, on the objective
Contexts with embeddings close to e_i will be pushed towards outputting z_i (and thus to predict the correct next token).
Model. p(\cdot | i) = \mathrm{softmax}(We_i), with p(i) \propto i^{-\alpha}. Embedding dimension h, d possible next tokens.
We can get some intuition of the storage capacity with the following Hebbian model
Predicting the next-token for context i:
The prediction for context i is perturbed by the other embeddings close to e_i.
Hebbian model. W = \sum_i z_i e_i^\top.
Querying. (W e_i)_y = \sum_{j \,:\, y_j = y} e_j^\top e_i: compare the query e_i with all the other embeddings and see which color dominates.
Interference. The prediction for context i is perturbed by the other embeddings close to e_i.
As a result, capacity increases with d and h.
Exercise. Demonstrate that it is proportional to hd up to logarithmic factors.
In theory, capacity is proportional to the number of parameters, up to logarithmic factors. We fix d=256 and vary h.
At large h the trained model is exactly parallel to N (slope 1.01) and stores 0.16 memories per parameter.
Hebbian gets 0.02, an order of magnitude less.
To learn more. (Cabannes et al., 2024) for the full analysis, (Nichani et al., 2025) for other architectures, (Zucchet et al., 2026) for several possible next tokens.
In practice, capacity is also proportional to the number of parameters. For Mixture of Experts, the total number of parameters matters, so they store more knowledge per active parameter.
Stored synthetic facts, 1000 exposures each (Allen-Zhu & Li, 2024)
Random bitstrings memorized (Morris et al., 2025)
Now that we have some idea of how the capacity scales as a function of the model size, how do we get scaling laws?
For the model size: assume infinite data, and that the first contexts below capacity get 0 loss and some value l afterwards.
Now that we have some idea of how the capacity scales as a function of the model size, how do we get scaling laws?
For the model size: assume infinite data, and that the first contexts below capacity get 0 loss and some value l afterwards.
As a result:
Average the per-context loss over how often each context appears.
Contexts below capacity are free; each one above costs l.
The tail sum is set by where it starts, and capacity grows like N.
For the model size: the loss is a step function of the context index, and the step moves right as N grows.
Note that in this case there is no residual entropy and the loss will converge to 0 under infinite compute and data.
For the dataset size: assume instead that the model gets right every context it has seen at least once.
Context i has a high chance to appear in the first D tokens as soon as
Only the first D^{1/\alpha} contexts are ever seen.
Using the same argument as before,
Same tail sum, cut at D^{1/\alpha} instead of the capacity.
We now have intuition for how the loss scales as a function of model size N and data D.
Power laws in the data become power laws in the loss, in both model size N and tokens D.
Whatever the value of \alpha, the bottleneck is the data D.
As \alpha increases and the tail becomes rarer, the data becomes even more of a bottleneck.
Note. Faster loss decay is not necessarily a good thing here, as the long tail takes longer to learn.
If we further assume Chinchilla scaling, we get:
Optimizing under the constraint C=6ND gives
Does this back of the envelope calculation predict what is happening in simulation?
At fixed compute budgets, we sweep the model size and fit a parabola in \log N.
Six IsoFLOP profiles. Learning rates are tuned for each run.
The loss decreases as C^{-0.091} (predicted C^{-0.091}), the model size grows as C^{0.449} (vs. C^{0.455}).
We assumed the Chinchilla scaling and recovered similar compute exponents as in theory, but how good is the full law?
N and data D treated as independent.
Fit on 75 low-compute runs, then test on 4 held-out runs at 8–16\times the fitting budget.
Relative rms error on L-E; E is known here (0).
Additive scaling laws do not extrapolate well.
Idea: couple model size and data.
To learn more. (Videau et al., 2026) on why this functional form makes sense and how it better extrapolates.
How good are the scaling exponents compared to the theory as we change the data distribution, in particular \alpha?
To estimate those coefficients, we train models of a given size until the loss plateaus (model coefficient), and train the largest reasonable model varying the dataset size (data coefficient).
Our super simple theory is not too bad!
Same experiment as before, but only keeping first 2,000 contexts (instead of infinitely many).
The loss starts decreasing as a power law and finishes as an exponential one.
This is the regime deep learning used to be in: with a bounded amount to learn, held-out loss saturates.
But score those same models on the full distribution and the gain largely vanishes: the held-out loss flattens, and stays worse than training on everything.
At the macro level, performance improves smoothly, but at the task level it can suddenly jump with scale: emergent abilities (Wei et al., 2022) (Olsson et al., 2022).
Our toy model can capture some of this behavior!
To learn more. This toy model does not capture emergence through learning, see (Zucchet et al., 2025) for an example of such a toy model.
We modeled language modeling with an associative memory exposed to a power law data distribution.
With simple theory, we were able to accurately predict scaling law exponents.
Scaling power laws naturally arise when the data itself follows power laws.
Warning. We cannot say anything on whether this mechanism is the bottleneck that yields specific scaling behavior in actual LLMs, we can just say how changing the data affects scaling (which would be much harder in practice!).
This is only one possible theory of scaling laws and we do not yet understand which mechanisms yield the scaling laws.
Some cool work in the field:
Problem. We have access to new important data and want to train the best possible (toy) model on it.
Constraints. You don’t know what the model is and can control the number of parameters N, the number of tokens D, as well as the learning rate, and what are the tuning experiments you do. The total amount of compute for both hyperparameter optimization and the final run is fixed.
Your job. Come up with the best model!
Notebook available here, underlying model and data are a black box on purpose.
lab = Lab("me", budget=1e13, rounds=3) # contains task (max 3 rounds), DON'T TOUCH
sweep = Sweep(
c=[4e9, 12e9],
n=[2**(13 + i) for i in range(5)], # fix C+N, C+D, or N+D
lr=[0.001, 0.01, 0.1], # either array, or a function of c, n, d
) # Cartesian product, test all combinations
sweep + Sweep(n=[2**14], d=[2**14], lr=[0.01]) # Sweeps can be added!
sweep.estimate(lab) # estimates FLOPs used by the sweep
round_results = lab.run_round("first round", sweep) # runs the sweep
# Fits are method specific, see notebook
result = lab.hero(n=n, d=d, lr=lr, predicted=loss) # final run!!
From the notebook. lab.hero(...) prints a form link with your three numbers already in it – click it, submit.
Or fill the form yourself.