Research note · updated 21 October 2026
What language model can you actually train on an 8 GB RTX 5060 Ti?
One card, one month, no cloud. The answer depends on three numbers — how fast the card processes tokens, how much memory each parameter costs, and how many tokens a model needs per parameter to be worth training at all. Everything below is derived from published work plus the card's specification. Nothing was measured on the card.
The short answer
250–350M parameters for a week. 250–600M for a month. About twenty training tokens per parameter.
On this card a week buys roughly seven billion tokens; twenty tokens per parameter turns that into a ~250–350M model. A month buys 25–30 billion tokens — enough for a 500–600M model at the optimal ratio, or a 250M model trained three to five times over, which is what the best published small models do.
What not to do: 100M parameters trained for thirty days. That is over a thousand tokens per parameter, fifty times past the point where extra data helps. The first few days are useful; the other four weeks are not.
- Time buys tokens; you choose how to spend them. This card makes about 12,000 tokens/s on a 250M model and 2,000 tokens/s on a 1B model, so a month is ~3×1010 tokens at the first size and ~5×109 at the second.
- Tokens per parameter is the dial that matters. Twenty per parameter is the compute-optimal point at this scale; ten to a hundred is the range that published small models occupy. Below ten, the model is starved.
- Memory, not time, sets the ceiling. AdamW costs 16 bytes per parameter before activations and stops at ~430M in 8 GB. 8-bit or Muon state reaches 860M–1.3B, which is what makes the larger plans possible.
Four things the research says
How far a number of days can carry a model
Each line is a model size; further right means more training time and therefore more tokens. The green band is the compute-optimal zone (10–35 tokens per parameter), the amber band is deliberate overtraining. Where a line leaves the bands, extra days stop paying — and a line that runs off the top has been given more data than it can use.
All rows use 8-bit AdamW (8 bytes of state per parameter) so that memory does not decide the comparison; the "memory" column is what the same model would need with that recipe. Throughput is the model in Method.
What to run, by budget
Pick the time you actually have. The plan is the largest model that can still be trained to a sensible tokens-per-parameter ratio, with a recipe that leaves headroom in memory.
Tokens per second, by model size
These are the raw rates the rest of the page uses. Efficiency falls with depth because narrow matrices cannot keep the tensor cores fed on a card with 448 GB/s of memory bandwidth.
Memory is the real limit
Training needs far more than the weights. With stock AdamW each parameter costs 16 bytes — weights, gradients, an FP32 master copy and two optimiser moments — before any activations. The optimiser is the single biggest lever on how large a model this card can hold at all.
Evidence
Published small models: what "good" looks like
Optimisers, honestly
Speed ratios: Wen et al. 2025, Fantastic Pretraining Optimizers. 8-bit state: Dettmers et al. 2021. 8-bit Muon: arXiv:2509.23106.
What to train on
Four things people believe that the numbers contradict
Method, and how to argue with it
What was done
- Assembled the evidence: hardware specification, scaling-law papers, published small models, optimiser benchmarks, repetition studies and dataset cards — all linked.
- Wrote the arithmetic once, in
lib/core.js, and used it for every number on the page. - Argued every plan by hand against three constraints: memory, tokens per parameter, and what the throughput model allows in the time.
- Ran no training. There is no GPU behind this page.
Labelling
SPEC manufacturer or dataset specification · PAPER published result · CALC our arithmetic from a stated formula · MODEL our assumption about hardware behaviour · EST judgement.
What would change the answer
- A measured tokens/s for one model on your card. The MFU model is the largest source of error; everything time-based moves with it.
- Better-than-claimed efficiency from a framework (FP8 training, fused kernels). That would scale all plans up, not change their shape.
- Evidence that very long training on small models beats the repetition decay measured so far. None exists that would justify 1,000 tokens/parameter.
The equations
- Training FLOPs
- C ≈ 6 · N · D PAPER
- Achieved FLOP/s
- peak × MFU, with MFU = 0.40 − 0.045·√L, clamped to 15–34% MODEL
- Tokens in time
- D = achieved FLOP/s × seconds ÷ 6N
- Memory
- N × bytes/parameter + activations (≈1.2 GB with checkpointing) + 0.35 GB framework
- Parameters
- N = L(2d² + 2d·kv·64 + 3d·ffn + 2d) + V·d, with ffn ≈ 2.67d, d ≈ 50·L, V = 32k, tied embeddings CALC
- Tokens per parameter
- D ÷ N, compared with Chinchilla's ≈20 PAPER
Where it breaks
Chinchilla's constants were fitted on models from 70M to 16B trained on books, code and Wikipedia. A small model on filtered web text will be a little better or worse than the fit predicts, and the fit does not know about data quality, duplicates or curriculum. Treat every "tokens per parameter" verdict as a direction, not a forecast.
Deliberately not on this page
Loss or accuracy predictions for a training run, benchmark scores for a model that does not exist, and "tokens/s" figures presented as measurements. If you want those, run the plan and measure them.