Growth in frontier training compute

Assembled from 21 blocks · 1 evidence · 25 related

1. Story
2. The most reliably measured trend in the field, and the one most often mistaken for a law. Epoch AI, which maintains a public database of notable models and reconstructs the arithmetic operations used to train them, reports that training compute for frontier models grew by roughly four to five times a year from 2010 to 2024, with frontier language models growing at about five times a year since 2020. That is far faster than semiconductor performance improved over the same period, so most of the increase is not from better chips: it is from spending more, on more accelerators, for longer. Two things follow that are frequently conflated. The trend is a measurement of inputs, not of capability, and the relationship between the two is empirical and imperfectly characterised. And an exponential in spending is bounded by things exponentials in physics are not — capital, electricity supply, grid connections and advanced-node fabrication capacity — which is why the interesting question about the trend is not how long the mathematics permits it to continue but which of those constraints binds first.
3. Knowledge
4. The transformer architecture
5. Growth in frontier training compute
6. Connections
7. The transformer architecture
8. Training compute
9. Data-centre electricity demand
10. Northern Virginia data-centre corridor
11. Epoch AI
12. The IEA publishes its first global analysis of energy and AI
13. The AI accelerator
14. Deep learning
15. Machine translation
16. Machine-generated code
17. Growth in frontier training compute
18. The transformer is presented at NeurIPS
19. GLUE is saturated and SuperGLUE is built to replace it
20. Evidence
21. Training compute for frontier AI models grew by roughly 4–5× per year from 2010 to 2024, with frontier language models growing at about 5× per year since 2020; Epoch also maintains a database of large AI data centres built from satellite imagery and permit filings, with concentrations of covered sites in Texas, Virginia, Ohio, Nebraska and Iowa. V55 verification basis: epoch.ai was not fetched. The 4–5× and 5× growth rates were carried in search results attributing them to Epoch's own published trends analysis of that title, and the data-centre database and its state-level site concentrations were carried with attribution to Epoch. Forward projections of compute growth, coverage percentages for the data-centre database, and GPU-equivalent counts appearing in secondary summaries were NOT confirmed and are excluded from this pack.