7/29 Scaling Transformers and Instruction Tuning
Emergent Features of Bigger Models
In Complexity Theory, we call emergent features of a complex system those that cannot be explained by their parts. We can observe such features as the model sizes grow. Let's look at the largest sizes in the GPT model families by OpenAI.
| name | number of transformer blocks | embedding dimension | feedforward dimension | total parameter count |
|---|---|---|---|---|
| GPT [1] | 12 | 768 | 3072 | 117M |
| GPT-2 [2] | 48 | 1600 | Not Available | 1542M |
| GPT-3 [3] | 96 | 12288 | 49152 | 175B |
| GPT-4 [4] | Not Available | Not Available | Not Available | Not Available |
| GPT-4.5 [5] | Not Available | Not Available | Not Available | Not Available |
| GPT-5 [6] | Not Available | Not Available | Not Available | Not Available |
In GPT-2, it was observed that the models are starting to get nontrivial results in zero- and few-shot learning. This paradigm means that the model is not finetuned in the traditional sense, but only receives a textual description of the task and maybe a few examples.
GPT-3 actually got good at this. Moreover, its text generation capabilities received general media coverage, as you may well know. These gigantic models that have been pretrained on gigantic corpora are called Foundation Models.
Economic and Environmental Costs
Training these models incurs a huge economic and environmental cost. Training GPT-3 is very roughly estimated to have cost $10M https://hai.stanford.edu/news/ai-index-state-ai-13-charts while emitting 85000 kg of CO2 equivalents, the same amount produced by a new car in Europe driving 700,000 km, or 435,000 miles; in contrast, the Earth--Moon distance is about 240,000 miles https://www.theregister.com/2020/11/04/gpt3_carbon_footprint_estimate/.
Instruction Tuning
Given a Text Generation Foundation Model, Instruction Tuning is a finetuning task to create helpful digital assistants. They work by making the model generate multiple answers, and giving feedback on which answer was preferred by the human annotators or an evaluation model.
Reinforcement Learning with Human Feedback
This is the first widely known method to create a helpful digital assistant [7]. It has four steps.
- Supervised Learning. Human annotators provide example dialogues and a foundation model is finetuned on them via next token prediction.
- Generated Answer Ranking. The model trained in Step 1 is made to generate multiple answers in dialogues. Human annotators rank the answers by preference.
- Reward Model Training. A foundation model \(r\) -- could be the same, could be different --, is finetuned to provide scores for the generated answers so that if answer \(A\) is preferred by humans to answer \(B\), then we have \(r(A)>r(B)\).
- Reinforcement Learning. The answer generator model is further finetuned with Reinforcement Learning, originally PPO. That is, it is trained so that it generates answers that maximize the expected score by \(r\).
Steps 2-4 can be repeated at will.
Direct Preference Optimization
As it later turned out, steps 3-4 can be replaced by an objective directly on the generator model [8].
References
[1] Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-Training. link
[2] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. link
[3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS '20). Curran Associates Inc., Red Hook, NY, USA, Article 159, 1877–1901. link
[4] OpenAI (see the Authorship, Credit Attribution, and Acknowledgements section), 2023. GPT-4 Technical Report. link
[5] OpenAI , 2025. OpenAI GPT-4.5 System Card. link
[6] OpenAI , 2025. GPT-5 System Card. link
[7] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (NeurIPS). link
[8] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Advances in Neural Information Processing Systems 36 (NeurIPS). link