From the Transformer to ChatGPT: The Birth of LLMs (2017–2022)

Version note: This post covers the 2017–2022 period of LLMs, the road from the Transformer to ChatGPT. Dates were verified against arXiv records and the companies' own announcements.

Guess First

GPT-3 came out in 2020 with 175 billion parameters. Two years later, OpenAI compared a 1.3-billion-parameter model trained with human feedback against the 175-billion GPT-3 and asked human evaluators whose answers they preferred. Which one do you think won? The one a hundred times smaller, or the one a hundred times bigger?

The answer is in the middle of this post. That result also explains why ChatGPT felt so different from GPT-3. In the first seventeen posts of the series we built the concepts by running them on local models; now we look at where those concepts came from.

2017: Attention Is All You Need

The story starts with a paper uploaded to arXiv on June 12, 2017: "Attention Is All You Need" by Ashish Vaswani and colleagues. The paper proposed a new architecture for machine translation called the Transformer.

At the time, most models that worked with text were RNN-based. An RNN reads a sentence word by word, in order; every step needs the result of the previous one. Think of it as a single-lane road: no matter how fast the cars are, they can't leave the line. This caused two problems. In long sentences, information from the beginning faded on its way to the end, and since training couldn't run in parallel, it was slow on large data.

The Transformer stopped reading in order. With the attention mechanism, every token looks at all the other tokens in the text at once and decides which ones matter to it. Because all tokens are processed in parallel, training fit GPUs much better. The "input is processed fast, output slowly" difference we saw in post 16 has its root here: the whole input can be processed in one go, but the output is still generated token by token. The context window concept also comes from this architecture; since every token looks at every other token, the compute and memory cost grows quickly as the window gets bigger.

2018: Read First, Learn Later

The next big idea was to train a model on a large amount of unlabeled text first, instead of training it from scratch for a specific task. On June 11, 2018, OpenAI announced what we now call GPT-1 in a post titled "Improving language understanding with unsupervised learning": first train a Transformer by predicting the next token, then fine-tune it for the task with a small labeled dataset. The "Generative Pre-Training" in the name GPT comes from here.

On October 11 of the same year, the BERT paper by Jacob Devlin and colleagues was published. BERT also used the Transformer with pre-training, but in a different way: it was an encoder that read text in both directions at once, and it did very well on understanding tasks such as classification and search. GPT stayed on the decoder line, generating text left to right. Almost all the LLMs we chat with today come from this second line; the "next-token prediction" definition from post 1 is exactly this.

2019: GPT-2 and the Staged Release

On February 14, 2019, OpenAI announced the 1.5-billion-parameter GPT-2 in a post titled "Better language models and their implications". The model could write coherent paragraphs without being trained for a specific task. OpenAI didn't release the full model right away; citing concerns about misuse, it shared smaller versions first and opened the 1.5-billion model on November 5, 2019. The debate about when and to whom a model's weights should be released, today's open-weight versus closed model split, became this visible for the first time.

2020: Scaling and GPT-3

On January 23, 2020, the paper "Scaling Laws for Neural Language Models" by Jared Kaplan and colleagues came out. Its observation: as model size, data and compute grow, the model's loss falls in a predictable way. This showed that investing in bigger models had a predictable return.

On May 28, 2020, GPT-3 arrived with the paper "Language Models are Few-Shot Learners" by Tom B. Brown and colleagues: 175 billion parameters and, in the paper's words, a context window of 2,048 tokens. The real surprise was not the size but the few-shot behavior. Shown a few examples, the model could do a new task without its weights being touched at all. The example-based prompts we wrote in post 8 have an academic name: in-context learning. On June 11, 2020, OpenAI opened its first API, and calling GPT-3 with an HTTP request became possible. We will cover how that API evolved into today's in detail in post 20.

GPT-3 had a problem, though: it was trained to continue text, not to follow an instruction. When you typed "recommend me a book", it could continue with a list of similar questions instead of a recommendation. The base model versus instruct model distinction from post 1 is exactly this.

2022: InstructGPT and RLHF

The seed of the solution had actually been planted on the same day as the Transformer paper. On June 12, 2017, Paul Christiano and colleagues proposed, in "Deep reinforcement learning from human preferences", learning by asking people which of two behaviors they prefer instead of writing the reward function by hand.

OpenAI applied this idea to language models and announced InstructGPT on January 27, 2022, in the post "Aligning language models to follow instructions"; the paper, by Long Ouyang and colleagues, went up on arXiv on March 4, 2022. The method had three steps. First, the model was fine-tuned on example answers written by people. Then people ranked different answers from the model from best to worst, and a reward model was trained on those rankings. Finally, the model was trained again with reinforcement learning using that reward model. The method's name is RLHF: reinforcement learning from human feedback.

The answer to the opening question is in the paper's abstract: the answers of the 1.3-billion-parameter InstructGPT were preferred to those of the 175-billion GPT-3, despite having a hundred times fewer parameters. It was not size alone that decided the quality users felt, but what the model was trained for. In the same month, on January 28, 2022, the chain-of-thought paper by Jason Wei and colleagues had also been published; it is the root of the "think step by step" approach we used in post 9.

November 30, 2022: ChatGPT

On November 30, 2022, OpenAI announced ChatGPT in the post "Introducing ChatGPT" and presented it as a sibling model to InstructGPT. Technically it was not a new invention; it was the combination of the pieces above: a large pre-trained Transformer, fine-tuning for instruction following and RLHF, plus a chat interface anyone could use.

What changed was access. Reaching this capability no longer required reading papers, getting an API key, or framing the prompt as text to be completed; typing a question was enough. The chat format wasn't a separate mechanism inside the model either. The system, user and assistant roles we saw in post 5 are laid out in front of the model as a single text, and the model still predicts the next token. You can see this on your own machine; one of the experiments below is exactly that.

What Changed for Developers?

The most important thing these five years left developers is that natural language became an interface. Before GPT-3, if you wanted a text classifier you collected data, labeled it and trained a model. With GPT-3 it became possible to try the same job with a prompt containing a few examples. With ChatGPT those prompts turned into instructions.

Almost every concept in the series comes from this period. Tokens and the context window come from the Transformer, next-token prediction and pre-training from the GPT line, few-shot prompts from GPT-3, the system prompt and role separation from instruct models, chain-of-thought from 2022. Knowing them in historical order makes it easier to tell, when a new model or API comes out, what is really new and what is an old idea in new packaging.

Common Pitfalls

First, treating ChatGPT and the model as the same thing. ChatGPT is a product, meaning an interface and the system behind it; the model running behind it has changed over time. What you call in the API is a model name.

Second, using Transformer and GPT as synonyms. Transformer is the name of the architecture; GPT is a model family on that architecture's decoder line. BERT is also a Transformer, but it is not designed to generate text.

Third, thinking RLHF teaches the model new knowledge. RLHF shapes how the model answers, meaning instruction following, format and tone. What the model knows comes mostly from pre-training; that is why the hallucination we discussed in post 15 continues in RLHF-trained models too.

Fourth, concluding "a bigger model is always better" from the scaling laws. The InstructGPT result showed that quality in the user's eyes also depends on the training objective; that is also what we saw when comparing 4b and 12b in post 16.

A Small Experiment

★☆☆ Look at the chat template of the gemma3:4b model: ollama show gemma3:4b --template. Find which special markers (such as <start_of_turn>) turn your messages into a single text. Which role does the system message become in this template?

★★☆ See the pre-GPT-3 "continue the text" behavior: pass "raw": true to an /api/generate request, so the chat template is bypassed. Send half a sentence as the prompt (e.g. The three best-selling programming books of all time are) and ask the same thing as a question through /api/chat. How does the shape of the answers differ?

★★★ Try in-context learning: run a prompt that guesses a book's genre from its title, first with zero examples, then with three. Use temperature: 0 and a fixed seed; measure with a small set of ten books how much the examples change accuracy.

In the next post we move to the period from 2023 to today: GPT-4, the rise of open-weight models with Llama, Mistral and Gemma, the context window growing from a few thousand tokens to a million, models that can read images, and the reasoning models brought by o1 and DeepSeek-R1.