LLMs from 2023 to Today: Open-Weight Models, Long Context and Reasoning

Version note: This post covers the period of LLMs from 2023 to November 2026, when it was written. Dates were verified against arXiv records and the companies' own announcements; current model rankings and prices are left out on purpose.

Guess First

In 2020, GPT-3's context window was 2,048 tokens. How large a window does gemma3:4b, the 4-billion-parameter model we have been running on a laptop in this series, support? Twice as much, ten times, more? And one more: if anyone can download and use this model, is it right to call it "open source"?

We saw the answer to the first question in the context length line of the ollama show gemma3:4b output: 131072, that is 128K, sixty-four times GPT-3. The answer to the second is "not always", and a good part of this post is about that distinction.

Early 2023: GPT-4

OpenAI announced GPT-4 on March 14, 2023. The model came with two window sizes: 8,192 tokens and, with limited access, 32,768 tokens. The "GPT-4 Technical Report", uploaded to arXiv the next day, showed that a habit had changed: citing the competitive landscape and safety, the report gave no details about the architecture, model size or training data. The "who should get the weights" debate that started in the GPT-2 days turned into a clear answer on the closed-model side at this point.

The Rise of Open-Weight Models

The same year a strong current started in the opposite direction. Meta's LLaMA paper went up on arXiv on February 27, 2023 ("LLaMA: Open and Efficient Foundation Language Models", Hugo Touvron and colleagues). In the same period, tools for running large language models on a personal computer also developed quickly. The "k-quants" formats such as Q4_K_M that we discussed in post 16 were added to llama.cpp on June 5, 2023. Ollama's first release, v0.0.1, came out on July 8, 2023.

On July 18, 2023, Meta announced Llama 2 together with Microsoft, and this time the model was free for research and commercial use. On September 27, 2023, Mistral AI released Mistral 7B under the Apache 2.0 license; we will see this widely used open source license that allows commercial use again later with Qwen3 and Gemma 4. Llama 3 arrived on April 18, 2024. On April 29, 2025, the Qwen team announced Qwen3, the family of the model we use in this series for tool calling and thinking.

The Gemma Family: From 1 to 4

Since Gemma 3 is the series' default model, let's look at Google's open-weight family separately. The first Gemma was announced on February 21, 2024; the "Gemma Terms of Use, Last modified: February 21, 2024" line in the ollama show gemma3:4b output carries that date. Gemma 2 came on June 27, 2024. Gemma 3 was announced on March 12, 2025 with the claim of being the most capable model you can run on a single GPU or TPU; its technical report went up on arXiv on March 25, 2025. Gemma 3's 4B, 12B and 27B sizes support a 128K context and image input, and cover more than 140 languages.

Gemma 4 was announced on April 2, 2026, and the family came with an important change: the license moved from Gemma-specific terms of use to Apache 2.0. In the Ollama library the gemma4 models are listed with tools and thinking capabilities. In this series we saw that gemma3:4b does not support tool calling; the next generation of the same family closes that gap. Here you see how a model family can change from generation to generation in both capability and license.

The Growth of the Context Window

The most concrete change of this period was in the context window. After GPT-3's 2,048 tokens, GPT-4 came with 8K and 32K in March 2023. Anthropic announced a 100K-token window for Claude on May 11, 2023. At OpenAI's DevDay on November 6, 2023, GPT-4 Turbo was introduced with 128K. On November 21, 2023, Claude 2.1 went up to 200K. Google announced Gemini 1.5 on February 15, 2024; the standard window was 128K, and 1 million tokens was offered in a limited preview. At the time of writing, the major providers' documentation lists models with a 1-million-token window.

Locally the picture is similar: on paper gemma3:4b supports 128K. But as we saw in post 4, Ollama allocates 4096 tokens by default on this machine and silently cuts off the rest. The window "the model supports" and the window "allocated to your request" are not the same thing. A long window isn't free either: as we calculated in post 16, every token is a cost, and as we saw in post 18, the attention computation grows as the window gets bigger.

Multimodality: Beyond Text

Models also stopped reading only text. On September 25, 2023, OpenAI brought image and voice support to ChatGPT. Google introduced Gemini on December 6, 2023. On May 13, 2024, OpenAI announced GPT-4o, which works with text, audio and images, in the post "Hello GPT-4o".

This capability came down to small open-weight models too. The capabilities in the ollama show output of the gemma3:4b we use include vision, and in post 16 we saw this model's 416-million-parameter vision encoder in the tensor types. According to the Gemma 3 technical report, images are resized to 896×896 pixels and turned into 256 tokens; so an image, too, ends up as nothing more than tokens taking space in the context window.

Reasoning Models

The 2022 chain-of-thought paper had shown that telling a model to "think step by step" works. The next step was to make this part of training instead of leaving it to the prompt. On September 12, 2024, OpenAI announced o1-preview in the post "Learning to reason with LLMs": a model that produces a long thinking process before it answers. On January 20, 2025, DeepSeek released DeepSeek-R1; the code and weights were opened under the MIT license, and the paper went up on arXiv on January 22, 2025. The paper's title sums up the approach: incentivizing reasoning capability through reinforcement learning.

After that the approach spread. On February 24, 2025, Anthropic announced Claude 3.7 Sonnet with an "extended thinking" mode. Qwen3, on April 29, 2025, combined thinking and non-thinking modes in a single model; that is what we used in post 9 when turning the think setting on and off with qwen3:4b. Ollama added the think parameter and the separate thinking field with v0.9.0 in May 2025.

Why No Ranking?

I didn't say "the best model is this one" in this post, and I am doing that on purpose. Model rankings change within months, sometimes weeks; benchmark results also move with the task type, the prompt and the measurement method. A ranking written today would make this post stale in a few months. What lasts is the trends: windows got bigger, small models gained capabilities, the licenses of open-weight models loosened, and reasoning became a separate mode. When you choose a model, read the provider's current documentation and measure with your own golden set, as in post 15.

Common Pitfalls

First, confusing open-weight with open source. Being able to download the weights doesn't mean you can use them however you like. Gemma 3 comes with Gemma-specific terms of use, while Gemma 4 comes with Apache 2.0. DeepSeek-R1's small distilled models also carry the licenses of the models they are based on. Read the license before you put a model into a product.

Second, mistaking the advertised context window for the window you use. Locally, check the window actually allocated with ollama ps; in the cloud, check the model's limit and the price of long input.

Third, using a reasoning model for every job. Thinking tokens mean both time and cost; in post 9 we saw that on simple questions that price doesn't pay off.

Fourth, assuming the model's knowledge is current. Every model has a knowledge cutoff; Gemma 3's is August 2024. If you ask gemma3:4b about the 2025 and 2026 events in this post, it is normal that it doesn't know them; if it answers as if it does, you are facing the hallucination from post 15.

A Small Experiment

★☆☆ Check the model's supported context length with ollama show gemma3:4b, and after sending a request, the CONTEXT value actually allocated with ollama ps. Compare both with GPT-3's 2,048 tokens.

★★☆ Ask qwen3:4b a simple arithmetic puzzle, first with think: true, then with think: false. Compare the eval_count and eval_duration values. Are the answers the same? How many tokens did the thinking cost?

★★★ Send gemma3:4b a photo of a book cover you have through the images field and ask for the book's title, author and publication year. How does it answer the information that isn't on the cover (such as the publication year)? Verify the answer and think about how it relates to the knowledge cutoff.

In the next post, the last of the series, we move from models to APIs: from the first API that completed text to chat, from function calling to JSON mode and structured outputs, from reasoning parameters to the Model Context Protocol and agents, we will look at how the way a developer talks to an LLM has changed.