The Evolution of LLM APIs: From Completions to Tool Calling, Structured Outputs and MCP

Version note: This post covers the evolution of LLM APIs from 2020 to November 2026, when it was written. Dates were verified against the providers' own announcements, changelogs and release notes.

Guess First

You want JSON for a book from the Bookstore assistant in the shape {"title": ..., "genre": ..., "year": ...}. You have three options: write "return JSON only" in the prompt, turn on JSON mode, or pass a JSON schema. Which one guarantees valid JSON? Which one guarantees that the JSON matches your schema?

The answer: the prompt guarantees neither, JSON mode only valid JSON, and schema-constrained structured output also aims at matching the schema. These three options were not invented at the same time; they are the stops of an evolution spread over three years. In the last post of the series we move from models to APIs and look at how the way a developer talks to an LLM has changed.

The Start: An API That Continues Text

OpenAI opened its first API on June 11, 2020. The model behind it was GPT-3, and as we saw in post 18, that model was trained to continue a text. The API offered a matching, very simple contract: you send a prompt, and the model writes its continuation. Giving instructions, showing examples, describing the output format: all of it was framed inside a single text, as "this is how it should continue".

This shape still lives today. Ollama's /api/generate endpoint takes a prompt, and on the OpenAI-compatible side there is /v1/completions. The num_predict (max tokens) and stop sequences we used in the early posts of the series are also a legacy of this period: while the model continued the text, you had to tell it where to stop.

March 1, 2023: The Chat Format

Three months after ChatGPT's success, on March 1, 2023, OpenAI opened the gpt-3.5-turbo model to the API in the post "Introducing ChatGPT and Whisper APIs". The price was $0.002 per 1,000 tokens; according to the announcement, ten times cheaper than the previous GPT-3.5 models. But the lasting change was not the price but the format. A request was no longer a single text but a list of messages with roles: system, user, assistant.

We used these roles in post 5, and in post 18 we saw that they are still laid out in front of the model as a single text. So why separate fields? Because making the split at the API level leaves the template to the provider. You write role: "system", and turning that role into the model's special markers (<start_of_turn> in gemma3) is the provider's job. This also made switching providers easier. The reason we could call Ollama with the OpenAI SDK in post 17 is that the chat format became a de facto common language.

June 13, 2023: Function Calling

The next step was letting the model produce not just text but a proposed action. On June 13, 2023, in the post "Function calling and other API updates", OpenAI announced the functions and function_call parameters with the gpt-4-0613 and gpt-3.5-turbo-0613 models. The idea is exactly the flow we built in post 11: you describe your functions with a JSON schema, the model returns a call request instead of text when needed, and you run the call and send the result back. The model never runs the code itself.

This capability spread across providers quickly. At DevDay on November 6, 2023, OpenAI added parallel function calling, which can request more than one call in a single answer. On May 30, 2024, Anthropic made tool use generally available in the Claude 3 family. Ollama brought tool support with v0.3.0 on July 25, 2024, and started sending tool calls with streaming on May 28, 2025. Today most APIs call this field tools, and that is how we used it in the series too.

From JSON Mode to Structured Outputs

Function calling relied on the model producing JSON, but there was no guarantee that the model would always produce valid JSON. The first fix came at DevDay on November 6, 2023: JSON mode, turned on with response_format. This mode made sure the output was valid JSON, but it didn't guarantee which fields there would be or what types they would have. The second half of the opening question is exactly that gap.

The gap closed on August 6, 2024 with the post "Introducing Structured Outputs in the API". You could now pass a json_schema inside response_format, and ask for strict schema adherence in tool definitions with strict: true. On OpenAI's own schema test, gpt-4o-2024-08-06 scored 100 percent with this feature; gpt-4-0613 in the comparison had stayed below 40 percent. Ollama also started accepting a JSON schema in the format field with v0.5.0 in early December 2024; that is the feature we used with gemma3:4b in post 10. The rule from post 10 still holds, though: JSON that matches the schema is not the same as correct JSON. Do the validation in your own code.

Reasoning Parameters

The reasoning models from post 19 brought a new setting to the API: how much the model should think. When OpenAI opened o1 to the API on December 17, 2024, it added the reasoning_effort parameter to the changelog. Ollama brought the think parameter with v0.9.0 at the end of May 2025; it takes true/false or a level such as "low", "medium", "high", and returns the thinking text separately from the answer, in the thinking field. Ollama's OpenAI-compatible endpoint also accepts a reasoning_effort value. On the Microsoft.Extensions.AI side, ChatOptions has a Reasoning property.

Here you see an abstraction problem again. The thinking setting comes with a different name and different details at every provider. The rule from post 17 applies: stand behind a shared interface and keep provider-specific settings in one place.

Model Context Protocol

Function calling made it possible for the model to call tools, but every application had to define every tool itself. When you wanted to connect the Bookstore assistant to the order database, the shipping company's API and the company's documents, you wrote a separate tool definition and separate integration code for each one. If another application wanted to connect to the same sources, it would do the same work again.

On November 25, 2024, in the post "Introducing the Model Context Protocol", Anthropic proposed an open standard for this problem. In MCP, an MCP server offers tools and data sources; the LLM application connects to that server as an MCP client and introduces the offered tools to the model as tools. Think of it like USB: one common plug instead of a separate cable for every device. You write an MCP server for the order database once, and every application that speaks MCP can use it. The mechanism underneath is still the tool calling from post 11; MCP adds a distribution and sharing layer on top of it.

State on the Server and Agents

There is a fact we have repeated since the beginning of the series: chat APIs are stateless, and you send the history with every request. Providers also tried moving this load to the server. On November 6, 2023, OpenAI announced the Assistants API in beta, which keeps the conversation and tools on the server; and on March 11, 2025, in the post "New tools for building agents", it introduced the Responses API and the Agents SDK. Ollama's OpenAI-compatible side also has /v1/responses, but according to the Ollama docs only in a stateless form.

The word "agent" is everywhere in this period. But if you look at what we built in posts 11 and 12, an agent's skeleton is very simple: a model, a list of tools, and a loop that runs until the model stops asking for tools. UseFunctionInvocation in Microsoft.Extensions.AI ran that loop for you. Agent frameworks add layers on top of this skeleton such as planning, memory, coordinating multiple agents and observability. As long as you know the skeleton, which framework you use becomes secondary.

Common Pitfalls

First, mistaking JSON mode for a schema guarantee. JSON mode gives you valid JSON, not your fields. If you need a schema, use schema-constrained structured output and still validate the result in your code.

Second, blindly copying examples written with old parameter names. The mid-2023 functions/function_call format and today's tools format are not the same. When you take code from a blog post or an old answer, check its date and the provider's current API reference.

Third, mistaking MCP for a security solution. MCP makes it easier to connect tools; limiting what a tool can access and which operation needs user approval is still your job. The rules from post 12 on limiting permissions and validating arguments apply to MCP servers just the same, and the prompt injection topic from the Web Security series matters even more here.

Fourth, assuming state kept on the server is free. Even if the provider keeps the conversation, the history is tokens as long as it enters the model's context, and it shows up on the bill. Do the cost math from post 16 when you use a stateful API too.

A Small Experiment

★☆☆ Ask Ollama the same question three ways: with /api/generate as a prompt, with /api/chat as messages, and with /v1/chat/completions in the OpenAI format. Put the JSON structure of the three answers side by side; which fields hold the text, the token counts and the stop reason?

★★☆ Try the three options from the opening with gemma3:4b: prompt only, with format: "json", and with a JSON schema in the format field. Run each with ten different seed values; how many answers passed JSON.parse, and how many matched your schema?

★★★ With qwen3:4b, define the same tool through two APIs: Ollama's /api/chat endpoint, and /v1/chat/completions in the OpenAI format. Is the type of the arguments field inside the returned tool_calls the same in both APIs? When writing provider-independent code, where should that difference be hidden?

That brings us to the end of the LLM Fundamentals series. From tokens to the context window, from sampling to structured output, from tool calling to RAG, we built the core structure by running it on local models, then looked at where that structure came from and how the APIs evolved. The field changes fast; the series will keep being updated as new developments arrive. In the next AI series we will build on this foundation: AI applications with .NET, building RAG in detail with vector databases, agents and writing MCP servers, and LLM security. In all of them we will keep the same approach: the concept first, then code that actually runs.