/notes / 26 Oct 2026

ChatGPT Isn't Magic, It's Autocomplete on Steroids

Part 2 of AI From Zero to Agents. How a large language model really writes (one word at a time, from probabilities learned on a huge slice of the internet), why it makes things up, what “thinking” models changed, and how an LLM becomes an agent.

· 6 min read

Series · AI From Zero to AgentsPart 2 of 9
  1. 1 AI for Newbies: The 10-Minute Guide That Finally Makes It Click
  2. 2 ChatGPT Isn't Magic, It's Autocomplete on Steroids
  3. 3 Not All AI Is the Same Brain: Meet the Model Family
  4. 4 ChatGPT vs Claude vs Gemini vs Grok vs Llama vs Perplexity: Which AI Should You Actually Use?
  5. 5 Free, Open and on Your Laptop: Open-Source Models and the Secret Currency Called Tokens
  6. 6 From Chatbot to Coworker: What Agentic AI Is and How to Build Your First Agent
  7. 7 Stop Prompting, Start Automating: AI Workflows That Save You Hours Every Week
  8. 8 The USB-C Moment for AI: MCP vs A2A vs ACP, and How Agents Talk to Each Other
  9. 9 Let's Build It: A Hands-On Agent-to-Agent Walkthrough
A vintage typewriter whose paper ribbon dissolves into a stream of glowing blocks rising into the air

Type "Happy birth" on your phone and the keyboard suggests "day". Honestly, that little feature is the best starting point I know for understanding ChatGPT, Claude and Gemini. They do the same kind of guessing, except they've read something close to the whole internet and can keep guessing for pages at a time.

In Part 1 we found generative AI sitting in the middle of the Russian dolls. This time we take it apart.

What "LLM" stands for

LLM means large language model, and each word earns its place. It's large because it has hundreds of billions of internal settings (called parameters) learned from trillions of words. It's about language because reading and writing text is its core skill, even though the newer ones can also look at images and listen to audio. And it's a model because what you end up with is a trained file holding all those learned patterns.

It helps to separate the app from the engine. ChatGPT is a product; the LLM inside it is what does the work, and right now that's OpenAI's GPT-6 family. Claude runs on Anthropic's Opus, Sonnet and Haiku models, and Gemini on Google's Gemini models. Same basic idea, different engines under the bonnet. We'll go through the engines properly in Part 3.

It only knows one move

Everything an LLM does comes down to a single question, asked over and over very quickly:

Given all the text so far, what's the most likely next bit of text?

Bar chart showing the model's top guesses for the next word after “The capital of the UAE is”: Abu Dhabi has by far the highest probability, followed by much smaller bars for Dubai, a, located and the
One step of an LLM: score every possible next word, pick one, repeat. Illustrative probabilities.

It picks a word (strictly speaking a token, which we'll get to in Part 5), adds it to the text, and asks the question again. A typical answer takes hundreds of rounds. That's why replies appear on your screen word by word; you're watching the predictions happen live.

What surprised even the people building these things is how far that one trick goes. To get good at predicting the next word across every kind of writing out there, a model has to absorb grammar, facts, reasoning patterns, coding habits, tone, even a sense of humour (of sorts). Guessing the next word turns out to force it to build a rough picture of the world.

How a model gets trained

Three-stage pipeline: 1 pre-training reads a huge slice of the internet and books to learn language and facts; 2 fine-tuning on curated examples of good answers teaches it to be an assistant; 3 human feedback ranks answers to teach it to be helpful, honest and safe; the result is the chat assistant you use
From raw text to a helpful assistant

There are roughly three stages. First comes pre-training, where the model reads an enormous amount of text and plays "guess the next word" billions of times, adjusting itself a little after every miss. This takes months on thousands of chips and costs a fortune. What comes out knows a great deal but makes a pretty awkward conversation partner.

So next it's fine-tuned on lots of examples of good assistant behaviour, question in, excellent answer out, until it picks up the shape of being helpful. Finally it learns from feedback: people (and more and more often, other AI systems) compare pairs of answers and say which is better, and the model gets nudged towards the clearer, more honest and safer ones.

One side effect matters a lot in everyday use. A model's built-in knowledge stops at a knowledge cutoff date. Ask about last week's news and a model without web search will either tell you it doesn't know or, worse, have a go anyway. That's exactly why the modern assistants can search the web: it hands them fresh facts to work from.

Why it makes things up

Go back to the one move. The model writes the most plausible next word, and most of the time plausible and true line up nicely. But when it's seen very little about something (an obscure person, a niche regulation, a book that doesn't exist) the most plausible-sounding continuation can be pure invention, delivered in exactly the same confident tone.

Three habits cut this down a lot in my experience. Give it the facts, either by pasting the document or switching on web search, and ask it to answer from that. Ask for sources and actually click them. And give it permission not to know, with something like "If you're not sure, say so." You'd be surprised how often that last line changes the answer.

Its short-term memory: the context window

An LLM doesn't remember you from one chat to the next unless the product bolts on a memory feature. Within a single conversation, it can only "see" a limited amount of text at once, and that limit is called the context window. Everything you paste in, every earlier message and all of its own replies have to fit inside it.

That window has grown enormously:

Bar chart on a logarithmic scale showing context windows growing from about 2 thousand tokens in 2020 to around 1 million tokens in 2024, which became common on flagship models by 2026
From a few pages to a few novels: how much text a model can see at once

In 2020 a model could hold about three pages. Today's flagship models regularly take around a million tokens, which is roughly ten novels, so you can drop in a whole archive of contracts or an entire codebase and ask questions about it.

Thinking before it speaks

The classic LLM answers in one pass, so the first word it writes commits it to a direction. Since late 2024 a different style has caught on: reasoning models, usually behind a "Thinking" or "Deep think" switch. Before answering they work through the problem privately, check their steps, back up when something doesn't add up, and only then give you the final answer.

Comparison: a fast model answers straight away and is great for simple tasks; a reasoning model first works through steps, checks and corrects itself, then answers, which is slower but better for maths, planning, code and tricky analysis
Fast mode vs thinking mode: same engine family, different working style

My rule of thumb: fast mode for writing and quick questions, thinking mode for anything you'd want a colleague to sleep on. Numbers, plans, multi-step logic and debugging all belong in the second group.

From talking to doing

So far the LLM is a brilliant talker. It produces text, and you decide what to do with it. Agentic AI is what you get when you let it act. You give it tools (search, email, calendar, a spreadsheet, a browser) and let it run in a loop: plan, use a tool, look at what happened, decide the next step, and keep going until the job's done.

Left: an LLM takes a question and returns an answer. Right: an agent wraps the LLM in a loop with tools and memory: it plans, acts with tools such as search, email and calendar, observes the result and repeats until the goal is reached
An LLM answers. An agent gets things done.

That's where this whole series is heading, and Parts 6 to 9 are all about agents: how they work, how to build one, and how they talk to each other.

See the cutoff for yourself

Here's a quick one. Ask an assistant "Who won the most recent Formula 1 race?" with web search turned off (or in a model that doesn't have it). Then ask the same thing with search switched on. Compare the two answers, and especially how confident each one sounds. You've just watched the knowledge cutoff, and its cure, in action.

In Part 3 we'll look at what a "model" really is, why every company sells big, medium and small versions, and how text, image, voice and video models differ from one another.

AILLMChatGPTAI for beginnersAgentic AI

More notes