How large language models like Llama work

When you type a question into the chat, Llama does not look the answer up anywhere. It writes the answer, one small piece at a time, based on patterns it learned in training. Understanding that one idea explains most of what large language models (LLMs) do well and most of the ways they fail. This guide walks through how they work, without any maths, using the Llama models in our chat as examples.

Tokens: how a model reads text

Language models do not see letters or whole words. Text is split into tokens: a common word is often a single token, longer or rarer words are split into pieces, and punctuation counts too. Each token is turned into a list of numbers that the model can calculate with.

Tokens explain some odd behaviour. Ask a model how many times the letter "r" appears in a long word, or to spell a word backwards, and it may get it wrong, because it was never shown the separate letters in the first place. Tokens are also why limits (how much text a model can read, how long an answer can be) are measured in tokens rather than words. Languages that need more tokens per sentence reach those limits sooner.

Prediction: one token at a time

At its core, an LLM does one thing: given the text so far, it estimates how likely each possible next token is. It picks one, adds it to the text and repeats. A long answer is hundreds or thousands of these steps.

The model does not always pick the single most likely token. A little controlled randomness, usually set by a parameter called temperature, keeps answers varied and natural. That is why the same question gives different wording each time, and why pressing regenerate can produce a better answer.

Training: from text predictor to assistant

Pre-training

First, the model reads an enormous amount of text and learns to predict the next token. Meta's model cards describe the pre-training data for Llama 3.3 and 3.1 as "a new mix of publicly available online data" of more than 15 trillion tokens. Llama 4 was trained on a mix of publicly available and licensed data and information from Meta's products, about 40 trillion tokens for Scout and 22 trillion for Maverick. Along the way the model picks up grammar, facts, styles, code and patterns of reasoning, because all of these help it predict better.

Instruction tuning

A raw predictor just continues text; it does not answer questions helpfully. So the model is trained further on examples of instructions with good responses, and on feedback from people about which answers are better. Meta's Llama 3.x model cards name these steps supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF). The result is the "Instruct" version you chat with.

The knowledge cut-off

Training data stops at some date, and the model knows nothing after it. According to Meta's model cards, the cut-off is August 2024 for Llama 4 Maverick and Scout and December 2023 for Llama 3.3 70B and Llama 3.1 8B. Ask about anything later and the model either says it does not know or, worse, guesses. Our chat has no web access to fill that gap.

Context: the model's working memory

Everything the model considers while writing an answer (the instructions, the conversation so far and your new message) is called its context. There is no memory beyond it. When you start a new chat, the model knows nothing about your earlier ones.

The maximum size is the context window, measured in tokens. Meta's model cards list 128K tokens for Llama 3.3 and 3.1, 1M for Llama 4 Maverick and 10M for Llama 4 Scout; the provider we use offers about 524K for Maverick and about 1.3M for Scout. Our chat sends a shorter slice of each conversation to keep answers quick, which is why very early messages in a long chat can stop influencing the replies.

Mixture of experts: how Llama 4 is built

Llama 3 models are dense: every parameter (the learned numbers inside the model) is used for every token. Llama 4 uses a mixture-of-experts design instead. The model contains many specialised sub-networks called experts, and for each token only some of them are used. Meta's model card gives the figures: Llama 4 Scout has 17 billion active parameters out of 109 billion in total, with 16 experts; Llama 4 Maverick has 17 billion active out of 400 billion, with 128 experts. The benefit is a model that stores a lot of knowledge but costs less to run per token than a dense model of the same total size.

Llama 4 is also natively multimodal. Meta describes "early fusion": text and images are processed together in one model rather than by a separate add-on. That is why Maverick and Scout can answer questions about a photo you attach, while the text-only Llama 3 models cannot.

Why models make things up

A language model is trained to produce plausible text, not to check whether it is true. When it has learned a fact well, plausible and true coincide. When it has not (a rare name, a precise number, a recent event, a citation) it still produces something that sounds right. This is usually called hallucination. It is not lying: the model has no separate database of facts to consult.

• Exact numbers, dates, quotes and references are the most likely parts to be wrong.
• Asking "are you sure?" can trigger a correction, or an apology for something that was right. It is not a reliable test.
• Giving the model the source text and asking it to work only from that usually reduces errors.

You can see the difference with two versions of the same request:

From memory (riskier):
When was the town library in [my town] founded, and who funded it?

From a source (safer):
Using only the text below, say when the library was founded and who
funded it. If the text doesn't say, answer "not stated".
"""
[paste the paragraph from the library's own website]
"""

In the first case the model has probably seen little or nothing about your local library and may produce a confident guess. In the second, the facts are in its context, so it only has to read them. The summarizer works the same way: it answers from the text you give it.

Our guide on fact-checking AI answers turns this into a practical routine.

What this means when you use the chat

To see these ideas in action, ask Llama to explain itself. For prompting technique, read how to write prompts; for the models themselves, see what Llama is and the history of the Llama family.

FAQ

Does Llama understand what it says?

That depends on what you mean by "understand", and researchers disagree. In practice it handles meaning well enough to summarise, translate and reason through many problems, yet it can fail at things a child finds easy. Judge it by checking its work, not by how fluent it sounds.

Does it learn from my conversations?

No. The model's parameters do not change while you chat. Within one conversation it uses what you said as context; once you start a new chat, that context is gone.

What is the difference between Llama and ChatGPT?

Both rely on large language models built on similar principles. Llama is a family of models made by Meta and released with open weights under Meta's license, so anyone can download and run them; ChatGPT is OpenAI's app, which runs OpenAI's models as a hosted service. See Llama vs ChatGPT.

What are parameters?

They are the numbers inside the model that are adjusted during training. More parameters can store more patterns but need more computing power. Size alone does not decide quality; the training data and tuning matter too.

Why do only some models accept images?

Llama 4 Maverick and Scout were trained on text and images together. Llama 3.3 70B and Llama 3.1 8B are text-only models, so they cannot see pictures. Our guide on choosing a Llama model compares all four.