Ask how a large language model is trained and you will usually hear a three-part story. It reads a large part of the internet, learns to predict the next word, then receives human feedback so it behaves like an assistant. That story is not wrong. It is now too compressed to explain the systems people actually use.
A frontier model is not made in one training run. Public technical reports increasingly describe a sequence of learning loops. Data preparation feeds pretraining. Pretraining is followed by a more selective phase often called continued pretraining, annealing or midtraining. Supervised examples install basic behaviours. Preference training and reinforcement learning amplify some behaviours and suppress others. Distillation transfers selected capabilities between models. Evaluation and safety work send failures back into another round. The released model is the residue of all of them.
The important shift is not another acronym. It is that each loop supplies a different kind of judgment about what the model should become.
The first judgment appears before training starts. A laboratory may begin with an enormous pool of webpages, books, code, scientific material, images, audio, licensed collections and human-created data. It does not simply pour that pool into a cluster. It removes duplicates, filters unwanted material, scores quality, classifies domains and chooses a mixture. A model trained on more code will not acquire the same repertoire as one trained on more conversation or mathematics. The dataset is already a set of decisions about which patterns deserve to survive.
Pretraining then turns that mixture into a base model. For text, the core task remains remarkably simple: predict the next token, compare the prediction with the actual token, and adjust the model’s parameters to make the next prediction slightly better. Repeated across trillions of tokens, that objective forces the model to learn statistical structure that supports language, factual recall, coding and some forms of reasoning. The Llama 3 report remains one of the clearest public accounts of this stage at scale.
The result is not yet the assistant in a chat window. A base model is a powerful continuation engine with a very broad space of possible behaviour. It may answer a question, continue it, imitate its style or wander into an adjacent genre. Pretraining creates capability without fully specifying how that capability should be presented or when it should be used.
Modern laboratories increasingly insert a more selective loop before turning that base model into an assistant. The terminology varies, but the shape is recognisable. Training continues on a smaller, higher-quality mixture that may place more weight on mathematics, code, scientific reasoning, long documents or tool-use traces. Context windows may be extended here. Microsoft AI’s MAI-Thinking-1 report calls this midtraining and treats it as preparation for later reasoning work. Other reports describe annealing or continued pretraining instead.
This stage matters because reinforcement learning cannot reliably amplify a behaviour the base model almost never produces. If a model has weak habits of verification, backtracking or structured problem solving, a later reward may have little useful variation to select from. Midtraining is partly curriculum design. It changes what the model is ready to learn next.
Supervised fine-tuning then teaches the interaction grammar. The model sees examples of prompts and good responses. It learns chat formats, instruction following, tool-call syntax, refusal patterns and the expected shape of an answer. This stage is often described as teaching the model what to do, but that overstates it. Much of its role is to make capabilities learned earlier accessible in a predictable form.
The next loops decide which accessible behaviours should become more likely. Preference training compares outputs and favours the ones humans or other models judge better. Some systems train a reward model from those comparisons and optimize against it. Others use direct preference methods. Either way, the source of learning has changed. The model is no longer learning only from what people wrote. It is learning from judgments about which of its own possible answers should win.
Reasoning models add another important signal: outcomes that can be checked. Code can be executed. A mathematical answer can be verified. A tool-using agent can be placed in an environment where success or failure is observable. Reinforcement learning can then reward successful trajectories rather than merely polished prose. DeepSeek-R1 showed how far this could be pushed on reasoning tasks, while Ai2’s open Tulu 3 recipe combined supervised tuning, preference tuning and reinforcement learning with verifiable rewards.
This explains why post-training is no longer well described as a final coat of alignment. It can materially change capability. It can teach a model to search longer, test an answer, use tools and recover from a failed attempt. It also has a natural boundary. Verification is cheap for code and mathematics because correctness can often be checked automatically. It is much harder to construct a trustworthy reward for whether a strategy is wise, an explanation is fair or a judgment is honest. The model will optimise the signal it receives, not the intention behind it.
The model also becomes part of its own data factory. A stronger model can generate exercises, responses and reasoning traces for another model. Successful attempts can be filtered and fed back into supervised training. A large teacher can be distilled into a smaller student. Specialist models can be trained on separate domains and their behaviour consolidated into a general model. Synthetic data is therefore not one category with one risk. Generated mathematics checked by a verifier is a different object from unverified prose recursively copied from another model.
Safety and evaluation sit across this pipeline rather than at its end. Data can be filtered before pretraining. Instruction hierarchies and policy examples can be trained during supervised stages. Preference and reinforcement signals can reward helpful compliance and penalise unsafe behaviour. Red teams and capability evaluations can expose failures that become new training examples. Deployment may add separate monitors, filters and access controls around the weights.
This does not mean the builders can point to one loop and explain any individual answer. They control data mixtures, objectives, rewards and tests. They observe the behaviours that emerge from their interaction. The causal chain is partly engineered and partly empirical. A refusal may reflect training data, supervised policy examples, preference optimization, a system instruction or an external safety classifier. A confident error may survive every stage for different reasons. Public reports disclose enough to show the pipeline, but rarely enough to reconstruct the exact recipe of a closed frontier model.
That distinction matters when evaluating a provider’s claim. “The model was trained for safety” is incomplete. Which loop supplied the safety signal? What behaviour was rewarded? Which evaluations measured the result? Were those evaluations independent, contaminated by training, or limited to tasks with easy scoring? What remains a system-level control outside the model? These questions turn a description of activity into a chain of evidence.
They also prevent a common category error. The thing a user experiences is not just a trained model. It is a model combined with system instructions, retrieval, tools, memory, routing, monitors and a user interface. Training explains the repertoire and tendencies in the weights. The surrounding system determines what the model can see, what it can do and which failures reach the user.
The best mental model is therefore not a student who reads the internet and then attends an etiquette class. A frontier model is a stack of compressed decisions. Pretraining compresses choices about data. Midtraining compresses choices about curriculum. Supervised tuning compresses examples of desired conduct. Preference learning compresses comparative judgment. Reinforcement learning compresses definitions of success. Distillation compresses another model’s selected behaviour. Evaluation and safety loops compress what the laboratory noticed and chose to repair.
By the time the model reaches a chat window, all those decisions occupy the same weights and are difficult to separate. That is why asking how it was trained has no one-line answer anymore. The useful question is which learning loops shaped the behaviour you care about, what signal each loop supplied, and what evidence shows that the signal produced what its designers intended.