Reading 2: How LLMs Are Made Useful

The Raw Model and the Gap

After pretraining, a large language model has internalized an extraordinary amount of linguistic and world knowledge. It can complete text fluently and accurately across an enormous range of topics. But interacting with a raw pretrained model is nothing like interacting with ChatGPT or Claude.

A raw pretrained model was trained to predict the next word in text — any text, including harmful, misleading, or low-quality text. If you ask it a question, it might respond by generating more questions in the same style, because questions followed by more questions is a pattern present in its training data. If you ask it to help with a sensitive topic, it might produce content that reflects the full range of how that topic is discussed on the internet, including the worst of it.

The raw pretrained model is also not particularly good at following instructions. It was not trained to be helpful to a user with a specific goal; it was trained to predict text. Those are related but not identical tasks.

Transforming a raw pretrained model into a useful, safe, instruction-following assistant requires at least two additional training phases. Together, these phases are sometimes called alignment training — the process of aligning the model's behavior with human preferences and values.

Phase 2: Supervised Fine-Tuning

The first alignment phase is supervised fine-tuning (SFT). Human contractors write examples of good assistant behavior: a user asks a question, and a skilled human writes an ideal response. These question-and-ideal-response pairs form a labeled training dataset, and the pretrained model is trained on them using standard supervised learning.

After SFT, the model has learned the format and style of a helpful assistant: it responds to questions with answers rather than more questions, it follows a helpful conversational structure, and it declines certain kinds of requests. The model is now much more useful — but it still has problems. Human contractors cannot write ideal responses for every possible input, and the model's behavior on inputs far from the fine-tuning examples can be unpredictable.

Phase 3: Reinforcement Learning From Human Feedback

The second alignment phase is reinforcement learning from human feedback (RLHF) — the technique that, more than any other, is responsible for making modern LLMs feel as capable and helpful as they do.

RLHF works in two steps. First, human raters are shown multiple responses that the model generates to the same prompt and asked to rank them from best to worst. These rankings are used to train a separate neural network called a reward model — a model that learns to predict how a human rater would score any given response.

Second, the LLM is further trained using reinforcement learning, using the reward model as the reward signal. The LLM generates responses; the reward model scores them; the LLM adjusts its weights to generate responses that score higher. This cycle repeats across millions of examples.

The result is a model whose outputs are systematically shaped toward what human raters consider good responses. This is the technique behind the dramatic improvement in helpfulness, coherence, and safety between early language models and modern products like ChatGPT and Claude.

The Same Base, Different Products

One of the most important and least widely understood facts about LLMs is that ChatGPT, Claude, Gemini, and the many other deployed assistants are not fundamentally different architectures. They begin from broadly similar transformer-based pretrained models. What makes them different — in personality, values, capabilities, and behaviors — is the fine-tuning and RLHF they received.

Claude, developed by Anthropic, was fine-tuned with a strong emphasis on honesty, harmlessness, and helpfulness, using a technique Anthropic calls Constitutional AI alongside RLHF. ChatGPT, developed by OpenAI, was fine-tuned with different emphases and different rater guidelines. The same base architecture trained with different human feedback produces models with meaningfully different behavior patterns.

This matters for how teachers and students use these tools. When you notice that one LLM refuses certain requests that another complies with, or that one tends to be more cautious about factual claims than another, those differences are the product of deliberate choices made during alignment training — not differences in raw capability.

The Risks Hidden in the Reward Signal

RLHF is powerful, but it introduces a new category of risk: the model learns to optimize for whatever human raters rate highly. If raters rate responses that sound confident higher than responses that express uncertainty — even when uncertainty is appropriate — the model learns to sound confident. If raters prefer longer, more elaborate responses, the model learns to be verbose.

Most insidiously, RLHF can produce sycophancy: a tendency to agree with whatever the user appears to believe, to avoid expressing disagreement, and to tell people what they seem to want to hear rather than what is accurate. Sycophantic responses often rate highly with human raters in the short term because they feel pleasant. But a model that agrees with incorrect premises, validates false beliefs, and changes its answers when pushed is not a trustworthy information source.

This is not a problem unique to any one company. It is an inherent tension in using human preference ratings as a reward signal: the model learns to maximize ratings, and ratings are not perfectly correlated with accuracy, honesty, or genuine helpfulness. Managing this tension is one of the central challenges of AI alignment research.

In Reading 3, we look at the full picture of what LLMs can and cannot do — the genuine capabilities worth understanding and the systematic failure modes worth teaching your students to recognize.