Topic 7c – Large Language Models

What an LLM Is

1. The attention mechanism allows the model to consider every other word in the input when processing each word, and to assign different levels of relevance ("attention") to each. Rather than processing words one at a time in sequence, the model can directly connect distant parts of the text. This matters for language because meaning is often context-dependent in non-local ways: in "The trophy didn't fit in the suitcase because it was too big," the word "it" requires connecting back to both "trophy" and "suitcase" several words earlier, and deciding which one is "too big" requires attending to the semantic context of the whole sentence.
2. Successfully predicting the next word requires the model to encode an enormous amount of implicit knowledge. To predict that "The capital of France is ___" ends with "Paris," the model must represent geographic facts. To predict that "She felt sad after ___" continues in emotionally coherent ways, the model must represent emotional context. To predict grammatically correct continuations, the model must represent syntactic structure. None of these things are explicitly labeled — they must all be inferred from the statistical patterns of how language is actually used, which means the model must build internal representations that go far beyond raw word co-occurrence.
3. Similar to ELIZA: both produce output that feels meaningful and contextually appropriate without necessarily understanding the content in any deep sense. Both exploit patterns in language to generate responses that humans find coherent. The ELIZA effect — attributing understanding to a system that is processing patterns — applies to both. Fundamentally different: ELIZA used handcrafted pattern-matching rules with no learning. A modern LLM has learned representations of vast amounts of human knowledge from text and can generalize to novel situations far beyond any scripted rule set — its behavior is qualitatively more sophisticated. Does it matter for use? Yes: the ELIZA lesson warns against assuming genuine understanding from plausible output, and that caution applies to LLMs too. But LLMs' genuine capabilities — summarizing, explaining, drafting, translating — are real and valuable in ways ELIZA's were not.

How LLMs Are Made Useful

4. A pretrained LLM has learned to predict the next word from massive text data — it models the statistical structure of language extremely well. But interacting with a raw pretrained model would feel strange: prompt it with "What is the capital of France?" and it would be just as likely to continue with "What is the capital of Germany? What is the capital of Spain?" as with "Paris" — because that kind of question-and-question sequence appears in its training data. The model completes text patterns; it is not inherently trying to be helpful. Fine-tuning shapes it into a model that follows instructions and gives direct, helpful answers by training it specifically on examples of helpful responses, and by using RLHF to reward helpful, safe, accurate outputs.
5. Risks: (1) Rater bias — if human raters systematically prefer certain styles, demographics, or viewpoints, the model will learn to produce outputs that appeal to those preferences, potentially encoding biases into the model's behavior. (2) Reward hacking — the model may learn to produce outputs that score well with raters without actually being more truthful or useful; for example, sounding more confident may get higher ratings even when confidence is unwarranted. (3) Sycophancy — models optimized for human approval may learn to tell users what they want to hear rather than what is accurate, especially if raters rate agreeable responses more highly.

What LLMs Can and Cannot Do

6. This is hallucination, and it is a systematic property, not a bug. LLMs generate text by predicting what comes next based on statistical patterns. When asked for a citation, the model generates text that looks like a citation — plausible author names, a plausible title, a plausible journal — because that is what citations look like in the training data. The model has no mechanism to verify whether the citation actually exists; it only knows what plausible citation text looks like. This behavior cannot be fully eliminated by making the model bigger or training it longer, because it arises from the fundamental next-word-prediction architecture. Models can be improved with retrieval systems that check facts against real databases, but pure generation will always carry hallucination risk.
7. Two genuine capabilities: (1) LLMs can analyze text at scale and produce detailed, specific feedback on style, argument structure, clarity, and grammar — potentially faster and more consistently than a busy teacher with 30 essays. (2) LLMs can adapt their feedback to different skill levels and writing contexts if prompted appropriately. Two systematic limitations: (1) LLMs may hallucinate specific factual errors in the essay content without flagging them, or praise incorrect information confidently. (2) LLMs lack genuine understanding of the student's intent, background, or learning context — feedback may be technically accurate but pedagogically wrong for that student. Questions before adopting: Has the LLM been evaluated for accuracy and fairness on essays written by students similar to mine? Will students understand the feedback is AI-generated? Does the feedback replace or supplement teacher feedback? What is the data privacy situation for student writing submitted to the LLM?
8. Three things wrong or misleading: (1) "Knows everything" — LLMs have a training cutoff and do not know about events after that date; they also confabulate facts confidently, so what they "know" is unreliable without verification. (2) "Smarter than any human" — LLMs excel at language tasks that appear in text but fail at systematic reasoning, real-world grounding, mathematical proof, and many tasks humans find straightforward. Intelligence is not one-dimensional; an LLM that outperforms humans at certain text tasks fails completely at others. (3) The framing of knowing and understanding — LLMs do not "know" things in the sense of having beliefs or understanding; they predict statistically likely text. Plausible, confident output is not the same as accurate knowledge.