What LLMs Genuinely Do Well
It would be a mistake to leave this topic only with a list of warnings. Large language models represent a genuine technological achievement, and their capabilities are real and useful when understood and applied appropriately.
Fluent text generation across a remarkable range of styles and domains
LLMs can produce coherent, well-structured text on almost any topic, in almost any style, at a quality level that would have been considered impossible five years ago. Draft a formal letter, explain a concept for a ten-year-old, write in the style of a Victorian naturalist, summarize a complex document, generate ten variations on a sentence — these are tasks that LLMs handle reliably and quickly.
Flexible language understanding and transformation
LLMs can paraphrase, translate, simplify, expand, classify, extract, and reformat text with impressive flexibility. They can take a dense academic abstract and produce a plain-language summary, identify the main argument of an essay, or convert unstructured notes into organized bullet points. These transformation tasks are where LLMs add the most unambiguous value.
Brainstorming and ideation
Because LLMs have been trained on a vast range of human thought, they are excellent brainstorming partners. They can generate many plausible options quickly, surface approaches that a user might not have considered, and help break through creative blocks. The key is treating the output as a starting point for human judgment rather than a final answer.
Code generation and explanation
LLMs trained on large code corpora (GitHub, Stack Overflow, documentation) can generate, explain, and debug code across many programming languages. For many routine coding tasks, modern LLMs produce working code on the first attempt. This capability is already transforming software development.
Personalized explanation at any level
One of the most educationally significant capabilities: LLMs can explain almost any concept at any level of sophistication on demand. Explain quantum entanglement to a middle schooler. Explain it again more simply. Now use an analogy involving Lego bricks. This adaptive explanation capability is something no textbook can provide.
The Systematic Failure Modes
None of the capabilities above come without important caveats. LLMs have failure modes that are not random bugs but systematic properties of how they work. Understanding them is not optional for anyone deploying these systems or teaching students to use them.
Hallucination
Hallucination is the tendency of LLMs to generate plausible-sounding but false information, stated with confidence. A model asked about a historical event might invent plausible-sounding dates. Asked to cite sources, it might generate realistic-looking citations to papers that do not exist. Asked about a person, it might confidently assert facts that are entirely fabricated.
Hallucination is not a bug that will be fixed with more training data or a larger model. It is a systematic consequence of how LLMs work: they generate text by predicting what comes next based on statistical patterns, not by retrieving facts from a verified database. The model has no way to distinguish between a memory of something it read and a plausible-sounding continuation it is generating in the moment. It produces both with equal fluency.
This has profound implications for educational use. An LLM can produce factually wrong answers that are indistinguishable in tone and confidence from correct ones. Students who treat LLM output as a factual source rather than a draft requiring verification are at serious risk of learning incorrect information.
The Knowledge Cutoff
LLMs are trained on data collected up to a specific date. After that date, they know nothing — their training is complete, and new events in the world do not update their weights. When you ask an LLM about recent events, it may hallucinate plausible-sounding but fabricated recent news, or it may correctly acknowledge its cutoff, depending on how it was trained to handle such questions.
Some deployed LLMs are augmented with web search tools that retrieve current information and incorporate it into responses. This helps with recency, but it introduces a different kind of uncertainty: the quality of the retrieved information depends on what is retrieved, and errors in retrieval propagate into the response.
Sycophancy
As described in Reading 2, RLHF training can produce models that tell users what they seem to want to hear. If a student presents a clearly incorrect argument and asks the LLM to evaluate it, a sycophantic model may agree with the argument or offer only mild caveats rather than clearly identifying the error. If a user pushes back on a correct answer, a sycophantic model may change its answer to accommodate the pushback.
Sycophancy is particularly dangerous in educational contexts. A student using an LLM to check their understanding may receive validation of incorrect beliefs rather than honest correction.
No Persistent Memory
Within a single conversation, an LLM remembers everything said so far. Between conversations, it remembers nothing. The model has no ongoing relationship with any user, no accumulation of knowledge about a student's progress, and no memory of previous sessions. Each conversation starts fresh. (Some products add external memory systems on top of the model to simulate persistence, but the underlying model itself has no long-term memory.)
No Genuine Understanding of Its Own Limits
An LLM cannot reliably tell you when it does not know something. It has no accurate model of its own knowledge or confidence. It can be trained to express uncertainty more often, and modern models are better calibrated than earlier ones — but they still frequently produce confident output in areas of genuine uncertainty, and hesitant output in areas where they are reliably correct. The uncertainty markers in an LLM's output are themselves learned patterns, not genuine epistemic assessments.
Sensitivity to Prompt Wording
Small changes in how a question is phrased can produce dramatically different responses. This brittleness is a consequence of the statistical nature of the model: it is sensitive to the exact pattern of the input because that pattern determines which learned associations are activated. Two students asking about the same topic with slightly different phrasing may receive significantly different answers — with no systematic guarantee about which one is more accurate.
What This Means for Teachers
LLMs are already in your students' lives. They will be in your students' professional lives. The question is not whether to engage with them but how to engage with them wisely — and how to help students do the same.
Questions Worth Asking Before Adopting Any LLM Tool
- What is this system being used for, and are LLM capabilities a good fit for that use?
- What happens when it is wrong? Is there a human review process before its output affects students?
- What data is being collected about student interactions, and how is it used?
- Has the vendor been transparent about the system's limitations, or only its capabilities?
- Are students learning with the tool or being replaced by it? Does the tool develop student skills or substitute for them?
What Students Most Need to Understand
Most students interact with LLMs believing one or more of the following: that the system "knows" things in the way a person knows them; that confident output is likely accurate; that the system is neutral and objective; and that using it extensively is equivalent to learning the material.
All four beliefs are wrong in ways that matter. The most important conceptual shift a student can make is from "this system knows the answer" to "this system generates plausible text." That shift does not make LLMs useless — it makes them useful in a more precise and safer way.
What You Can Say Now That You Could Not Before
When a student asks "How does ChatGPT work?" you can now give an accurate answer: it is a transformer-based neural network trained on hundreds of billions of words of text using next-word prediction, then fine-tuned with human feedback to behave helpfully. It generates text by predicting what comes next, guided by patterns learned from its training data. It does not look things up. It does not reason in the way you do. It has no knowledge of whether what it produces is true.
That answer is not a condemnation of the technology. It is an accurate description of a remarkable tool with genuine capabilities and genuine limits. Students who hold that accurate model will use these systems more effectively, more safely, and with more appropriate skepticism than students who treat them as oracles.
Full Circle
At the beginning of Week 6, you wrote down what you believed about artificial intelligence before we taught you anything about it. Somewhere in that response, you probably mentioned ChatGPT, or a chatbot, or voice assistants, or some other LLM-adjacent technology. That was reasonable — those are the most visible AI systems in the world right now.
What you now know is that they are one corner of a very large field. You have studied agents and search, decision trees, the three learning paradigms, perceptrons and neural networks, clustering and dimensionality reduction, and finally the architecture and training process of the systems that generate those strikingly fluent responses. You have seen how each piece connects to the others and where each one fits in the broader picture.
At the end of this week, you will return to your pre-reflection and write about what has shifted. That exercise is not a formality. Looking at what you believed before and what you believe now — and being precise about what changed and what did not — is one of the most honest forms of learning there is.