Attacks from Within
Reading 1 addressed the threat from outside: unauthorized users trying to gain access to the system. But there is a second, equally serious threat: what happens after someone is inside.
An attacker who gains access to an ordinary user's account has limited options — they can only do what that account is permitted to do. But that does not mean they are harmless. Their next goal is usually to escalate privileges: to trick the OS into granting them access beyond what their account allows. Common tactics include:
- Tricking the memory manager into allowing a process to access memory outside its allocated area — which might contain data belonging to other processes, or even the OS itself
- Tricking the file manager into providing access to files the account is not permitted to read or modify
- Exploiting flaws in OS software to gain administrator-level control
These are not hypothetical scenarios. They describe the mechanics of most real-world security exploits. The OS cannot rely solely on software checks to prevent them — software can have bugs, and a sufficiently clever attack may find a way around a purely software-based defense. This is why modern security is enforced at the hardware level.
Memory Bounds: The Hardware Defense
Recall from Topic 3a that the memory manager assigns each process its own region of RAM. A well-behaved process stays within that region. A malicious or buggy process might try to read from or write to memory outside its allocated area — accessing another process's data, modifying OS code, or worse.
To prevent this, CPUs designed for multiprogramming include special-purpose registers that store the upper and lower memory limits of the currently executing process's allocated area. Every time the process references a memory address, the CPU automatically compares that address to these limit registers. If the address falls outside the permitted range, the CPU does not execute the memory access — it generates an interrupt and transfers control back to the OS.
This hardware check happens automatically, at the speed of the CPU, on every single memory access. It cannot be bypassed by software running in the process being monitored, because the check happens in hardware before the software's instruction is allowed to complete.
But there is a problem: what if a malicious process simply changes the limit registers to expand its own permitted memory range? Without additional protection, a process could write a new upper limit into the register and then freely access memory it was never supposed to touch.
This is exactly the problem that privilege levels solve.
Privilege Levels
Modern CPUs operate in one of (at least) two privilege levels:
| Mode | Who Uses It | What Is Permitted |
|---|---|---|
| Privileged mode (also called "kernel mode" or "supervisor mode") |
The operating system | All instructions, including those that change memory limit registers, modify privilege levels, and access hardware directly |
| Nonprivileged mode (also called "user mode") |
Application and utility software — ordinary processes | A restricted set of instructions. Certain sensitive instructions are simply not available. |
The instructions that are only available in privileged mode are called privileged instructions. Classic examples include:
- Instructions that modify the memory limit registers
- Instructions that change the current privilege level of the CPU
- Instructions that directly access certain hardware components
- Instructions that disable interrupts
If a process running in nonprivileged mode attempts to execute a privileged instruction, the CPU does not execute it. Instead, it generates an interrupt — exactly like the time-slice interrupt from Topic 3b — and transfers control to an interrupt handler in the OS. The OS then decides how to respond: typically, this means terminating the offending process.
How the CPU Transitions Between Modes
When a computer first powers on and boots, the CPU starts in privileged mode. This makes sense: the boot loader and the OS itself need unrestricted access to hardware to set the machine up. While the OS is running and managing the system, it operates in privileged mode.
Each time the dispatcher hands a time slice to an ordinary process, it executes a "change privilege mode" instruction (itself a privileged instruction) that switches the CPU to nonprivileged mode before the process begins. From that moment, the process runs with restricted capabilities. It can do ordinary computation, but it cannot modify its own memory limits, change privilege levels, or access hardware directly.
When the time slice ends — triggered by the timer interrupt — control transfers back to the OS's interrupt handler, which runs in privileged mode. The OS saves the process's state, selects the next process, and hands over another time slice, again switching the CPU back to nonprivileged mode.
The security insight: No ordinary process ever runs in privileged mode. Every time a process is executing, the CPU is in nonprivileged mode, and the dangerous instructions are locked away. The only code that ever runs in privileged mode is OS code — code that has been deliberately written and installed by the system administrator. A process cannot promote itself to privileged mode because "change privilege mode" is itself a privileged instruction. The lock cannot be opened from the inside.
What Can Go Wrong
Privilege levels and hardware memory enforcement are powerful protections. But they are not invulnerable. Consider what becomes possible if any of these protections fail or are circumvented:
- If a process can modify the timer that controls time slices, it can extend its own time slice indefinitely and effectively monopolize the CPU, starving all other processes.
- If a process can access peripheral devices directly (bypassing device drivers and the file manager), it can read files it is not permitted to access, send data over a network connection without OS supervision, or write directly to a disk without permission checks.
- If a process can access memory outside its allocated area, it can read data belonging to other processes (including sensitive data like passwords or encryption keys), modify that data, or overwrite OS code to take control of the system entirely.
These are not theoretical scenarios. Security vulnerabilities that allow privilege escalation — where a process in nonprivileged mode gains privileged capabilities — are among the most serious and sought-after exploits. When such a vulnerability is discovered, OS vendors release emergency patches. The Spectre and Meltdown vulnerabilities from the Week 2 SEC material were exactly this kind of flaw: they allowed processes to read memory they were not supposed to access, by exploiting the speculative execution mechanism in a way the CPU designers had not anticipated.
The fundamental tension: Security and functionality are in permanent tension. Every restriction that prevents malicious behavior also prevents some legitimate behavior. Every time the OS requires a process to ask permission before accessing a resource, it adds overhead. Every additional privilege check is another place where a design flaw can be introduced. OS security is not a problem that gets solved — it is an ongoing engineering discipline, and errors continue to be found in every major OS.
Closing: Security as a System Property
The security measures covered across these two readings — accounts, passwords, MFA, auditing software, memory bounds registers, privilege levels, privileged instructions — are not independent features. They form a layered defense.
The login procedure keeps unauthorized users out. Privilege levels and hardware memory enforcement contain authorized users (and the processes running under their accounts) within permitted boundaries. Auditing software watches for anomalies that suggest one or both layers have been breached. The administrator has the authority to respond when they are.
No layer is individually sufficient. An attacker who breaks through the login barrier faces privilege level enforcement. A vulnerability in privilege level enforcement may still be caught by auditing software. And good password practices and MFA make breaking through the first layer dramatically harder in the first place.
For K-12 teachers, this layered model is worth communicating to students: security is not one thing that either works or does not work. It is a set of overlapping protections, and its strength comes from the combination.