Computer Memory Hierarchy: Registers, Cache, RAM, and Virtual Memory

In the previous chapters, we built logic gates, arithmetic circuits, an ALU, and a Control Unit. Together, those components can process instructions, but they still need somewhere to find instructions, store operands, preserve intermediate results, and keep program data.

That is the job of the memory system.

A computer does not use one single type of memory. It uses a hierarchy of technologies with different trade-offs in speed, capacity, cost, energy use, and persistence. Registers are extremely fast but scarce. CPU caches are larger but still limited. Main memory provides gigabytes of working space, while SSDs and hard drives retain much larger amounts of data after power is removed.

This chapter explains the computer memory hierarchy, how CPU caches reduce memory latency, how virtual addresses are translated through the Memory Management Unit, and what actually happens during a page fault.

What You Will Learn

By the end of this chapter, you will understand:

• why computers need a memory hierarchy;
• how registers differ from cache and RAM;
• what L1, L2, and L3 cache do;
• why cache is faster than main memory;
• how cache hits, misses, and cache lines work;
• how RAM differs from SSD and hard-drive storage;
• what a memory address represents;
• why 64-bit does not always mean every address uses all 64 bits;
• what virtual and physical addresses are;
• how page tables, the MMU, and the TLB work together;
• what happens during a TLB miss;
• why a page fault does not always mean data must be loaded from an SSD;
• how the complete hierarchy affects program performance.

Why Computers Need a Memory Hierarchy

Ideally, a computer would have one enormous memory that is:

• as fast as a CPU register;
• as inexpensive as mass storage;
• as large as an SSD;
• persistent without power;
• accessible with almost no delay.

No current memory technology provides all of those properties at once.

Fast storage structures require chip area, power, and complex circuitry. Technologies designed for large capacity are generally slower. Persistent storage must also preserve data in ways that are different from the volatile circuits used near a CPU core.

Computer systems solve this problem by placing several levels between the processor and long-term storage.

At a high level:

1. Registers hold values being used directly by instructions.
2. CPU caches keep recently or nearby used blocks close to the cores.
3. RAM holds active programs and their working data.
4. Persistent storage holds files and programs when they are not actively executing.

Each level attempts to make the slower level appear less costly.

The Computer Memory Hierarchy

A simplified hierarchy looks like this:

Exact capacities and latencies vary widely between processors and systems.

The important pattern is:

• capacity generally increases farther from the CPU;
• access latency generally increases;
• cost per stored byte generally decreases;
• persistence appears at the storage level.

Persistent storage is often drawn below RAM in a memory-hierarchy diagram, but the CPU does not normally execute ordinary loads directly from an application file on an SSD. The operating system and storage system must first make the required data available through memory.

CPU Registers

Registers are small storage locations inside a processor core.

Instructions use registers to hold:

• integer operands;
• memory addresses;
• temporary values;
• function arguments;
• return values;
• vector data;
• control information;
• status flags.

Registers are directly named or implied by machine instructions. An addition instruction, for example, may read two registers and write its result to another register.

Because registers are integrated into the processor’s execution datapath, they can be accessed with very low latency. Their number is limited because large register files consume area, power, and routing resources.

Common Register Types

Processor architectures expose different register sets, but common categories include:

General-Purpose Registers

General-purpose registers, or GPRs, hold integers, addresses, and temporary values.

Examples include:

R0
R1
R2

or architecture-specific names such as x86-64’s RAX and RBX.

Program Counter

The Program Counter, also called an Instruction Pointer in some architectures, represents the address associated with the current or next instruction in the architectural execution model.

Pipelined processors may internally track several instruction addresses at once, but software still observes the program counter through the rules defined by the instruction set.

Stack Pointer

The Stack Pointer identifies the current position of the program stack.

The stack is commonly used for:

• function-call information;
• saved registers;
• local variables;
• temporary data.

Status or Flags Register

A status register stores condition information such as:

• zero;
• carry;
• signed overflow;
• negative result;
• interrupt state.

Not every architecture exposes the same flags.

Vector and SIMD Registers

Vector registers hold multiple smaller values in one wide register.

A single instruction can then operate on several integers or floating-point values at once. Depending on the architecture, vector registers may be 128, 256, 512, or another number of bits wide.

This is one reason it is inaccurate to say that every register in a 64-bit processor is exactly 64 bits wide.

See x86-64 Assembly Hello World with NASM on Linux if you want to experiment with registers in a small assembly program.

What Does 32-Bit or 64-Bit Mean?

The labels 32-bit and 64-bit summarize important properties of an architecture, but they do not describe only one measurement.

Depending on context, the label may refer to:

• the width of general-purpose integer registers;
• the natural integer width;
• the instruction-set operating mode;
• pointer width under a particular software ABI;
• the maximum virtual-address model;
• the width of arithmetic operations.

These values are related, but they are not always identical.

A 64-bit architecture may expose 64-bit general-purpose registers while implementing fewer than 64 virtual-address bits and fewer than 64 physical-address bits.

Using fewer implemented address bits reduces hardware complexity while still providing a very large address space.

Register Size Is Not the Same as Memory Capacity

A processor with 64-bit registers does not automatically support 2^64 bytes of installed RAM.

Several limits may apply:

• the architecture’s implemented physical-address width;
• the virtual-address width;
• the processor model;
• the motherboard;
• the memory controller;
• firmware;
• operating-system editions;
• the number and size of installed memory modules.

The theoretical number 2^64 is useful for understanding the scale of a 64-bit address, but it should not be treated as the amount of memory every 64-bit machine can use.

CPU Cache

A CPU cache is a small memory that keeps copies of instructions and data likely to be used by a processor core.

Cache exists because processor execution can proceed much faster than main memory can deliver arbitrary data. Without caching, the CPU would spend more time waiting for RAM.

Most general-purpose processors automatically manage their main CPU caches in hardware. Software normally accesses memory by address, while the cache hierarchy determines whether the requested block is already available near the core.

Why CPU Cache Is Faster Than RAM

CPU cache is faster than RAM for several reasons.

Cache Uses Fast On-Chip Memory

CPU caches are commonly built from SRAM-like structures optimized for low latency.

Main memory normally uses DRAM, which stores data more densely but requires different access and refresh behaviour.

Cache Is Physically Close to the Core

On-chip caches avoid much of the delay involved in communicating with external memory modules.

Main-memory requests must travel through the processor’s memory subsystem, memory controller, electrical links, and DRAM devices.

Cache Holds a Smaller Search Space

A cache stores far less data than RAM. Its structures are designed to find matching cache entries quickly.

Cache Is Designed Around CPU Access Patterns

Caches transfer and store blocks aligned to common memory-access behaviour. They also use parallel lookup logic, prefetching, and replacement policies to reduce waiting.

The result is a hierarchy in which a cache hit can be much less expensive than a main-memory access.

L1, L2, and L3 Cache

Modern CPUs often contain several cache levels.

The exact organization differs between processor families, so the labels describe hierarchy levels rather than one universal physical layout.

L1 Cache

L1 cache is normally the smallest and lowest-latency cache level.

It is commonly private to a core and divided into:

• L1 instruction cache, containing recently used instruction bytes;
• L1 data cache, containing recently used program data.

Separate instruction and data caches allow the processor to fetch instructions and access data efficiently.

L2 Cache

L2 cache is larger than L1 and normally has higher access latency.

On many processors, each core has a private L2 cache. Other designs may share L2 between a small group of cores.

L2 catches requests that missed in L1 before they reach a more distant cache or main memory.

L3 or Last-Level Cache

The next cache is often called L3 or the last-level cache.

It is commonly shared by several cores, but this is not universal. Some processors divide it into slices, share it within groups, or use another cache organization.

A shared last-level cache can reduce main-memory traffic and allow cores to access a larger combined cached working set.

Cache Layout Is Architecture-Specific

It is safer to say:

• L1 is usually closest to each core;
• L2 is larger and somewhat farther away;
• the last-level cache is larger again and may be shared.

Statements such as “L2 is always private” or “L3 is always shared by every core” are too absolute.

Cache Lines

Caches usually move data in fixed-size blocks called cache lines.

When the CPU requests one byte, the cache hierarchy may fetch an entire line containing that byte and its neighbours.

This supports spatial locality: programs often access nearby addresses close together.

For example, when a program reads elements from an array in order, one cache-line fill may provide several upcoming elements.

Cache-line size depends on the architecture and implementation. Software should not assume one value unless it is targeting a specific platform.

Temporal and Spatial Locality

Caches work well because programs often exhibit locality.

Temporal Locality

If a program accesses data now, it may access the same data again soon.

Examples include:

• a loop counter;
• a frequently used object;
• a function’s local variables;
• instructions inside a loop.

Spatial Locality

If a program accesses one address, it may soon access nearby addresses.

Examples include:

• sequential array elements;
• consecutive instructions;
• fields in a compact structure.

Programs with strong locality tend to use the cache hierarchy efficiently.

Cache Hits and Cache Misses

A cache hit occurs when the requested data is present in the cache level being checked.

A cache miss occurs when it is absent.

On a miss, the request continues to a lower level:

L1 -> L2 -> last-level cache -> main memory

If the data is found at a lower cache level, it is normally supplied to the core and may also be placed into higher levels.

A miss is not a failure. It is a normal part of cache operation. The performance cost depends on how far the request must travel.

Cache Replacement

A cache has limited capacity. When a new line must be inserted and the relevant area is full, an existing line is selected for eviction.

The hardware uses a replacement policy to choose a candidate.

Real cache policies are implementation-specific and may approximate ideas such as least-recently used replacement.

If an evicted line contains modified data under a write-back policy, it must be sent to a lower level before its cache storage can be reused.

Write-Through and Write-Back Caches

When the CPU stores data, the cache hierarchy must eventually keep lower memory levels consistent.

Two common policies are:

Write-Through

A write updates the cache and is also passed to the next level.

This can simplify consistency but increases lower-level traffic.

Write-Back

A write updates the cache and marks the line as modified, or dirty.

The modified line is written to a lower level later, often when it is evicted.

Modern high-performance caches commonly use write-back behaviour, although exact policies vary by level and processor.

Cache Coherence in Multicore CPUs

In a multicore processor, more than one core may cache a copy of the same memory line.

If one core modifies that line, the system must prevent another core from indefinitely using an outdated copy.

Cache coherence protocols coordinate ownership and visibility between participating caches.

Coherence does not make every memory operation instantly visible in one simple global order. Processor architectures also define memory-ordering rules, and multithreaded software must use synchronization correctly.

The important idea is that multicore caching requires extra communication to maintain a consistent shared-memory model.

Main Memory: RAM

RAM, or Random Access Memory, provides the main working storage used by active software.

It holds:

• program instructions;
• stack and heap data;
• operating-system structures;
• file-cache data;
• shared libraries;
• memory-mapped files;
• buffers used by devices.

RAM is much larger than CPU cache but has much higher access latency.

Why RAM Is Called Random Access

Random access means the system can address a memory location directly rather than reading all earlier data first, as with a sequential medium.

It does not mean every possible access takes exactly the same amount of time.

RAM latency can be affected by:

• whether the needed DRAM row is already open;
• bank conflicts;
• memory-controller scheduling;
• refresh operations;
• queueing;
• NUMA placement;
• data-transfer size.

The term distinguishes the addressing model, not perfectly uniform timing.

DRAM, SDRAM, and DDR

Main memory in general-purpose computers is usually based on DRAM.

DRAM

Dynamic RAM stores each bit in a small structure whose charge must be refreshed periodically.

Its high density makes it suitable for large main-memory capacities.

SDRAM

Synchronous DRAM coordinates transfers with a memory-interface clock.

This does not mean it operates at the CPU core’s internal clock frequency. The memory subsystem has its own timing and signaling rules.

DDR SDRAM

DDR means Double Data Rate.

DDR memory transfers data on multiple portions of the memory clock cycle. Generations such as DDR4 and DDR5 change many aspects of bandwidth, signaling, organization, and power behaviour.

Greater bandwidth does not automatically mean every individual memory access has proportionally lower latency.

Memory Bandwidth vs Latency

Latency is the delay before requested data becomes available.

Bandwidth is the amount of data that can be transferred over time.

A memory system can have high bandwidth while an individual random access still takes significant time.

This distinction matters because workloads behave differently:

• pointer-heavy code often depends strongly on latency;
• large streaming operations may depend more on bandwidth;
• parallel execution can hide some latency by keeping multiple requests in flight.

NUMA Memory

Large systems may use Non-Uniform Memory Access, or NUMA.

In a NUMA system, a processor or core can access all system memory, but some memory is physically closer to one processor package or node than another.

Local memory is generally less expensive to access than remote memory.

Operating systems and performance-sensitive applications may attempt to place threads and their data on the same NUMA node.

This is another reason a statement such as “every RAM address takes roughly the same time” is only a simplified introduction.

Persistent Storage

Persistent storage retains data after power is removed.

Common storage technologies include:

• solid-state drives;
• hard disk drives;
• removable flash storage;
• network storage.

Storage holds:

• operating-system files;
• applications;
• documents;
• databases;
• media;
• inactive program data.

HDDs and SSDs

Hard Disk Drive

An HDD stores data magnetically on rotating platters.

A mechanical head must move to the required region, making random access relatively slow. HDDs remain useful when large capacity and low cost per byte are more important than latency.

Solid-State Drive

An SSD stores data in non-volatile flash memory and contains no moving read/write head.

SSDs provide much lower random-access latency than HDDs, but they are still extremely slow relative to CPU caches and DRAM.

NVMe

NVMe is an interface designed for communication with non-volatile storage.

NVMe commonly operates over PCI Express for local SSDs, but NVMe is not simply a synonym for “an SSD connected directly to the CPU.” The physical path can vary by system, and NVMe also supports other transports.

Its command and queue model is designed to exploit the parallelism and lower latency of modern storage devices more effectively than older storage interfaces.

Storage Is Not Just Slow RAM

RAM and storage serve different purposes.

RAM is:

• byte-addressable by normal CPU load and store instructions;
• volatile;
• directly involved in the active memory system;
• optimized for frequent access.

Storage is:

• accessed through device and operating-system interfaces;
• non-volatile;
• organized into larger transfer units;
• optimized for persistence and capacity.

Memory-mapped files can make storage-backed data appear inside a process’s virtual address space, but the operating system still manages the movement of data between storage and physical memory.

How Programs Move from Storage to RAM

When you launch a program, the operating system creates a process and maps its executable code and data into a virtual address space.

The entire executable does not necessarily need to be copied into RAM immediately.

Instead, the system may load pages as they are first accessed. Frequently used code and data become resident, while untouched regions may remain absent.

File data may also be retained in the operating system’s page cache, reducing repeated storage access.

The phrase “the operating system loads the application into RAM” is therefore a useful summary, but actual loading is often demand-driven and page-based.

What Is a Memory Address?

A memory address is a number used to identify a location in an address space.

Most modern general-purpose architectures are byte-addressable, meaning each address identifies one byte.

A larger value occupies several consecutive byte addresses.

For example, a four-byte value beginning at address 1000 may occupy:

1000
1001
1002
1003

The byte order used to interpret those locations is called endianness.

Words and Data Widths

A word is an architecture-defined natural data unit, but the term is not perfectly consistent across all systems.

A processor may support operations on:

• bytes;
• halfwords;
• words;
• doublewords;
• quadwords;
• vectors.

A 64-bit CPU can still load and operate on 8-bit, 16-bit, and 32-bit values.

It is therefore safer to describe a processor’s supported operand widths explicitly rather than assuming every operation uses one “word size.”

Alignment

A multi-byte value is aligned when its address follows the preferred boundary for its size or instruction.

For example, an eight-byte value may be naturally aligned at an address divisible by eight.

Alignment can affect:

• performance;
• the number of cache lines touched;
• whether one or several memory transactions are needed;
• whether an instruction permits the access.

Some architectures support unaligned accesses directly. Others restrict them or handle them less efficiently.

Pointers

A pointer is a program value that represents an address.

Pointers allow programs to refer to:

• variables;
• arrays;
• objects;
• functions;
• dynamically allocated data.

The numerical value seen by an ordinary user-space program is usually a virtual address, not a direct physical RAM location.

Pointer size depends on the execution environment and ABI. It should not be inferred only from the installed RAM capacity.

Program Counter and Sequential Execution

Instructions occupy addresses in memory.

The architectural program counter identifies the current or next instruction position according to the ISA.

When execution proceeds sequentially, the processor advances to the following instruction. The amount added depends on the instruction encoding.

A branch, jump, call, return, interrupt, or exception can replace the next address with another value.

In a modern pipelined CPU, instruction fetching may run ahead and follow predicted paths, but the program still behaves according to the architecture’s defined control flow.

Address and Data Communication

Introductory diagrams often show:

• an address bus carrying where to access;
• a data bus carrying the transferred value;
• control signals indicating read or write.

This is a useful conceptual model.

Modern computers use more complex interconnects, integrated memory controllers, transaction queues, packetized links, caches, and coherence protocols. A memory request may not appear as one address placed on a single shared wire.

The logical distinction still remains:

• the request identifies a location;
• the system transfers data associated with that location;
• control information specifies the type and properties of the access.

Virtual vs Physical Addresses

Modern operating systems normally give each process a virtual address space.

The addresses used by a program are virtual addresses.

Physical addresses identify locations in the physical memory system.

A virtual address does not need to equal the physical address that ultimately stores the data.

The mapping between them allows the operating system to provide:

• process isolation;
• access permissions;
• flexible memory placement;
• sparse address spaces;
• shared memory;
• shared libraries;
• memory-mapped files;
• copy-on-write;
• demand paging.

Virtual memory is therefore much more than a mechanism for pretending that the computer has additional RAM.

Does Every Program See One Continuous Block?

A process often receives the illusion of a large private address space, but that space is not necessarily one fully allocated continuous block.

It commonly contains separate mapped regions for:

• executable code;
• read-only data;
• writable global data;
• heap allocations;
• shared libraries;
• memory-mapped files;
• thread stacks;
• operating-system interfaces.

Large ranges between mappings may be unused.

The physical pages backing nearby virtual addresses also do not need to be physically adjacent in RAM.

The Memory Management Unit

The Memory Management Unit, or MMU, is processor hardware that participates in translating virtual addresses and enforcing memory-access rules.

During a memory access, the translation system determines:

• which physical page backs the virtual page;
• whether the page is present or valid;
• whether reading is allowed;
• whether writing is allowed;
• whether execution is allowed;
• whether the access is permitted at the current privilege level;
• memory attributes used for caching or ordering.

The MMU works together with page tables created and maintained by the operating system.

Pages and Page Frames

Virtual memory is divided into blocks called pages.

Physical memory is managed in corresponding blocks often called page frames.

A common base page size is 4 KiB, but this is not universal. Architectures and operating systems may support:

• different base page sizes;
• large pages;
• huge pages;
• multiple page sizes at once.

Larger pages reduce the number of translation entries required for a region, but they also reduce allocation granularity and may waste more space.

Page Tables

A page table stores mappings and permissions for virtual-memory regions.

Conceptually:

Virtual page -> Physical page frame + attributes

The attributes may include:

• present or valid;
• readable;
• writable;
• executable;
• user-accessible;
• accessed;
• modified or dirty;
• caching properties.

Page tables are normally hierarchical rather than one enormous flat array.

A multilevel structure avoids allocating entries for every possible virtual page in a large sparse address space.

Splitting a Virtual Address

For a simple page translation, a virtual address can be viewed as:

Virtual page number | Page offset

The virtual page number identifies the mapping.

The offset identifies the byte within that page.

Translation replaces the virtual page number with a physical page-frame number while preserving the offset.

For a 4 KiB page:

4 KiB = 2^12 bytes

so the lowest 12 address bits form the page offset.

Correct Address-Translation Example

Consider this virtual address:

0x00403A10

Using 4 KiB pages:

Virtual page number = 0x403
Offset              = 0xA10

Suppose the page table maps virtual page 0x403 to physical page frame 0x1A2C.

The physical address is constructed as:

(0x1A2C << 12) | 0xA10

which produces:

0x1A2CA10

The page-frame number is shifted to make room for the 12-bit offset. It is not simply treated as an ordinary byte address before the offset is added.

The Translation Lookaside Buffer

Walking several page-table levels for every memory access would be expensive.

Processors therefore use a translation cache called the Translation Lookaside Buffer, or TLB.

A TLB entry can store recently used information such as:

• virtual page number;
• physical page frame;
• permissions;
• page size;
• address-space identifier.

On a TLB hit, the processor can obtain the cached translation quickly.

Some processors use separate instruction and data TLBs, multiple TLB levels, or specialized translation structures.

What Happens on a TLB Miss?

A TLB miss means the required translation is not currently cached in that TLB.

It does not automatically mean the virtual page is invalid or absent from RAM.

The processor must obtain the mapping from the page tables.

Depending on the architecture, this may involve:

• a hardware page-table walk;
• software-managed TLB refill;
• a combination of hardware and software.

If the page-table walk finds a valid permitted mapping, the translation can be inserted into the TLB and the access continues.

A TLB miss is therefore different from a page fault.

Page-Table Walks Also Use Memory

Page tables themselves are stored in memory.

A hardware page-table walker may need several memory accesses to traverse the hierarchy.

Caches can hold page-table entries, making many walks less expensive than going all the way to DRAM for each level.

Large pages can also reduce TLB pressure because one translation covers a larger address range.

This shows how virtual-memory performance and cache performance are connected.

What Is a Page Fault?

A page fault is an exception raised when a memory access cannot proceed under the current page-table mapping.

Possible causes include:

• the page has not yet been physically allocated;
• file-backed data has not yet been brought into memory;
• a page was moved out of the process’s resident set;
• a write targets a copy-on-write mapping;
• the access violates read, write, execute, or privilege permissions;
• the virtual address is not valid;
• a mapping needs operating-system assistance.

A page fault is not automatically an error, and it does not always require storage access.

Minor and Major Page Faults

Operating systems may classify faults differently, but a useful distinction is:

Minor or Soft Fault

The operating system can resolve the fault without reading the required page contents from persistent backing storage.

Examples include:

• allocating a fresh zero-filled page;
• completing copy-on-write from a page already in RAM;
• reusing a page that is resident but not currently mapped into the process;
• installing an existing shared mapping.

Major or Hard Fault

Resolving the fault requires storage I/O, such as reading:

• part of an executable;
• a memory-mapped file;
• data from paging or swap storage.

Major faults are far more expensive because storage latency is much greater than memory latency.

The exact terminology and accounting rules are operating-system-specific.

Page-Fault Handling Step by Step

A simplified valid demand-paging fault can proceed as follows:

1. A CPU instruction accesses a virtual address.
2. Translation cannot produce a usable mapping.
3. The processor raises a page-fault exception.
4. The operating system pauses the faulting thread.
5. The kernel examines the address and access type.
6. It determines whether the access belongs to a valid mapped region.
7. It resolves the fault by allocating, mapping, copying, or loading a page.
8. It updates the page table.
9. It invalidates or updates stale translation information as required.
10. The faulting instruction is restarted.
11. Execution continues as though the access had succeeded normally.

If the address is invalid or the permissions cannot be satisfied, the operating system reports an error to the process instead of resuming it normally.

Copy-on-Write

Copy-on-write, or COW, allows mappings to share the same physical page until one process attempts to modify it.

Initially, the page is mapped read-only into multiple address spaces.

When a process writes:

1. a protection fault occurs;
2. the operating system allocates another page;
3. it copies the original data;
4. it maps the new page as writable for the process;
5. the write is retried.

Copy-on-write is used in process creation, memory sharing, and other operating-system optimizations.

It demonstrates why a page fault can occur even when the original data is already in RAM.

Paging, Swap, and Memory Pressure

When physical memory becomes scarce, an operating system may move or discard some pages to make room.

Clean file-backed pages can often be discarded because their contents can be read from the original file again.

Modified anonymous pages may require backing storage such as:

• a swap partition;
• a paging file;
• another operating-system-specific mechanism.

If the page is needed later, accessing it can generate a major page fault.

Excessive paging can severely reduce performance because the system repeatedly waits for storage. This condition is commonly associated with thrashing.

Virtual Memory Does Not Guarantee Unlimited Memory

Virtual address space can be much larger than installed RAM, but that does not provide unlimited usable memory.

The system still needs backing resources and bookkeeping.

Allocation can fail because of:

• address-space limits;
• commit limits;
• insufficient RAM or backing storage;
• process limits;
• operating-system policy;
• fragmentation of required resources;
• exhausted kernel structures.

Virtual memory provides flexible mapping and abstraction, not infinite physical capacity.

Shared Memory and Shared Libraries

Virtual memory allows different processes to map the same physical pages at different virtual addresses.

This is useful for:

• shared libraries;
• inter-process communication;
• shared file mappings;
• read-only executable code.

Sharing reduces duplicate memory use and can allow efficient communication.

Permissions still control what each process may do with the mapping.

Memory Protection

Page-table permissions help isolate software.

A page can be marked:

• readable;
• writable;
• executable;
• accessible only in privileged mode;
• inaccessible.

A common security policy attempts to avoid pages being writable and executable at the same time.

If a process violates a mapping’s permissions, the processor raises a fault and the operating system decides how to respond.

Instruction Fetches Also Use the Memory System

The memory hierarchy is not used only for program data.

Instruction fetching also involves:

• virtual-address translation;
• instruction TLBs;
• instruction caches;
• lower cache levels;
• main memory;
• page faults when executable code is not resident or permitted.

This is why cache and TLB behaviour can affect both the instructions a program executes and the data those instructions access.

The Hardware and Operating-System Paths

It helps to separate two connected ideas.

Hardware Access Path

For a normal mapped access, the processor may use:

Register -> L1 cache -> L2 cache -> Last-level cache -> RAM

Address translation is accelerated by the TLB and page-walk caches.

Operating-System Backing Path

If the access requires operating-system help, data may be supplied from:

Zero-filled allocation
Shared resident page
Executable or mapped file
Paging or swap storage

An SSD is therefore not simply checked after an L3 cache miss. A normal cache miss requests data from physical memory. Storage enters the process when software and virtual-memory mechanisms arrange for data to be loaded into RAM.

Why the Memory Hierarchy Affects Performance

Processor cores can execute many instructions while one long-latency access is outstanding.

Performance depends heavily on whether programs:

• reuse data while it remains cached;
• access memory sequentially;
• avoid unnecessary cache-line transfers;
• fit their active working set into cache;
• minimize TLB misses;
• avoid frequent major page faults;
• place NUMA data near the threads using it;
• expose enough independent work to hide latency.

A program can perform few arithmetic operations yet still run slowly because it spends most of its time waiting for data.

This is often called being memory-bound.

Common Memory-Hierarchy Misconceptions

Every Register in a 64-Bit CPU Is 64 Bits Wide

Processors can contain flags, control registers, vector registers, and specialized registers with many different widths.

A 64-Bit CPU Can Always Use 2^64 Bytes of RAM

Real architectures implement limited virtual- and physical-address widths, and complete systems impose additional limits.

L2 Is Always Private and L3 Is Always Shared

Cache organization varies between processor designs.

RAM Access Always Takes the Same Amount of Time

DRAM state, memory-controller scheduling, NUMA placement, and contention can change latency.

NVMe Means the SSD Is Directly Connected to the CPU

NVMe defines communication with non-volatile storage across supported transports. The system’s actual PCIe and chipset path varies.

A TLB Miss Is a Page Fault

A TLB miss usually triggers a translation lookup. A page fault occurs only when the access cannot proceed with the current mapping and permissions.

Every Page Fault Loads Data from Disk

Many faults are resolved using pages already in RAM, zero-filled allocation, or copy-on-write.

An L3 Miss Causes the CPU to Read an SSD

A normal last-level cache miss continues toward physical memory. Storage is involved only through operating-system and I/O mechanisms.

Virtual Memory Is Only Extra RAM on Disk

Virtual memory also provides isolation, permissions, sharing, sparse address spaces, relocation, and memory-mapped files.

Frequently Asked Questions

What is the computer memory hierarchy?

The memory hierarchy is the layered system of registers, CPU caches, main memory, and persistent storage. Each level trades capacity, latency, cost, and persistence.

Why is CPU cache faster than RAM?

Cache uses low-latency on-chip structures located close to the processor core. It is smaller and designed specifically to serve common CPU access patterns quickly.

What is the difference between L1, L2, and L3 cache?

L1 is normally the smallest and closest cache. L2 is larger and slower. L3 or the last-level cache is larger again and is often shared, although the exact layout depends on the processor.

What is the difference between cache and RAM?

Cache holds selected copies of instructions and data near the CPU. RAM contains a much larger active working set and is accessed when data is not available in cache.

What is the difference between RAM and storage?

RAM is volatile, byte-addressable working memory used directly by normal CPU memory operations. Storage is non-volatile and accessed through device and operating-system interfaces.

What is virtual memory?

Virtual memory gives processes controlled address spaces that are translated to physical memory. It provides isolation, permissions, flexible placement, sharing, and demand paging.

What does the MMU do?

The MMU participates in translating virtual addresses, checks page permissions, and enforces memory-access rules using operating-system-managed page tables.

What is a TLB?

A TLB is a cache of recent virtual-to-physical address translations and related access attributes.

What happens on a TLB miss?

The processor obtains the translation from the page-table hierarchy. If a valid mapping exists, it can cache the translation and continue.

What causes a page fault?

A page fault occurs when an access requires operating-system assistance or violates the current mapping. Causes include demand allocation, file loading, copy-on-write, missing residency, invalid addresses, and permission violations.

Does a page fault always mean the computer is low on RAM?

No. Page faults are normal and can occur during first access, file mapping, copy-on-write, and other routine operations. Frequent major faults may indicate memory pressure or an I/O-heavy workload.

Why are 4 KiB pages common?

A 4 KiB page provides a compromise between allocation granularity and translation overhead. Other page sizes are also supported by many systems.

Can a CPU access an SSD with a normal load instruction?

Ordinary loads address mapped memory. Storage data must be made available through device I/O or a memory mapping managed by the operating system.

Key Takeaways

The memory hierarchy allows a computer to combine fast access with large capacity.

The central ideas are:

1. Registers hold values used directly by CPU instructions.
2. CPU caches keep recently or nearby used blocks close to the cores.
3. L1, L2, and last-level cache organization varies by processor.
4. Cache lines exploit spatial locality, while repeated access exploits temporal locality.
5. Main memory uses dense DRAM technology and has much higher latency than cache.
6. Persistent storage retains files but is not directly equivalent to slow RAM.
7. A 64-bit architecture does not necessarily implement all 64 address bits.
8. Programs normally use virtual addresses rather than direct physical addresses.
9. Page tables map virtual pages to physical page frames and store permissions.
10. The TLB caches translations so most accesses avoid a complete page-table walk.
11. A TLB miss is not the same as a page fault.
12. Page faults can be resolved through allocation, sharing, copying, file loading, or paging.
13. Major page faults require storage I/O and are much more expensive than ordinary memory accesses.
14. Program performance depends heavily on cache locality, translation behaviour, memory bandwidth, and latency.

We now have the storage structures needed to hold instructions, operands, and results. In the next chapter, we can bring the ALU, Control Unit, registers, and memory system together and follow an instruction through the complete fetch-decode-execute process.