Virtual Memory & the MMU: Teaching the CPU to Lie

Here is an uncomfortable truth about the kernel you have built so far: it has no memory protection. Right now, a single misdirected pointer in kernel_main could write to the UART registers, corrupt the stack, or overwrite the first instruction of our boot stub. The hardware would not stop it. QEMU’s virt machine would simply shrug. You might not even notice, until something breaks silently three function calls later, and you will spend an afternoon staring at a frozen serial output, wondering where you went wrong.

The Memory Management Unit prevents that. It is a hardware translation engine built into every modern processor, and its job is deceptively simple: intercept every memory access your code makes, consult the tables you provide, and either translate the address or raise a fault. The tables, orx so-called page tables, are yours to build. The hardware performs the lookup on every access, with no software overhead once they are configured. This post shows you how to build and configure those tables and then flip the switch.

Enabling the MMU is probably the hardest milestone in this series. The ARM Architecture Reference Manual’s chapter on the AArch64 translation table format runs to over a hundred pages. We are going to cover the 5% of those pages. We will discuss what matters most for getting our kernel running with protected memory. We are going to do it in a way that makes it feel approachable and helps you get comfortable applying it next time you need it.

What We’re Building Today

By the end of this post, the MMU is on. The kernel will have an identity-mapped address space where the virtual addresses equal physical addresses, so the kernel does not need to know about the mapping. The mappings will be protected by three levels of page tables. We will map the kernel region as normal cached memory and the MMIO region (which includes the UART) as device memory, telling the processor it may not reorder or merge those accesses.

Right now, make run will still print Hello, Kernel!. But now every memory access goes through the MMU hardware.

Here is the physical address space we are mapping. It’s the same layout we established in the previous post, but now described in page table terms:

And here is what the virtual address space looks like after we have enabled the MMU. Today it is an identity map, so it looks identical, but from this point forward, those addresses are virtual, and the MMU is actively translating every single one:

Why Virtual Memory?

Before I walk you through 50 lines of bit manipulation, let me give you three concrete reasons why all this matters.

• Memory protection. Right now, a bug in kernel code can write to any address in the system. The MMU lets you mark .text and .rodata as read-only so that an accidental store raises a fault rather than silently corrupting code or constants. Stack and heap regions can be marked no-execute, preventing an attacker from jumping into stack-sprayed shellcode. Device memory gets attributes that prevent the CPU from buffering or reordering register writes, which, as we discovered in Post 4, is exactly what the volatile keyword was trying to express in software. The MMU enforces it in hardware.
• Process isolation. When we reach the post discussing process scheduling, each process must believe it owns the entire address space. With the MMU, each process has its own set of page tables, so address 0x40000000 points to different physical pages for different processes. Without the MMU, multitasking means manually partitioning physical RAM between every process and hoping nothing overlaps. With it, hardware enforces isolation on every memory access.
• A fixed kernel address space. Higher-half kernels, in which the kernel always resides in the upper virtual address range (e.g., 0xFFFF_0000_4000_0000) regardless of its physical location, are a direct consequence of virtual memory. We will use an identity map today and introduce the kernel/user address space split in a future post, when processes arrive.

The AArch64 Address Space Split

AArch64 enforces a strict, hardware-level separation of the virtual address space based on the most significant bits of an address. This isn’t just a suggestion or an OS convention; the MMU hardware literally splits the world in two.

The hardware looks at the top 16 bits of every virtual address. Depending on what it sees, it chooses one of two “Base Registers”:

• The Lower Half (TTBR0_EL1): If bits [63:48] are all zeros, the MMU uses the table pointed to by TTBR0_EL1. This covers the range 0x0000_0000_0000_0000 to 0x0000_FFFF_FFFF_FFFF. This is where User-Space applications live.
• The Upper Half (TTBR1_EL1): If bits [63:48] are all ones, the MMU uses the table pointed to by TTBR1_EL1. This covers the range 0xFFFF_0000_0000_0000 to 0xFFFF_FFFF_FFFF_FFFF. This is reserved for the Kernel.

What happens if an address starts with a mix of ones and zeros (e.g., 0x5555...)? The MMU triggers a Translation Fault immediately. There is no third option and no “middle ground.” This “canonical hole” prevents user applications from even attempting to guess or “drift” into kernel memory. It provides a clean, architectural firewall that the OS doesn’t have to manually check at runtime.

For our PurgatoryOS, we are keeping things simple. Our kernel is loaded at 0x4000_0000. Since the top 16 bits of this address are all zero, it falls squarely into the lower half.

• We will configure TTBR0_EL1 to handle our mapping.
• We will leave TTBR1_EL1 at its reset value (effectively disabled).

While most “real” kernels live in the upper half, starting in the lower half allows us to get our MMU up and running with a single translation table. We will perform the “Great Migration” to the upper half in a future post when we introduce user-space processes.

Here is the state of the four system registers that control the MMU, after mmu_init() runs:

Four Levels Down: The Translation Walk

With a 4 KB granule and a 48-bit address space, the translation process is remarkably symmetrical. A single 4 KB table can hold exactly 512 entries (since each entry is 8 bytes). To map 48 bits, the hardware “slices” the virtual address into 9-bit chunks, each corresponding to a level in the page table tree. In a typical setup, the hardware decodes your virtual address like this:

Every memory access that misses the Translation Lookaside Buffer (TLB) triggers a “walk” through this tree. On real hardware, every extra level in the tree adds latency, which is why we often look for shortcuts.

Because our kernel is currently simple, we don’t need the fine-grained detail of 4 KB pages (Level 3). Instead, we terminate the walk early using Block Descriptors. This reduces the “pointer chasing” the hardware has to do. Here is how our specific walk behaves:

1. L0 (The Root): The hardware reads TTBR0_EL1 to find the L0 table and uses bits [47:39] to find the first entry.
2. L1 (The 1 GB Shortcut): We use bits [38:30] to index the L1 table. For the 0x0 region (Index 0), we provide a Block Descriptor. The walk stops here, mapping a massive 1 GB block in one go. For the Kernel region (Index 1), we provide a Table Descriptor that points the hardware to an L2 table.
3. L2 (The 2 MB Destination): The hardware uses bits [29:21] to index the L2 table. Here, we provide 2 MB Block Descriptors. The walk terminates.
4. Final Physical Address: The hardware takes the base address from the L2 block and adds bits [20:0] (the remaining offset) to find the exact byte.

We skip L3 entirely. L3 tables give you 4 KB granularity, which is what you need for fine-grained memory protection. We will revisit that when we add processes. For now, 2 MB blocks are more than adequate.

Try the interactive walk below. Type any virtual address your kernel might use, or pick one of the examples, and see exactly which entry in each table the hardware selects:

What’s Inside a Page Table Entry

Every page table entry is a 64-bit integer. At L2, we use block descriptors, which look like this (I am showing only the bits we actually set. The rest stay zero):

/* Bits [1:0] — descriptor type */

#define DESC_VALID      (1UL << 0)   /* bit[0]=1: descriptor is live        */

#define DESC_TABLE      (1UL << 1)   /* bit[1]=1: table (next-level ptr)    */

#define DESC_BLOCK      (0UL << 1)   /* bit[1]=0: block (terminal entry)    */

/* Bit [10] — Access Flag (AF) */

#define DESC_AF         (1UL << 10)  /* must be 1 or the MMU faults         */

/* Bits [9:8] — Shareability (SH) */

#define DESC_SH_INNER   (3UL << 8)   /* inner shareable — normal memory     */

#define DESC_SH_NONE    (0UL << 8)   /* non-shareable  — device memory      */

/* Bits [7:6] — Access Permissions (AP) */

#define DESC_AP_RW_EL1  (0UL << 6)   /* EL1 r/w, EL0 no access             */

/* Bits [4:2] — Memory Attribute Index (AttrIndx) */

#define DESC_ATTR(idx)  ((uint64_t)(idx) << 2)  /* 3-bit index into MAIR   */

/* Bits [54,53] — Execute-Never */

#define DESC_PXN        (1UL << 53)  /* privileged execute never            */

#define DESC_UXN        (1UL << 54)  /* unprivileged execute never          */

The most important bits in practice are the Access Flag (forget it, and you get a fault), the AttrIndx (get it wrong and the CPU treats RAM like device registers or vice versa), and the Shareability (get it wrong and cache coherency breaks in ways that are extraordinarily hard to debug on multi-core hardware).

MAIR_EL1: The Memory Attribute Menu

The Memory Attribute Indirection Register (MAIR_EL1) acts like a 64-bit “menu” for your CPU. Instead of cramming complex caching rules into every single Page Table Entry, we define a menu of up to eight rules here. Each PTE then simply carries a 3-bit index to pick its preferred “flavour” of memory.

Each menu slot is one byte, split into two 4-bit halves. The Upper 4 bits define the “Outer” cache policy (system-level), and the Lower 4 bits define the “Inner” cache policy (local CPU level).

• For standard RAM, we use 0xFF or the fully spelt-out 0b1111_1111. This translates to Write-Back, Read-Write. The CPU is allowed to cache data, buffer writes lazily, and aggressively pre-allocate space in the cache. This is essential for speed; without it, every memory access would be throttled by the speed of your RAM chips.
• For hardware registers (like the UART), we use 0x00 or the fully spelt-out 0b0000_0000. This is known as Device-nGnRnE. The nG means (non-Gathering), indicating no merging multiple small writes into one. The nR (Non-Reordering) says that operations must occur in the exact order specified by the code. The final nE (No Early) says that the write isn’t “done” until the hardware actually receives it. All these settings are important because if you mapped a UART as “Normal” memory, the CPU might reorder your writes or keep them in the cache. Your uart_putc might print “Hello” as “olleH,” or worse, never send the characters to the screen at all.

We pack these two rules into the first two slots of the MAIR_EL1 register. Note that the register is filled from the right (Byte 0 is the least significant byte).

#define MAIR_NORMAL_WB  0xFFUL   /* Attr0: Normal write-back cacheable   */
#define MAIR_DEVICE_NG  0x00UL   /* Attr1: Device-nGnRnE                 */

/* Result: 0x000000000000FF00 ... wait, actually 0x00000000000000FF when
   Attr0 is in byte 0 and Attr1 is in byte 1 → 0x000000000000FF  with
   MAIR_DEVICE_NG << 8 = 0x0000  → MAIR_VALUE = 0x00FF             */
#define MAIR_VALUE      ((MAIR_DEVICE_NG << 8) | MAIR_NORMAL_WB)

When the MMU performs a table walk, it finds the index in the descriptor:

1. BLOCK_NORMAL descriptors use index 0. The hardware looks at MAIR_EL1[7:0], sees 0xFF, and enables full caching.
2. BLOCK_DEVICE descriptors use index 1. The hardware looks at MAIR_EL1[15:8] and sees 0x00, then follows strict device rules.

This indirection saves space in your page tables while giving you global control over how the CPU treats different regions of physical space.

TCR_EL1: Tuning the Translation Engine

If the page tables are the map, the Translation Control Register (TCR_EL1) is the GPS configuration. It tells the hardware how to read those maps, defining the size of the virtual world, the size of each “tile” (page), and how to cache the map for speed. The configuration fields are:

1. Address Space Size (T0SZ). Modern 64-bit ARM doesn’t usually use all 64 bits for addresses (which would be a staggering 16 exabytes). Instead, we define a smaller “window”. The size of our “window” is defined as $2^{(64−T0SZ)}$. We want a 48-bit address space, so we set $T0SZ=16 (64−16=48)$. This allows us to address up to 256 Terabytes, which is plenty for our kernel!
2. The Translation Granule (TG0). This defines the “base unit” of our memory. We use 4 KB (encoded as 0b00). This choice is the “anchor” for our entire table walk; it determines that each table level has 512 entries and dictates exactly which bits of the Virtual Address correspond to which level of the walk.
3. Caching the Walk (IRGN0 & ORGN0). Even with a TLB, the hardware still has to “walk” the tables in RAM occasionally. To make this faster, we can tell the MMU to cache the page table entries themselves in the CPU’s standard L1/L2 caches. To do this, we need to set both Inner and Outer regions to Write-Back Cacheable (0b01). This ensures that once the MMU reads a table entry, it stays in the high-speed cache for subsequent accesses.
4. Shareability and Physical Size (SH0 & IPS). We set SH0 to Inner Shareable (0b11). This ensures that if multiple CPU cores are modified or reading page tables, the hardware maintains consistency between their caches. IPS tells the MMU the maximum width of the physical bus. We set this to 48 bits (0b101) to match the capabilities of the QEMU “virt” machine.
5. Disable TTBR1 walks (EPD1). Bit 23 of TCR_EL1 is the “Translation table walk Disable for translations using TTBR1_EL1” flag. We have not configured TTBR1_EL1; it holds its reset value and does not point to valid page tables. Without EPD1=1, any speculative access to the upper address half could cause the MMU to walk whatever garbage TTBR1_EL1 happens to contain, producing an unpredictable fault. With EPD1=1, such accesses immediately raise a Translation Fault rather than triggering a walk into uninitialised memory.

We assemble these flags into a single 64-bit value to be clocked into the system register:

#define TCR_T0SZ      (16UL << 0)
#define TCR_IRGN0_WB  (1UL  << 8)
#define TCR_ORGN0_WB  (1UL  << 10)
#define TCR_SH0_INNER (3UL  << 12)
#define TCR_TG0_4K    (0UL  << 14)
#define TCR_EPD1      (1UL  << 23)
#define TCR_IPS_48BIT (5UL  << 32)
#define TCR_VALUE     (TCR_T0SZ | TCR_IRGN0_WB | TCR_ORGN0_WB | \
                       TCR_SH0_INNER | TCR_TG0_4K | TCR_EPD1 | TCR_IPS_48BIT)

By writing this value to TCR_EL1, we effectively “turn on” the logic that allows the MMU to understand the specific 4-level (or in our case, 3-level shortcut) tree we’ve built in RAM.

Code Walkthrough: mmu.c

Create a new file for our mmu include/kernel/mmu.h. For now, this file is just a declaration.

/* include/kernel/mmu.h */
#ifndef KERNEL_MMU_H
#define KERNEL_MMU_H

void mmu_init(void);

#endif

Now also create the file kernel/mmu.c. I’ll walk through each section of the MMU implementation.

The static page tables:

static uint64_t l0_table[512] __attribute__((aligned(4096)));
static uint64_t l1_table[512] __attribute__((aligned(4096)));
static uint64_t l2_table[512] __attribute__((aligned(4096)));

These values live in .bss; they are static so they do not pollute the global namespace. Each table is $512 × 8 = 4,096 bytes$, and the aligned(4096) attribute is mandatory: the hardware requires each table to be aligned to its own size. Our linker script aligns .bss to 8 bytes; this attribute tightens that to 4 KB for these specific arrays.

The mmu_init() function opens with an explicit zeroing pass over all three tables even though .bss is supposed to be zeroed by the boot stub. This is a defensive measure: if the BSS initialisation loop was incomplete (an easy mistake to make in early boot code), a single stale non-zero entry could be interpreted as a valid descriptor and produce a translation to an arbitrary physical address. Zeroing in mmu_init() costs three small loops and eliminates that entire class of silent corruption:

for (int i = 0; i < 512; i++) {
    l0_table[i] = 0;
    l1_table[i] = 0;
    l2_table[i] = 0;
}

Building the L0 table:

l0_table[0] = (uint64_t)l1_table | DESC_VALID | DESC_TABLE;

Building the L1 table:

/* L1[0]: 1 GB block, Normal memory — covers first GB including UART at 0x09000000 */

l1_table[0] = 0x00000000UL | BLOCK_NORMAL;

/* L1[1]: table descriptor → l2_table — covers 0x40000000–0x7FFFFFFF */

l1_table[1] = (uint64_t)l2_table | DESC_VALID | DESC_TABLE;

Building the L2 table:

for (int i = 0; i < 8; i++) {

  uint64_t pa = 0x40000000UL + (uint64_t)i * 0x200000UL;

  l2_table[i] = pa | BLOCK_NORMAL;

}

Cleaning the page tables to RAM:

Before we can enable the MMU, we must guarantee that the hardware table walker can actually see the entries we just wrote. The table walker is a separate hardware unit that fetches page table entries directly from physical RAM. It does not look inside the CPU’s data cache. If the entries are still sitting in the L1 data cache and have not been written back to RAM, the walker will see stale zeros and produce translation faults the moment the MMU is switched on.

The fix is a cache clean by VA to the Point of Coherency, which flushes dirty cache lines to the level of the cache hierarchy that is shared with the MMU walker:

void clean_cache_range(uint64_t start, uint64_t size) {

  uint64_t line_size = 64;

  uint64_t end = start + size;

  start &= ~(line_size - 1);

  for (uint64_t addr = start; addr < end; addr += line_size) {

      asm volatile("dc cvac, %0" :: "r"(addr) : "memory");

  }

  asm volatile("dsb sy" ::: "memory");

}

After building the tables we call this function once per table:

clean_cache_range((uint64_t)l0_table, 4096);
clean_cache_range((uint64_t)l1_table, 4096);
clean_cache_range((uint64_t)l2_table, 4096);

Each table is exactly one page (4,096 bytes), so passing 4096 as the size is correct.

Programming the registers and enabling the MMU:

asm volatile("msr mair_el1,  %0" :: "r"(MAIR_VALUE) : "memory");

asm volatile("msr tcr_el1,   %0" :: "r"(TCR_VALUE)  : "memory");

asm volatile("msr ttbr0_el1, %0" :: "r"((uint64_t)l0_table) : "memory");

asm volatile("isb");

asm volatile("tlbi vmalle1" ::: "memory");

asm volatile("ic iallu"     ::: "memory");

asm volatile("dsb sy"       ::: "memory");

asm volatile("isb");

uint64_t sctlr;

asm volatile("mrs %0, sctlr_el1" : "=r"(sctlr));

sctlr |= (1UL << 0);   /* M  — enable MMU          */

sctlr |= (1UL << 2);   /* C  — enable data cache   */

sctlr |= (1UL << 12);  /* I  — enable icache       */

asm volatile("msr sctlr_el1, %0" :: "r"(sctlr) : "memory");

asm volatile("isb");

Updating kernel_main

With the header and implementation in place, the change to kernel/main.c is two lines:

#include "drivers/uart.h"
#include "kernel/mmu.h"

void kernel_main(void) {
    mmu_init();          /* must come first */

    uart_init();
    kprint("Hello, Kernel!\n");
    while (1);
}

Call mmu_init() before uart_init(). The Makefile already picks up all .c files under kernel/ with its find command, so no build system changes are needed.

Build and Run

Now it’s time to rebuild our PurgatoryOS with the command:

make clean && make && make run

If the page tables are correct, QEMU will output exactly what it did before:

$ make run
qemu-system-aarch64 -M virt -cpu cortex-a53 -nographic -kernel kernel.elf
Hello, Kernel!

That is the correct outcome. Identical output, fundamentally different internals. Every memory access from this point forward, including the UART write that produced that line, went through the hardware MMU. The UART region is covered by the Normal write-back L1 block descriptor; compiler-level reordering is prevented by the volatile keyword in the UART driver. The kernel code region is marked as cacheable, meaning instruction fetches and data reads now benefit from the L1 cache.

If you see no output or no apparent error, QEMU instead hangs silently. Before you spend hours debugging, read the next section.

What Broke (And Why)

The first time I ran this, QEMU froze with no output. It was completely silent, and I sat with it for longer than I would like to admit before I figured out what happened.

The Access Flag fault. I had forgotten DESC_AF in my block descriptor attributes. When the MMU is enabled, and a descriptor has AF=0, the hardware raises an Access Fault on the first access to that region. The fault syndrome code is 0x21. At this point in the kernel, we have no exception vector table. We will add one in a later post, but now it means the fault fired, the CPU tried to find an exception handler at the default reset vector, found nothing useful, and spun forever. There was no output because the UART lives inside the first L1 block, and that block was also missing AF, so the very first UART register write triggered the fault before a single character could be transmitted.

The fix was a one-line addition: add DESC_AF to both BLOCK_NORMAL and BLOCK_DEVICE. It feels obvious in retrospect. The ARM Architecture Reference Manual mentions the Access Flag fault in a footnote several dozen pages into the descriptor format section, which is not where you look when you are debugging a silent hang.

The L1 table/block confusion. The second bug I encountered was subtler. I initially tried to map the entire first gigabyte as a single L1 block entry with bits [1:0] = 0b11. That is wrong because at L1, 0b11 means a table descriptor. The hardware treats the address as a pointer to an L2 table rather than a block base. An L1 block entry needs bits [1:0] = 0b01. I had put DESC_VALID | DESC_TABLE instead of DESC_VALID | DESC_BLOCK, effectively telling the MMU that address 0x00000000 was a valid L2 table. The hardware dutifully “walked” to that address, found whatever happened to be at physical address zero, and tried to interpret that as page table data. The resulting garbage translations produced an immediate fault on the first kernel memory access.

The cache visibility problem. After fixing those two bugs, QEMU still hung silently. The page tables looked correct when I printed them before the SCTLR write, but the moment the MMU flipped on, every single access faulted. The cause: the table entries were sitting in the CPU’s L1 data cache. The hardware table walker fetches entries directly from RAM and does not look inside the cache. From the walker’s perspective, all three tables were still zeroed. Every descriptor was invalid, so every access faulted. The fix was the clean_cache_range() call described above: flush the dirty cache lines to RAM before enabling the MMU, then issue a dsb sy to ensure the RAM writes are globally visible.

The missing TLB and instruction-cache invalidation. Even with the cache clean in place, I saw occasional translation faults on the very first instruction fetch after the SCTLR write. The TLB can hold stale entries from before the MMU was configured, either from QEMU’s reset state or from speculative accesses. The instruction cache can hold lines fetched before the page tables were in place. Adding tlbi vmalle1 (invalidate all EL1 TLB entries) and ic iallu (invalidate the whole instruction cache) before setting SCTLR_EL1 eliminated the problem. These are both followed by dsb sy and isb to ensure completion before the MMU enable write.

EPD1 and spurious upper-half faults. Without TCR_EPD1=1, any speculative instruction fetch or data prefetch that touched an address with bits [63:48] non-zero caused the MMU to walk TTBR1_EL1, which we had not configured. The result was a translation fault from a speculative access, which is impossible to trace without an exception handler. Setting EPD1=1 makes those speculative accesses fault immediately with a clean syndrome rather than triggering a walk into garbage.

All five bugs share the same lesson: when the MMU misbehaves without an exception handler to decode the syndrome, the only tool you have is methodical review of every precondition. Print your descriptor values as hex with kprint before enabling the MMU, compare them to what you intended, add the cache clean and TLB invalidation, and enable only once everything looks right.

What’s Next

The MMU is working, the address space is protected, and your kernel now has a real memory hierarchy. But there is a gaping hole: what happens when something goes wrong? A null pointer dereference, an access to an unmapped address, a permission violation? Right now, any of these will cause the CPU to search for an exception handler at an address we have not set up. The CPU will find nothing and spin forever.

The next post fixes that. We will build the ARM exception vector table (vbar_el1), set up the Generic Interrupt Controller (GIC) provided by QEMU’s virt machine, and write a real interrupt handler for a periodic timer. By the end of the next post, the kernel will be reactive for the first time: it will print a heartbeat message every time the timer fires, and it will print a meaningful diagnostic when an exception occurs rather than hanging silently. That is a very different kind of kernel from what you have right now.

Sources

• ARM Architecture Reference Manual for Armv8-A, Chapter D4 - Memory Management - the primary reference for everything in this post. Section D4.3 covers descriptor formats; D4.2.2 covers TCR_EL1 in detail.
• ARM Learn the Architecture - AArch64 memory attributes and properties - a gentler introduction to MAIR, shareability, and memory types than the full reference manual.
• TTBR0_EL1 Register Documentation - ARM developer documentation for TTBR0_EL1 field descriptions.
• MAIR_EL1 Register Documentation - field-by-field breakdown of the memory attribute encoding.
• OSDev Wiki - ARM Paging - a community reference for AArch64 page table setup with practical examples. Complements the official documentation well.
• Armv8-A Address Translation - ARM white paper - a concise walkthrough of the address translation process from a system design perspective.
• AArch64 MMU Programming - Löwenware blog - a practical bare-metal implementation reference.
• AArch64 memory and paging - krinkinmu.github.io - a detailed walkthrough of AArch64 page table setup, including MAIR and TCR configuration.