System Calls: Drawing the Line Between User and Kernel

In the last post, we got two tasks running concurrently, cooperatively context-switching and being preempted by the timer. That was a genuine milestone, but it came with a lie I’ve been quietly maintaining for ten chapters: “our tasks.” There are no tasks in any meaningful sense yet. Both task_a and task_b run at EL1, alongside the kernel, with full access to every system register and every byte of physical memory. If task_a decided to write to VBAR_EL1, or to the UART MMIO region, or to the kernel’s .text section, the hardware wouldn’t lift a finger to stop it. They are not user programs. They are kernel threads wearing a costume.

That’s fine for a scheduler demo, but it is not fine for an operating system. The whole point of writing an OS instead of just writing a program is that it draws a line between the code you trust and the code you don’t, and it uses the CPU itself to enforce that line. On AArch64, that line runs between EL0 (user) and EL1 (kernel). The only legal way across it, once the kernel is running, is the svc instruction.

After this post, task_a and task_b will run at EL0. They will have their own stacks, they will be unable to touch the kernel’s address space, and the only way they can ask the kernel to do anything at all will be through a small, explicit, auditable syscall table. We will implement four: write, exit, getpid, and yield. Every one of them makes exactly one crossing into the kernel and exactly one crossing back.

What We’re Building Today

The end state of this post is a kernel that creates two user-mode tasks, drops them into EL0 via eret, and serves four syscalls when they trap in via svc #0. The UART output from the tasks will look identical to the previous one, but every byte will have come from a write syscall that crossed a hardware privilege boundary twice . Five new or heavily-modified files make this happen:

• include/kernel/syscall.h: the syscall number enum and handler table API.
• kernel/syscall.c: the C-side dispatcher (syscall_dispatch) and the four handlers.
• arch/arm64/vectors.S: a new el0_sync_64 entry that decodes ESR_EL1 and routes svc to C.
• user/user.S: the EL0-side stubs, sys_write, sys_exit, sys_getpid, sys_yield.
• kernel/scheduler.c: task_create and a new drop_to_el0 helper that sets up SPSR_EL1, ELR_EL1, and SP_EL0 for the first eret into a user task.

Here’s the complete flow for a single write syscall. Click through each step, and the active side of the privilege boundary lights up. The register state at the bottom shows what the CPU is holding at that moment.

mov x8, #1          // SYS_WRITE
mov x0, #1          // fd = stdout
ldr x1, =msg        // buf
mov x2, #13         // len

Two round-trip across the privilege boundary. The rest of this post is the code that makes each of those seven steps actually work on real hardware.

The Privilege Boundary, Explained by the Hardware

ARMv8 defines four Exception Levels of EL3 through EL0, and the rule that makes them useful is simple: a svc instruction at EL0 always traps to EL1, and execution at EL0 cannot write to any register whose name ends in _EL1, _EL2, or _EL3. Try it, and you get an Undefined Instruction exception. The MMU enforces this at the page table level as well. User pages can be marked with an AP (Access Permissions) field that forbids EL0 access entirely.

That enforcement is what makes a syscall mean something. On a processor without privilege levels, a “syscall” is a function pointer call, with nothing stopping a well-aimed bug from overwriting the kernel. On AArch64, there is literally one legal user-to-kernel transition instruction, and the hardware refuses to let the user pick where it lands. The landing site is whatever is at the kernel’s vector table VBAR_EL1 + 0x400. This is the ARM Architecture Reference Manual’s description of the synchronous exception taken from a lower Exception Level running AArch64.

When the svc fires, the hardware atomically does four things before a single instruction of kernel code runs:

1. Copies PSTATE (the entire current processor state, including which EL we came from) into SPSR_EL1.
2. Writes the address of the instruction after svc into ELR_EL1.
3. Switches to EL1 and starts using SP_EL1.
4. Jumps to VBAR_EL1 + 0x400.

That’s the entire contract. Everything else, like saving general-purpose registers, decoding the syscall number, dispatching, returning the result, switching back, is software, and it’s ours to write.

Giving Tasks Their Own User Stacks

Before we can drop into EL0, each task needs something it didn’t have in the previous post: a user stack. In Post 9, every task ran at EL1 and shared the SP_EL1 pool. From now on, each task gets two stacks: one for user mode (SP_EL0) and one for the kernel to use on its behalf during a trap (SP_EL1).

Here’s the new physical layout after creating two tasks. Notice the two new user-stack regions that didn’t exist a chapter ago:

The important thing this diagram captures is that there are now two architecturally distinct 2 MB blocks. mmu.c marks the first one (0x40000000–0x401FFFFF) with AP=00 (EL1 read/write, EL0 no access whatsoever), and every block above it (0x40200000 upward) with AP=01 (EL1 and EL0 both read/write, EL0 can execute). The linker script cooperates by explicitly placing .text.user and .rodata.user into the second block at 0x40200000. An EL0 task that somehow branches below 0x40200000 will trap into the kernel with a translation fault before it executes a single kernel instruction. This is the real privilege boundary. svc is how you cross it intentionally; the MMU is how the kernel guarantees nothing else crosses it at all.

We are still single-address-space (both EL0 and EL1 share the same page table via TTBR0_EL1). Per-task virtual address spaces are a future post. What matters for this post is that the hardware automatically picks between SP_EL0 and SP_EL1 based on the current EL. The kernel never has to swap stacks by hand.

Extending the PCB

Because every task now carries two stacks and we need to be able to restart the whole story (syscall number, arguments, return PC, SPSR) on each eret, the PCB grows slightly. Here’s what we’re adding to our current structure. The code paths are untouched; it’s strictly additive:

/* include/kernel/scheduler.h — additions for Post 10 */

/* Extra per-task state for EL0 tasks.  When a task is running at EL0 these
 * are live in SP_EL0 / ELR_EL1 / SPSR_EL1.  When a task is preempted, we
 * snapshot them here before another task runs.  At scheduler_start time
 * they are initialised by task_create. */
typedef struct {
    uint64_t sp_el0;      /* user stack pointer                              */
    uint64_t elr_el1;     /* PC to resume at — initially the entry function */
    uint64_t spsr_el1;    /* saved PSTATE — always 0 for a fresh EL0 task   */
} user_context_t;

typedef struct pcb {
    uint32_t        pid;
    task_state_t    state;
    cpu_context_t   ctx;           /* kernel-side callee-saved state — Post 9 */
    user_context_t  uctx;          /* user-side ELR / SPSR / SP_EL0 — new     */
    uint8_t        *kstack_base;   /* per-task SP_EL1 backing memory          */
    uint8_t        *ustack_base;   /* per-task SP_EL0 backing memory          */
    size_t          kstack_size;
    size_t          ustack_size;
    void          (*entry)(void);
} pcb_t;

#define USER_STACK_SIZE  (12 * 1024)

The user context is intentionally small. Most of what a user task holds while it’s not running is already covered by the standard register save done in SAVE_REGS during the syscall. The saved x0–x30 live on the kernel stack for the duration of the trap. uctx is only for the registers the hardware itself reads (ELR_EL1, SPSR_EL1) and the user stack pointer, which is invisible to EL1 code while at EL1.

The svc Stubs in User Code

Before we can handle a syscall, something has to issue one. On Linux, this is libc’s job. We don’t have libc, so we write our own stubs. Each stub is four instructions.

.section .text.user

.global sys_write

.global sys_exit

.global sys_getpid

.global sys_yield

sys_write:

  mov  x8, #1

  svc  #0

  ret

sys_exit:

  mov  x8, #2

  svc  #0

  b    .

sys_getpid:

  mov  x8, #3

  svc  #0

  ret

sys_yield:

  mov  x8, #4

  svc  #0

  ret

Those four stubs are the entire userland-side ABI. Any EL0 code the shell or future user programs want to run will import them. From the user program’s perspective, a syscall is a C function. From the CPU’s perspective, it is a trap.

Wiring the Vector Table Entry

In Post 7 we had built a 16-entry vector table with all the el0_* slots pointing at an infinite spin loop. Those were bugs we haven’t handled yet. One of them: el0_sync_64 at offset +0x400, is no longer a bug. It is the syscall entry point for every user task. The update to vectors.S is small. We only need to replace the b . with a real handler:

/* Replacing the old el0_sync_64 stub */

el0_sync_64:

  SAVE_REGS

  mrs  x0, esr_el1

  mov  x1, sp

  bl   el0_sync_handler

  RESTORE_REGS

  eret

The C handler is the real substance. el0_sync_handler reads the exception class and routes:

/* kernel/syscall.c */

#include "kernel/syscall.h"
#include "kernel/scheduler.h"
#include "drivers/uart.h"

/* Saved register frame layout — must match SAVE_REGS in vectors.S.
 *
 * SAVE_REGS pushes in ascending pairs (x0/x1 first, x2/x3, …, then x30 on its
 * own). Because the AArch64 stack grows downward and each `stp ..., [sp, #-16]!`
 * pre-decrements, the *last* register pushed ends up at the *lowest* address
 * — which is where `mov x1, sp` points when the C handler is called.
 *
 * So the struct must list x30 FIRST (lowest offset), then x28/x29 (the
 * previous pair), …, and x0/x1 LAST (highest offset).  Getting the order
 * backwards silently corrupts every syscall — x0 reads the saved x30, etc. */
typedef struct regs {
    uint64_t x30;
    uint64_t x28, x29;
    uint64_t x26, x27;
    uint64_t x24, x25;
    uint64_t x22, x23;
    uint64_t x20, x21;
    uint64_t x18, x19;
    uint64_t x16, x17;
    uint64_t x14, x15;
    uint64_t x12, x13;
    uint64_t x10, x11;
    uint64_t x8,  x9;
    uint64_t x6,  x7;
    uint64_t x4,  x5;
    uint64_t x2,  x3;
    uint64_t x0,  x1;   /* highest address — pushed first by SAVE_REGS */
} regs_t;

/* Exception class field in ESR_EL1, bits [31:26]. */
#define ESR_EC_SHIFT   26
#define ESR_EC_MASK    0x3F
#define EC_SVC64       0x15

void el0_sync_handler(uint64_t esr, regs_t *r) {
    uint32_t ec = (esr >> ESR_EC_SHIFT) & ESR_EC_MASK;

    if (ec == EC_SVC64) {
        /* Write the return value into the saved x0 slot — the eret
         * that follows will restore it into the user's x0. */
        r->x0 = syscall_dispatch(r);
        return;
    }

    /* Anything else from EL0 is, for now, fatal to the task:
     * unaligned access, MMU fault in user space, undefined instruction.
     * We kill the task instead of taking down the whole kernel.  Print
     * the full ESR so the EC nibble (bits [31:26]) is obvious in the log. */
    kprint("[el0] unhandled sync, ESR=0x");
    for (int shift = 60; shift >= 0; shift -= 4) {
        uint8_t nibble = (esr >> shift) & 0xF;
        uart_putc(nibble < 10 ? '0' + nibble : 'a' + nibble - 10);
    }
    kprint("\n");
    task_kill_current();
}

The two guarantees here are worth dwelling on. First, r->x0 = syscall_dispatch(r) is the entire return-value mechanism. By writing to the saved frame, the RESTORE_REGS that runs a few instructions later puts that value into the hardware x0, which is exactly what the user’s ret reads. Second, there is no path by which an EL0 bug can skip this handler; VBAR_EL1 + 0x400 is the hardware-fixed landing point for everything the user can throw at us.

The Syscall Table

The dispatcher itself is short, because all it does is decode and call. The whole point of the table pattern (as the OSDev System Calls wiki notes, and as Linux and every other serious kernel use) is that adding a syscall is a one-line change: a new enum entry and a new table row. No match-cascade, no giant if-else.

/* include/kernel/syscall.h */

#ifndef PURGATORY_SYSCALL_H
#define PURGATORY_SYSCALL_H

#include <stdint.h>

/* Syscall numbers.  These are our ABI — once a user program is compiled
 * against them, we can't change the numbers without breaking it. */
typedef enum {
    SYS_WRITE  = 1,
    SYS_EXIT   = 2,
    SYS_GETPID = 3,
    SYS_YIELD  = 4,
    SYS_MAX
} syscall_nr_t;

/* Error codes — follow the Linux convention of negative errno. */
#define ENOSYS   -38    /* unknown syscall */
#define EINVAL   -22    /* bad argument */
#define EBADF    -9     /* bad file descriptor */

typedef struct regs regs_t;     /* full definition lives in syscall.c */

void    syscall_init(void);
int64_t syscall_dispatch(regs_t *r);

/* User-facing declarations.  The bodies are the svc stubs in user/user.S —
 * declaring them here lets EL0 C code (task_a, task_b) call them like
 * ordinary functions while the linker resolves the symbol to the svc wrapper
 * placed in the user-accessible memory block. */
int64_t sys_write (int fd, const char *buf, uint64_t len);
int64_t sys_exit  (int code);
int64_t sys_getpid(void);
int64_t sys_yield (void);

#endif

/* kernel/syscall.c — the dispatcher and the four handlers */

/* The kernel-side implementations are prefixed `k_sys_*` so they don't
 * collide with the user-facing `sys_*` svc stubs from user/user.S.
 * The linker keeps these in Block 0 (EL1-only) while `sys_*` lives in
 * Block 1 (EL0-accessible). */
static int64_t k_sys_write (int fd, const char *buf, uint64_t len);
static int64_t k_sys_exit  (int code);
static int64_t k_sys_getpid(void);
static int64_t k_sys_yield (void);

/* The table is indexed by syscall number.  NULL entries are illegal —
 * we explicitly bounds-check before indexing.  An entry is a plain
 * function pointer that takes up to 6 uint64_t args; the individual
 * handlers cast them to their real types. */
typedef int64_t (*syscall_fn)(uint64_t, uint64_t, uint64_t,
                              uint64_t, uint64_t, uint64_t);

static syscall_fn syscall_table[SYS_MAX] = {
    [SYS_WRITE]  = (syscall_fn)k_sys_write,
    [SYS_EXIT]   = (syscall_fn)k_sys_exit,
    [SYS_GETPID] = (syscall_fn)k_sys_getpid,
    [SYS_YIELD]  = (syscall_fn)k_sys_yield,
};

/* syscall_init — called once from kernel_main before any EL0 code runs.
 * The table is a static C initializer so there is nothing runtime-wise to
 * populate; the job of this function is to sanity-check the table and
 * announce readiness on the UART.  It mirrors the style of scheduler_init,
 * heap_init, etc. — one line of setup diagnostics per subsystem. */
void syscall_init(void) {
    /* Bounds check: every non-zero slot below SYS_MAX must point somewhere.
     * Catches the "added an enum entry but forgot to wire the table row"
     * bug at boot instead of at the first svc. */
    for (uint64_t nr = 1; nr < SYS_MAX; nr++) {
        if (syscall_table[nr] == NULL) {
            kprint("[sys] FATAL: syscall_table slot missing\n");
            while (1) { asm volatile("wfi"); }
        }
    }
    kprint("[sys] init: 4 syscalls registered (write/exit/getpid/yield)\n");
}

int64_t syscall_dispatch(regs_t *r) {
    uint64_t nr = r->x8;

    if (nr >= SYS_MAX || syscall_table[nr] == NULL) {
        kprint("[sys] ENOSYS: nr=");
        /* ... print nr, then return -ENOSYS ... */
        return ENOSYS;
    }

    /* Call the handler with up to six arguments from the saved frame.
     * Handlers that take fewer arguments simply ignore the extra ones;
     * AAPCS64 allows this. */
    return syscall_table[nr](r->x0, r->x1, r->x2, r->x3, r->x4, r->x5);
}

The four handlers themselves are small by design. None of them allocates. None of them blocks. sys_write and sys_yield are the only two that touch kernel state beyond the current PCB.

/* kernel/syscall.c — the individual handlers */

static int64_t k_sys_write(int fd, const char *buf, uint64_t len) {
    /* fd is ignored for now — we only have stdout, which is the UART.
     * A real kernel would look fd up in a per-process file descriptor
     * table and dispatch to the appropriate driver. */
    if (fd != 1) return -EBADF;
    if (buf == NULL) return -EINVAL;

    /* NOTE: no validation of `buf`.  A malicious EL0 task could pass a
     * pointer into kernel memory and have the kernel "write" it out via
     * the UART, leaking secrets.  Post 11 (Rust) and Post 12 (filesystem)
     * are where we introduce access_ok() checks against the user's page
     * table permissions.  For now, we trust EL0.  Documented sin. */
    for (uint64_t i = 0; i < len; i++) uart_putc(buf[i]);
    return (int64_t)len;
}

static int64_t k_sys_exit(int code) {
    (void)code; /* no exit status plumbing yet */
    task_kill_current();
    /* task_kill_current calls schedule_next, which never returns here. */
    return 0;
}

static int64_t k_sys_getpid(void) {
    return (int64_t)current_task_pid();
}

static int64_t k_sys_yield(void) {
    scheduler_yield();
    return 0;
}

sys_write is where the “don’t trust user pointers” problem is visible in its simplest form. In a production kernel, you’d walk the user’s page table to check that every byte of the (buf, buf+len) The range is readable by EL0. We skip that for now; the comment is our honest acknowledgement that we owe the check. The Linux kernel’s copy_from_userprimitive is the canonical version of the check we’re postponing.

Dropping a Task into EL0

This is the one genuinely new-to-the-kernel mechanic in this post: how do we get eret to land at EL0 when the task starts? start_first_task from Post 9 puts us into EL1 by loading a saved context and doing ret; it doesn’t switch ELs. And we don’t want to rewrite Post 9’s perfectly good context-switch machinery; we want to reuse it.

The trick that makes this fall out cleanly: every AArch64 task switch in this kernel already ends with ret into whatever was in ctx.lr. So for a brand-new task, we set ctx.lr to a one-shot C function whose entire job is to program the three “came from EL0t” system registers and execute eret. The first time the scheduler selects this task, the existing start_first_task / context_switch ath jumps into the trampoline, the trampoline does eret, and the CPU lands at EL0 running the task’s entry function. No new assembly, no special case in scheduler_start.

/* SPSR_EL1 value for a clean return to EL0t:
 *   M[3:0] = 0    (EL0t — SP_EL0 selected)
 *   D=A=I=F = 0  (all exceptions unmasked on entry)
 */
pcb->uctx.spsr_el1 = 0;

/* SPSR_EL1 value for a clean return to EL0t:
 *   M[3:0] = 0    (EL0t — SP_EL0 selected)
 *   D=A=I=F = 0  (all exceptions unmasked on entry)
 */
pcb->uctx.spsr_el1 = 0;

The concrete code is short because the setup is mechanical:

/* kernel/scheduler.c — the first-run trampoline and updated task_create */

/* task_entry_trampoline — first-run stub for EL0 tasks.
 *
 * context_switch restores the task's kernel registers (zero for a new task)
 * and does `ret`, landing here.  We load ELR_EL1/SPSR_EL1/SP_EL0 from the
 * current task's uctx and eret into EL0.  Never called a second time:
 * subsequent context_switch calls restore the return address inside
 * schedule_next, unwinding normally through svc/irq handlers back to eret. */
static void __attribute__((noreturn)) task_entry_trampoline(void) {
    pcb_t *pcb = tasks[current_task];
    asm volatile(
        "msr sp_el0,   %0\n"
        "msr elr_el1,  %1\n"
        "msr spsr_el1, %2\n"
        "mov x0, xzr\n"
        "mov x1, xzr\n"
        "mov x2, xzr\n"
        "eret"
        :
        : "r"(pcb->uctx.sp_el0),
          "r"(pcb->uctx.elr_el1),
          "r"(pcb->uctx.spsr_el1)
    );
    __builtin_unreachable();
}

pcb_t *task_create(void (*entry)(void)) {
    pcb_t   *pcb    = kmalloc(sizeof(pcb_t));
    uint8_t *kstack = kmalloc(TASK_STACK_SIZE);
    uint8_t *ustack = kmalloc(USER_STACK_SIZE);
    /* ... NULL checks omitted for brevity ... */

    zero_memory(pcb, sizeof(pcb_t));
    pcb->pid         = task_count;
    pcb->state       = TASK_READY;
    pcb->entry       = entry;
    pcb->kstack_base = kstack;
    pcb->ustack_base = ustack;
    pcb->kstack_size = TASK_STACK_SIZE;
    pcb->ustack_size = USER_STACK_SIZE;

    /* User-side state — consumed by task_entry_trampoline's eret. */
    pcb->uctx.sp_el0   = ((uint64_t)(ustack + USER_STACK_SIZE)) & ~0xFUL;
    pcb->uctx.elr_el1  = (uint64_t)entry;
    pcb->uctx.spsr_el1 = 0;           /* EL0t, all exceptions unmasked */

    /* Kernel-side state — used by the Post 9 start_first_task / context_switch
     * machinery.  ctx.lr points at the trampoline so the first ret lands there. */
    pcb->ctx.sp = ((uint64_t)(kstack + TASK_STACK_SIZE)) & ~0xFUL;
    pcb->ctx.lr = (uint64_t)task_entry_trampoline;

    tasks[task_count++] = pcb;
    return pcb;
}

scheduler_start itself is unchanged from Post 9. It still calls start_first_task(&first->ctx) and that’s it. The whole EL-switch mechanism lives in those ten lines of inline assembly in task_entry_trampoline, which reuse the context-switch path we already had.

The Register Snapshot at the Boundary

Here is what the system register state looks like the moment before eret fires inside task_entry_trampoline, setting up task A’s first entry into EL0. The msr instructions have just run; the hardware is about to consume the top three rows:

And the state after the first svc #0 from task A, the moment the kernel gains control at el0_sync_64:

Wiring Everything Into kernel_main

The task bodies stop calling kprint directly (that’s EL1 code) and instead call sys_write. The more subtle change is where they live: both functions and their message strings are annotated with __attribute__((section(".text.user"))) and ".rodata.user", so the linker places them into Block 1 (the EL0-accessible region at 0x40200000). Without this, task A’s first instruction would be in Block 0, the MMU would see AP=00, and the first fetch from EL0 would trap immediately.

/* kernel/main.c — the EL0 task bodies and the updated boot sequence */

/* Strings go in .rodata.user so they land in Block 1 alongside the code
 * that reads them.  task_a would otherwise point at a kernel-only address
 * and sys_write's loop would fault on the first byte. */
__attribute__((section(".rodata.user")))
static const char msg_a[] = "[task A] tick\n";

__attribute__((section(".rodata.user")))
static const char msg_b[] = "[task B] tick\n";

/* The task bodies themselves have to live in .text.user so EL0 can execute
 * them.  This attribute is what tells the linker "place me in Block 1".  */
__attribute__((section(".text.user")))
static void task_a(void) {
    while (1) {
        sys_write(1, msg_a, sizeof(msg_a) - 1);
        sys_yield();
    }
}

__attribute__((section(".text.user")))
static void task_b(void) {
    for (int i = 0; i < 5; i++) {
        sys_write(1, msg_b, sizeof(msg_b) - 1);
        sys_yield();
    }
    sys_exit(0);   /* voluntarily leave the runqueue */
}

void kernel_main(void) {
    mmu_init();
    uart_init();
    kprint("PurgatoryOS booting...\n");

    exceptions_init();       
    kprint("Exceptions: ready\n");
    
    gic_init();              
    kprint("GIC: ready\n");
    
    timer_init(500);         
    kprint("Timer: armed\n");
    
    heap_init();             
    kprint("Heap: ready\n");

    scheduler_init();
    syscall_init();          
    kprint("Syscalls: ready\n");

    task_create(task_a);
    task_create(task_b);

    asm volatile("msr daifclr, #2");
    kprint("Interrupts enabled.\n");

    kprint("Starting scheduler...\n");
    scheduler_start();
}

The linker has to agree with these attributes; there’s a matching snippet in link.ld that physically routes the .user_space output section to 0x40200000:

/* --- BLOCK 1: USER (0x40200000 - 0x403FFFFF) --- */
. = ALIGN(0x200000);           /* match mmu.c's 2 MB L2 block */
.user_space : {
    _user_start = .;
    build/kernel/main.o(.text.user)     /* task_a, task_b */
    build/kernel/main.o(.rodata.user)   /* msg_a, msg_b   */
    build/user/user.o(.text .text.*)    /* sys_* svc stubs */
    _user_end = .;
}

EXCLUDE_FILE clauses in the earlier .text and .rodata sections make sure the ordinary wildcards don’t scoop main.o’s task bodies are placed in Block 0 by accident. This is the unglamorous plumbing that makes the privilege boundary mean something. Every EL0 instruction the CPU can ever reach has to be between _user_start and _user_end.

Now build and run our PurgatoryOS:

PurgatoryOS booting...
Exceptions: ready
GIC: ready
Timer: armed
[heap] init: free-list allocator
Heap: ready
[sched] init: round-robin scheduler
[sys] init: 4 syscalls registered (write/exit/getpid/yield)
Syscalls: ready
[sched] task created: pid=0 el=0
[sched] task created: pid=1 el=0
Interrupts enabled.
Starting scheduler...
[sched] starting first task (pid=0) at EL0
[task A] tick
[task B] tick
[task A] tick
[task B] tick
[task A] tick
[task B] tick
[task A] tick
[task B] tick
[task A] tick
[task B] tick
[sys] exit: pid=1 code=0
[task A] tick
[task A] tick
[task A] tick
...

The output is almost identical to Post 9, but every single [task X] tick line is now a full EL0→EL1→EL0 round trip through sys_write. Task B voluntarily exits after five ticks; sys_exit marks its PCB TASK_DEAD, and the scheduler’s dead-task skip simply never selects it again. Task A prints forever.

What Broke (And Why)

The bug that ate my entire Saturday was the quietest kind: the kernel booted fine, dropped into EL0 fine, and then the first svc #0 simply froze. No output, no diagnostic, no repeated fault chain… just silence. QEMU’s -d int,mmu log showed a synchronous exception being taken, then nothing.

The cause was that I had written the new el0_sync_64 handler but forgotten to rebuild vectors.o. My Makefile’s dependency for vectors.o listed only vectors.S, not the headers or SAVE_REGS macro, which was fine, because neither had changed. But I had changed the branch target for the el0_sync_64 slot, and make didn’t rebuild. The VBAR_EL1 + 0x400 slot was still the old infinite spin. svc dutifully jumped into b . and never came back.

The fix was trivial (make clean && make). The lesson is less so: in a bare-metal kernel, silent infinite loops are the debugging equivalent of a bricked phone. I added two things after this

• A boot-time sanity print that reads back VBAR_EL1 and the first 4 bytes at offset +0x400. If the byte matches b . (encoded as 0x14000000), the kernel panics early with a “vector table not updated” message. Cheap, but it turns a 40-minute debugging session into a 2-second one.
• A timer_tick counter that kprints every N ticks even if nothing else is happening. If the kernel is alive, that counter ticks. If it isn’t, the silence is diagnostic.

The other bug, much faster to catch, was misaligning SP_EL0. I had USER_STACK_SIZE = 3 * 4096 = 12288, but an earlier draft set sp_el0 = ustack + USER_STACK_SIZE - 16, which is 16-byte aligned. I was also writing a single u64 marker at the new top of the stack for debugging, which caused it to fall back to 8-byte alignment in some configurations. The first stp in the entry prologue (AArch64’s ABI-required 16-byte prologue save) then triggered an SP alignment fault with ESR_EL1 EC = 0x26. The fix is the unconditional & ~0xFUL mask we already had for kernel stacks. I had to apply it to user stacks too, and don’t ever write past the aligned top.

What’s Next

The boundary between the kernel and user space is now real. Every output from the tasks crosses that boundary twice. Every time a task yields, it does so via a syscall. The tasks are cleanly isolated from each other’s memory in the sense that they can’t branch into each other, but they still share a single page table, so a malicious pointer can still wander anywhere in physical memory. True address-space isolation (per-task TTBR0_EL1) is a future post.

The next post is, by design, a left turn. We are going to rewrite the heap allocator we built earlier in no_std Rust, and call it from C via FFI. The reason we’re doing it now, and not earlier, is that the kernel finally has enough surface area to make the payoff visible: a kmalloc that cannot double-free, called from a scheduler that cannot misuse pointers, serving a syscall layer that desperately wants the extra safety. The privilege boundary we just built says “don’t trust user code”; Rust’s type system says “don’t trust any code, even ours.” Together, they become mutually reinforcing, which is exactly what you want in a kernel.

The two things we now have: a privileged scheduler and an enforced syscall interface, are the architectural minimum required to run untrusted programs. Every remaining post builds on this foundation. The kernel has a real border, and the only door through it is one we wrote ourselves.

Sources

• ARM Architecture Reference Manual (ARM DDI 0487) — AArch64 Exception Model — the authoritative description of EL0/EL1/EL2/EL3, the svc instruction semantics, SPSR_EL1 fields, and the vector table layout. Section D1 “The AArch64 System Level Programmers’ Model” is required reading for the privilege boundary.
• ESR_EL1, Exception Syndrome Register — ARM Developer — the definitive register reference for the EC and ISS fields we decode in el0_sync_handler. EC = 0x15 is explicitly defined here as “SVC instruction execution in AArch64 state.”
• 5.1 User Processes and System Calls — raspberry-pi-os (s-matyukevich) — the closest public teaching kernel to our setup: drops tasks into EL0, sets up SPSR/ELR/SP_EL0 in a pt_regs structure, and dispatches syscalls by table lookup. Our architecture is an independent but parallel implementation.
• Linux System Call Flow in ARM64 — embeddedvenkatpari — walks through Linux’s el0t_64_sync_handler and the way ESR_EL1.EC steers synchronous exceptions into el0_svc. Reading this alongside our own dispatcher makes clear how minimal ours is by comparison.
• ARM64 System Calls — ElseWhere (duetorun.com) — a compact explainer of svc #0, the x8 convention for syscall numbers, and what the CPU does automatically on the trap. Good companion read for users who want Linux-style examples.
• AArch64 Procedure Call Standard (AAPCS64) — ARM Software ABI — the register calling conventions we rely on. x0 is both the first argument and the return register; x8 is the indirect-result register but is overloaded by Linux and us for syscall numbers. Section 6.1 covers register roles.
• System Calls — OSDev Wiki — general background on syscall dispatchers, table design, and the argument-passing trade-offs. The “NULL-free table” recommendation we follow originates here.
• ARM System Calls — OSDev Wiki — ARM-specific (though 32-bit-leaning) notes on svc/swi handling, vector table positioning, and the old-school TRAP approach. Useful for comparing against our AArch64 implementation.
• AArch64 Exception Handling — Wenbo Shen — detailed walkthrough of the 16-entry vector table and what each group (current EL / lower EL / AArch64 / AArch32) is for. The reason el0_sync_64 lives at +0x400 is explained clearly here.
• AArch64 Interrupt and Exception Handling — Mike Krinkin — companion post to the above, focusing on the software side: SAVE_REGS macros, ESR decoding, and the eret return path. The structure of our el0_sync_64 handler follows the pattern described here.
• arm64.syscall.sh — Linux ARM64 Syscall Table — handy reference for the Linux numbering we partially emulate. Our SYS_WRITE = 1 collides with Linux’s SYS_IO_SETUP intentionally — we’re building our own ABI and only borrowing the conventions.
• Linux Syscalls Reference — GitHub gist by yamnikov-oleg — cross-architecture syscall reference, useful for understanding why the x8-carrier convention is a Linux/AArch64 choice rather than an ARM architectural requirement.
• Transitioning from EL0 AArch32 to EL1 AArch64 — System on Chips — exact mechanics of eret and the SPSR M-field. Sanity-check for the spsr_el1 = 0 value we use.
• copy_from_user — Linux Kernel Documentation — the safety primitive we haven’t implemented yet. Every production kernel puts this between sys_write and the for loop that dereferences the user pointer. Read it to understand what we still owe.
• bare-metal-aarch64 (rlepigre) — step-07 — a minimal AArch64 kernel that covers EL-level transitions in isolation. Good small-codebase companion to the monster-sized Linux source.