Step 1 of 20

T8: the only way down is a return

Every hart wakes up in machine mode, the most privileged. RISC-V has no “enter supervisor mode” instruction. The only way to lower the mode is to return from a trap: mret sets the mode from mstatus.MPP and jumps to mepc. So start fills in both as if a trap had come from supervisor mode at main, and returns from it (Tour 46: Boot: returning from a trap that never happened tells this story).

That’s the pattern of the whole tour: a mode change never moves sp or satp. Those are separate instructions, and the code has to arrange them itself.

T8 happens once per hart, and machine mode is never re-entered afterwards. Nothing in this tree writes mtvec, all traps are delegated to S-mode, and the timer is the supervisor timer that Sstc provides (Mode, stack and page table: the master question).

Step 2 of 20

Boot's other change, paging on

The page-table answer changes once during boot, outside all eight transitions: hart 0’s kvminithart writes the kernel page table into satp (0x8000000000087fff in our run). Harts 1 and 2 do the same once hart 0 has built the table.

It’s worth being exact about where satp is written, because there are only four places in the whole tree:

So the claim that “the page table changes only in the trampoline” is true once boot is over, and not before.

Step 3 of 20

main becomes the scheduler, a role change only

main calls scheduler, which never returns. Mode, page table and the memory under sp are all unchanged: this is not a transition. But the role of the stack changes. The boot stack becomes this hart’s scheduler stack, the one place a hart can stand when it is running no process (The stacks of xv6).

From here on, every change on every hart is T1–T7.

Step 4 of 20

T6: the scheduler chooses cat

Hart 1’s scheduler finds cat (pid 3) RUNNABLE. cat was preempted earlier, so its kernel stack holds a frozen call chain ending in swtch. Holding cat’s p->lock, the scheduler marks it RUNNING and calls swtch.

T6 is the one transition that changes no mode at all. Before and after, the hart is in S-mode with the kernel page table. Only the stack changes: from the scheduler stack to cat’s kernel stack, or back. It is never directly from one process to another (Tour 13: swtch and the lock handed across a context switch).

Step 5 of 20

T6: one load moves sp

swtch stores 14 registers into the old context and loads 14 from the new one. One of the loads, line 26, is sp. The instant it executes, hart 1 is on cat’s kernel stack. ret then jumps into cat’s sched, where cat called swtch when it was preempted.

Notice what swtch does not touch: no CSR, no sret, no satp. It saves 14 registers, not 32, because the C calling convention already preserved the rest on the stack. cat continues in sched → yield (which releases p->lock) → usertrap, and heads for user mode.

Step 6 of 20

T5, part 1: prepare_return arms the way out

T5 is spread over three steps, because the mode, the page table and the stack each change at a different instruction. First prepare_return sets up the CSRs that sret and the next trap will read:

All three answers are still S, kernel stack, kernel page table. Only the next trap’s destination has changed. Tour 43: A system call, CSR by CSR shows each of these writes with measured values.

Step 7 of 20

T5, part 2: the page table goes first

usertrap returns into userret in the trampoline. Line 111 installs cat’s page table:

This is the odd combination: supervisor mode on a user page table, with no usable stack. It starts here and lasts until the sret on line 153, the second of the trampoline’s two stretches with no usable stack (The stacks of xv6). For its first five instructions sp keeps the kernel-stack address; from line 118 it holds the user’s value, still unusable. It is safe because the trampoline pushes nothing. The code keeps running only because the trampoline page sits at the same virtual address in both page tables.

Step 8 of 20

T5, part 3: sp, then the mode

Line 118 loads cat’s user sp from the trapframe; the rest of the registers follow, a0 last. Then line 153, sret: mode ← SPP (U), pc ← sepc, SIE ← SPIE.

So T5 changes the page table, then sp, then the mode, in that order, in userret, and the user stack becomes usable only with the mode change: supervisor code cannot use user pages (sstatus.SUM is never set). It is the mirror of T1, where the trap makes the user stack unusable before sp and satp change. Each order is forced: the page table must change while the code is still privileged enough to write satp, and the user registers, sp included, can be loaded only after it, because they come from the trapframe at TRAPFRAME, which only the user page table maps.

Step 9 of 20

T1: cat calls read, and ecall changes only the mode

cat reads README with read(fd, buf, 512). The stub puts SYS_read (5) in a7 and executes ecall, at 0x3c6 in cat’s image.

The hardware does very little: mode ← S, sepc ← the ecall’s address, scause ← 8, stval ← 0, SPP ← 0, SPIE ← SIE, SIE ← 0, pc ← stvec. It goes to S-mode and not M-mode only because start delegated it (kernel/start.c:31).

T2, T3 and T4 enter exactly the same way. Only scause differs.

Step 10 of 20

T1–T4, entry: uservec in supervisor mode with user everything

stvec sent the hart to uservec, into the same odd combination: the first of the trampoline’s two stretches with no usable stack, which lasts until line 92. In its first half sp is still the user’s:

With no stack and no free register, uservec parks a0 in sscratch, loads the constant TRAPFRAME, and stores all 31 registers into cat’s trapframe through cat’s own page table (Tour 7: The trampoline and the trapframe).

Step 11 of 20

T1–T4, entry: a kernel sp before a kernel page table

Line 76 loads sp from kernel_sp: the top of cat’s kernel stack. But satp still names cat’s page table, which doesn’t map kernel stacks. So for the four instructions up to line 92, sp holds the right value and still isn’t usable. This is the second half of the stretch, the mirror of T5 part 2:

tp (line 79) becomes the hart number, and t0 and t1 get usertrap's address and the kernel satp.

Step 12 of 20

T1–T4, entry: csrw satp, and everything is kernel

Line 92 installs the kernel page table, and in that instant the sp loaded on line 76 becomes a real stack. All three answers now agree: S-mode, cat’s kernel stack, kernel page table. jalr t0 calls usertrap, with ra pointing at userret.

Look back at the order. On the way in: stack value, then page table. On the way out (T5): page table, then stack value. Both orders keep one rule: the user’s sp is never paired with the kernel page table, and C code never runs without a mapped stack.

cat’s kernel stack is empty at this point, as it is on every entry from user mode. kernel_sp is always the top (kernel/trap.c:117).

Step 13 of 20

usertrap: one entry, four transitions

T1, T2, T3 and T4 share every instruction up to here. usertrap tells them apart by scause:

The top bit of scause separates interrupts (asynchronous, sepc = the next instruction to run) from exceptions (synchronous, sepc = the instruction that trapped). For cat’s read, it’s 8: syscall → sys_read, with interrupts on. From here on the hart may take T7 traps (Tour 5: Life of a system call follows the whole call).

Step 14 of 20

T2: an exception, survived or fatal

Suppose cat had a bug and stored through a null pointer. The hardware traps exactly as for ecall, with scause = 15 (store page fault), stval = 0 (the bad address) and sepc = the faulting store.

At this commit a page fault isn’t automatically fatal. vmfault is asked first: if the address lies inside the process’s size but has no page yet (memory from sbrklazy), it allocates one, and usertrap returns normally. sepc was not advanced, so the faulting instruction runs again and succeeds. Address 0 is cat’s first text page, which is mapped: readable and executable but not writable. So the fault isn’t about a missing page, vmfault refuses (it checks ismapped), line 78 marks cat killed, and line 82 makes it exit.

Either way, T2 enters through T1’s door and leaves through T5’s (or never, via kexit). Exceptions in the kernel are a different story: they arrive as T7, and kerneltrap panics (Tour 10: Exceptions and faults).

Step 15 of 20

T3 and T4: devintr reads the cause

While cat runs in user mode, two other kinds of event can pull hart 1 into the kernel the same way. Both are interrupts: the top bit of scause is 1, and sepc is the instruction that didn’t run yet.

• T3, code 9: a device, through the PLIC. plic_claim names it (UART 10, disk 1), the driver runs, plic_complete re-opens it. devintr returns 1.
• T4, code 5: this hart’s time passed its stimecmp. clockintr re-arms it. devintr returns 2, and usertrap calls yield.

Neither one touches sepc before returning, and neither needs to. T5 will resume cat exactly at the instruction it was about to run.

Step 16 of 20

T4 leads to T6: yield and sched

A timer tick is where most T6s come from. yield takes cat’s p->lock, marks it RUNNABLE, and calls sched, which switches to hart 1’s scheduler stack. cat stays behind, frozen on its kernel stack: usertrap → yield → sched → swtch. That’s the frozen chain step 5 resumed. T4 and T6 close the loop.

The second half of this T6 is the scheduler picking someone, maybe cat again, maybe on a different hart. When cat continues, yield releases the lock and usertrap takes T5 back to user mode.

Step 17 of 20

T7: a trap while already in the kernel

The shell is printing its $ prompt: write → … → uartwrite, holding the UART’s sleep-lock, with interrupts on. A tick arrives. (We caught exactly this in our run: sepc = 0x8000094c, just after a byte was stored into the UART, and sp = 0x3fffffbe70 on the shell’s kernel stack, slot 1.) Because usertrap set stvec to kernelvec (line 47), this trap doesn’t go near the trampoline:

No answer changes at all. kernelvec saves 17 registers (the 16 caller-saved ones and gp) on the current stack and calls kerneltrap. The tick could be taken at all only because the shell held no spinlock: a sleep-lock doesn’t count in noff, and SIE is never 1 while noff > 0 (Locks and interrupt state). The current stack can also be a scheduler stack: an interrupt taken in the scheduler’s window (proc.c lines 441–442) lands there (Tour 44: One interrupt, three landing sites).

Step 18 of 20

T7, the way back: kerneltrap and sret to S-mode

kerneltrap handles the device or tick. If it was a tick and a process is running, it yields: T6 from inside a T7. The shell did, in our run, while still holding the UART’s sleep-lock (allowed: a sleep-lock may be held across a switch, a spinlock may not). That’s how a long system call is preempted. yield’s acquire runs with SIE already off, so it records intena 0, and its release leaves interrupts off until sret restores them (Locks and interrupt state). Then it puts back the sepc and sstatus it saved, and kernelvec’s sret returns to the interrupted kernel instruction. SPP is 1, so the mode stays S.

So sret serves two transitions: T5 (SPP = 0, down to user mode) and the end of T7 (SPP = 1, back to supervisor mode). The instruction is the same; the saved SPP decides.

An exception in kernel code arrives here too, but devintr returns 0 for it and kerneltrap panics. The kernel has no recovery for its own faults.

Step 19 of 20

Every combination that exists

Put the eight transitions together and you get every combination of the three answers that ever occurs at this commit:

Eight rows out of the many conceivable ones (3 modes × 5 stacks × 3 page tables); the concept page lists nine rows because it splits one by stvec. The trampoline accounts for two of them, both with interrupts off and both a few instructions long. These lines are one of them. Some combinations never happen: U-mode on a kernel stack, C code on a user page table, the user’s sp with the kernel page table. Each absence is a design decision (Mode, stack and page table: the master question).

Step 20 of 20

Five myths, and is there a ninth transition?

Five beliefs this tour has shown to be false:

1. “The hardware switches page tables on a trap.” It changes the mode, the pc and a few CSR fields. satp and sp are software’s job (steps 9–12).
2. “swtch changes privilege.” It is called in S-mode and returns in S-mode, and touches no CSR (step 5).
3. “Each process has its own kernel page table.” There is one, shared by every hart and process. Each process has its own user page table and its own kernel stack, mapped in the shared table.
4. “The trapframe is on the kernel stack.” It is a separate page, mapped at TRAPFRAME in the user page table (The stacks of xv6).
5. “A context switch saves all 32 registers.” swtch saves 14. The user’s 31 are in the trapframe, and the kernel’s caller-saved ones are on the kernel stack.

A ninth transition? If you count mode changes, no: every one at this commit is T1–T5, T7’s sret, or T8. But two things changed an answer without being one of the eight: _entry's add sp, sp, a0 (kernel/entry.S:17), which gives the hart its first stack, and kvminithart's satp write in boot (step 2). And T5 has two entrances: usertrap returning into userret, and forkret calling it directly for a process’s first trip. With those footnotes, the eight cover everything.