Step 1 of 20

Before the ecall, the CSRs hold leftovers

echo has called the system call stub write. li a7, 16 has run; the ecall at address 0x354 is next. We stopped hart 0 here with a gdb breakpoint on 0x354 that fires only when a7 is 16 and a2 is 2, and read the registers:

The stale rows matter. A CSR keeps whatever the last trap on this hart left in it. echo’s last trap on this hart was its own exec system call, made while it was still a copy of the shell: sscratch still holds that call’s a0, 0x2020, the shell’s buffer holding the string “echo”. sepc = 0x7c is what prepare_return wrote at the end of that exec: 0x7c is echo’s entry point, start. scause is newer. It comes from an interrupt taken in the kernel during the exec (a UART interrupt in this run, a timer tick in an earlier one), and kerneltrap puts back sepc and sstatus but never scause. Nothing cleans these leftovers up, because nothing needs to.

The raw sstatus value carries other fields too. 0x200000000 is UXL = 2 (user code is 64-bit). The tour decodes only the three bits xv6 uses.

Step 2 of 20

ecall: the hardware's whole contribution

We single-stepped the ecall and compared. Everything the instruction changed:

That is the complete list. The mode becomes S, not M, only because start delegated every exception to supervisor mode with medeleg (kernel/start.c:31).

sepc holds the address of the ecall, not the next instruction. The privileged spec says so for every synchronous exception: sepc points at the instruction that caused it, so that a fault can be retried. A system call is the one case where the kernel wants to skip the instruction, and it must add 4 itself (step 10).

Step 3 of 20

What ecall did not touch

Hart 0 is now at uservec's first instruction, in supervisor mode, and the rest of the machine is exactly as echo left it:

This is the most important fact about RISC-V traps, and the most common misunderstanding: the hardware does not switch page tables, does not switch stacks, and does not save registers. (Some architectures load a kernel stack pointer as part of a trap; x86 does, from a table the OS fills in. RISC-V leaves it to software.)

So supervisor code is running with a user page table, a user stack pointer it must not trust, and not one free register. The fetch of this very instruction works only because the trampoline page page is mapped at 0x3ffffff000 in echo’s page table as well as the kernel’s (kernel/proc.c:189). The sp is labelled “none”: it holds a value, but the kernel won’t push anything through a pointer the user controls. Mode, stack and page table: the master question has the full table of who changes what.

Step 4 of 20

sscratch: one register's worth of room

To save registers, the code needs an address in a register, and every register holds user data. csrw sscratch, a0 parks the user’s a0 in sscratch, a CSR the hardware never touches on its own, which the spec sets aside for the supervisor’s use. Then li a0, TRAPFRAME (assembled as three instructions, lui/addiw/slli, at 0x3ffffff004) loads a constant.

Notice what sscratch holds: the user’s a0, for a few dozen instructions. It does not hold a pointer to the trapframe (some older xv6 releases used it that way). Between traps it holds stale junk, which we saw in step 1.

Why can a constant address work for every process? Because TRAPFRAME is a virtual address, and satp still holds echo’s page table, where it maps to echo’s own trapframe page.

Step 5 of 20

Thirty-one stores through the user page table

Every user register goes into the trapframe, at the offsets of struct trapframe (kernel/proc.h:40). The user a0 comes back out of sscratch into t0 and is stored at offset 112. Here is part of echo’s trapframe just before line 76, as gdb read it through the user page table:

The bottom four rows are a message the kernel left for itself. They are the only way uservec can learn where the kernel’s stack and page table are, because it can’t reach any kernel memory yet.

Step 6 of 20

ld sp: a stack that cannot be used yet

Four loads from the trapframe:

0x3fffffa000 is the top of KSTACK(2), the kernel stack of process slot 2, so it is empty: whenever a process is in user mode, its kernel stack holds nothing (The stacks of xv6).

Yet for the next four instructions, sp points at nothing usable. That address exists only in the kernel page table, and satp still names echo’s. A push here would fault. So the stack strip says “none”, even though sp already holds its final value. uservec doesn’t push anything until line 92 has run.

tp, like sp, comes from a field the kernel wrote itself: cpuid reads the hart number from it for the rest of the trap.

Step 7 of 20

csrw satp: the page table changes under the running code

A satp value has two parts that matter here: the top four bits are MODE = 8, which means Sv39 paging, and the low 44 bits are the physical page number of the root page table. The kernel’s root is at 0x87fff000, echo’s at 0x87f24000.

The instruction after csrw is fetched from 0x3ffffff096 using the new page table, which also maps the trampoline at that address (kernel/vm.c:47). The two sfence.vma instructions bracket the switch: the first makes sure earlier stores (the 31 into the trapframe) are finished with the old translation; the second throws away cached user translations from the TLB (translation lookaside buffer).

With that one write, sp became real: 0x3fffffa000 is now mapped, and the kernel stack is usable. That is the second half of the stack switch begun on line 76.

Step 8 of 20

jalr t0: into C, with ra pointing back into the trampoline

jalr t0 jumps to 0x800025ca, usertrap, and writes the return address into ra. gdb at the first instruction of usertrap:

So usertrap is an ordinary C function that, when it returns, lands in userret. The return trip needs no special jump: the jalr already arranged it. Note that ra is a virtual address in the trampoline page, valid in both page tables.

This is also the first C code of the trap, and C needs a stack: usertrap’s prologue moves sp to 0x3fffff9fe0, a 32-byte frame at the top of echo’s kernel stack.

Step 9 of 20

usertrap checks SPP, then moves stvec

usertrap first reads sstatus.SPP. It is 0, so the trap really came from user mode. This is the only way software can tell where it came from: no CSR reports the current mode, only the previous one.

Then line 47 rewrites the trap vector:

From now on, any trap on hart 0 is a trap from the kernel, and must go to the handler that assumes a kernel stack and the kernel page table. uservec assumes the opposite of both. With SIE = 0 no interrupt can arrive before this line, and the code before it can’t fault, so the order is safe.

This is one of exactly three places that ever write stvec: trapinithart at boot, this line, and prepare_return on the way out (step 13).

Step 10 of 20

Copy sepc before anything can overwrite it

Line 52 copies sepc (0x354) into p->trapframe->epc, replacing the stale 0x7c. Line 54 reads scause: 8, a system call. Line 62 adds 4, so the trapframe now says epc = 0x358, the ret after the ecall (gdb confirmed both).

Why copy at all? sepc, scause and stval are per hart, not per process. The next trap on hart 0, from any cause, will overwrite them. The trapframe is per process and survives anything. Once the copy is made, the CSRs can be lost without harm.

killed takes echo’s p->lock for an instant to read p->killed, so a kill from another hart is seen here at the latest.

Step 11 of 20

intr_on, and a tick that was waiting

intr_on is one instruction, csrsi sstatus, 2, at 0x80002668:

Our run showed exactly why the comment on lines 64–65 exists. The tick went pending while gdb held the hart: sip showed the supervisor timer pending (STIP) already at the ecall. Run freely, it would have been taken in user mode before the ecall; our single-stepping kept it pending. The moment SIE became 1, the interrupt was taken through kernelvec, at the very next instruction. When we next stopped, in sys_write, the CSRs read:

The system call’s own sepc and scause were gone. Because usertrap had copied them (step 10), nothing was lost. Had intr_on come first, echo would have resumed at a kernel address, faulted (scause = 12) and been killed: we ran it.

Line 66 is one of only three calls of intr_on() (the others are the scheduler’s line 441 and pop_off, when noff drops to 0 with intena 1), and like them it runs at noff 0, holding no spinlock. From here on, the first acquire of the system call records intena = 1, so the matching release turns interrupts back on (Locks and interrupt state).

Because it was a timer tick and echo was the current process, kerneltrap called yield: echo went through swtch to hart 0’s scheduler and straight back (it was the only runnable process) before syscall() began. (Tour 44: One interrupt, three landing sites follows a tick like this one.)

Step 12 of 20

The system call runs, deeper on the kernel stack

syscall reads a7 (16) from the trapframe and calls sys_write. gdb’s backtrace at sys_write’s first instruction, read on echo’s kernel stack:

#0  sys_write   sysfile.c:84     sp = 0x3fffff9fc0
#1  syscall     syscall.c:146
#2  usertrap    trap.c:68
#3  0x3ffffff09c                  (userret: where usertrap will return)

Line 146 puts the result, 2, into p->trapframe->a0.

None of the CSRs in this tour describe echo any more. They describe the most recent trap on whatever hart echo is running on. In our run, the UART’s “transmitter ready” interrupt arrived inside uartwrite, right after the byte was stored into the transmit register: when we stopped next, sepc was 0x8000094c and scause was 0x8000000000000009, a supervisor external interrupt. All that matters now lives in memory: the trapframe and the kernel stack.

Step 13 of 20

prepare_return: interrupts off, then stvec back to the trampoline

usertrap calls prepare_return. Two writes, in this order:

The order is the whole point. From line 112 until sret, the hart is in supervisor mode with a trap vector that expects to be entered from user mode. An interrupt in that window would jump to uservec with the kernel page table still installed. Its very first store, sd ra, 40(a0) into TRAPFRAME, would fault, because only user page tables map the trapframe. The fault goes to stvec, which is still uservec, and the same store faults again, forever: a silent hang. (We forced exactly this: over a million store faults at 0x3ffffff00c in 20 seconds, no message.) With SIE = 0, no interrupt can be taken, so the window is closed. The code between here and sret is also written so that it can’t fault.

Line 108 runs at noff 0: every lock the system call took is released, and the last pop_off turned SIE back on because intena was 1. Nothing after line 108 acquires a lock, so nothing can turn SIE on again before sret (Locks and interrupt state).

The uservec address is computed (TRAMPOLINE + (uservec - trampoline)) because the linker placed the code at 0x80006000. The hart must use the high alias that every user page table maps.

Step 14 of 20

The trapframe's message for next time

Four memory writes, no CSR writes. They refill the fields that uservec read in step 6, for the next trap from echo:

They are the same values as last time, because echo stayed on hart 0. Had a tick moved it to hart 2 during the system call, kernel_hartid would now become 2. That is correct: a process in user mode never changes hart, so the next trap will happen on the hart that is sending it there.

Step 15 of 20

Arming sret: SPP, SPIE and sepc

sret takes its orders from two CSRs, so they are written now:

In our run SPP and SPIE already had these values, so the csrw sstatus changed nothing that xv6 cares about. They are written anyway, because sstatus belongs to the hart, and the hart has had other business. A trap taken in the kernel sets SPP to 1; its own sret resets it to 0, but a thread that yields from inside such a trap leaves SPP = 1 behind on its hart, and the next thread that hart runs may come straight here (Tour 44: One interrupt, three landing sites measures exactly that kind of leftover).

sepc shows the problem plainly: it held 0x8000094c, the kernel address where the UART interrupt struck in uartwrite. p->trapframe->epc (0x358) is the only reliable copy of where echo should continue.

One subtlety, which the spec makes clear: while a hart runs in U-mode, supervisor interrupts are taken whatever SIE says. So SPIE = 1 is not what lets echo be interrupted. We checked by building a kernel that clears SPIE here: usertests preempt, whose children spin in user mode until a timer interrupt delivers a kill, still passed.

Step 16 of 20

usertrap hands back echo's satp

MAKE_SATP builds 0x8000000000087f24 from p->pagetable: MODE = 8 (Sv39) in the top bits, the root table’s page number below. usertrap returns it in a0.

gdb on usertrap’s final ret (0x80002690):

Everything is ready, and nothing has changed mode or page table yet. The kernel stack is empty because every C frame has been popped: the next trap will find it clean.

Step 17 of 20

userret: the page table goes back first

userret runs at 0x3ffffff09c. fence.i (line 107) makes this hart’s instruction fetches see the bytes now in memory, in case echo’s code was written into memory (by kexec, perhaps on another hart) since this hart last fetched instructions from those addresses. Then:

The mirror image of step 7: the code keeps running because the trampoline is mapped in both tables. But sp still holds 0x3fffffa000, and under echo’s page table that address means nothing. For the next five instructions the hart is in supervisor mode, on the user page table, with no usable stack, and it stays without one until sret (step 19). That’s fine, because the trampoline pushes nothing.

Step 18 of 20

ld sp, then every register, a0 last

li a0, TRAPFRAME, then 31 loads. Line 118 is the last of this round trip’s four stack switches: line 76, the two swtch loads of step 11’s yield, and this one. A system call that doesn’t yield has two.

a0 must be last, because it is the base register of every other load.

From line 118 until sret the hart is in supervisor mode with the user page table, and sp holds the user stack pointer, which supervisor code cannot use (it cannot access user pages): no usable stack, for 30 instructions, sret included. It is the same combination as the start of uservec before line 76, and sret, by switching to user mode, is what makes the user stack usable again. Mode, stack and page table: the master question lists every such combination.

Step 19 of 20

sret, exactly

The RISC-V privileged specification defines sret as these actions, all at once (plus bits of extensions xv6 doesn’t use):

1. pc ← sepc
2. mode ← sstatus.SPP (0 means U)
3. SIE ← SPIE
4. SPIE ← 1
5. SPP ← the least-privileged mode, U (0)
6. mstatus.MPRV ← 0, because the new mode is less privileged than M (xv6 never sets MPRV, so this changes nothing here)

Measured on hart 0:

Step 5 exists to catch bugs: after any sret, SPP reads 0. If kernel code ever ran a second sret without a new trap in between, it would drop to user mode instead of staying in supervisor mode, and fault at once. It would not quietly keep running with kernel privilege.

Step 20 of 20

Back in echo, and the ledger

ret returns to main with a0 = 2. From echo’s point of view one instruction ran. Here is everything that changed in between, and who changed it:

The hardware’s share fits in two rows. Everything else is policy, and RISC-V leaves policy to the kernel: which stack, which page table, what to save. That is why a 150-line trampoline can be the whole of xv6’s user/kernel boundary.

In our run the system call also absorbed two kernel traps (a timer, a UART interrupt), which overwrote sepc, scause and sstatus behind echo’s back. Next: Tour 44: One interrupt, three landing sites follows such a trap, and Tour 42: One hart's stacks, from power-on to the first user instruction follows sp from power-on.