Concept 2

Mode, stack and page table: the master question

Stop xv6 at any instruction, pick any hart, and ask three questions:

1. Which privilege mode? User (U), supervisor (S) or machine (M). It decides which instructions and which memory the code may use.
2. Which stack? Whichever memory sp points into: a user stack, a kernel stack, the hart’s boot or scheduler stack, or, briefly, none that may be used. The stacks of xv6 describes each one.
3. Which page table? The one satp names: a process’s user page table, the kernel page table, or none (paging off). It decides what every address means.

If you can answer all three at every step of a tour, you understand where you are. This page shows how the answers change. Two more settings decide what happens next, so they appear in the tables below as well: stvec, the address where the next trap in supervisor mode will start, and the SIE bit of sstatus, which says whether interrupts can be taken in supervisor mode.

Who changes what

The key to the whole page: the hardware changes the mode; xv6’s own instructions change the stack and the page table. A trap does not touch sp or satp, and neither do sret and mret. So every transition is a short sequence of steps, and in between the three answers can briefly disagree with each other.

When a trap is taken into supervisor mode, the hardware does exactly this: it saves the pc in sepc, the reason in scause (and a faulting address, if any, in stval), the previous mode in sstatus.SPP, copies SIE into SPIE and clears SIE, switches to supervisor mode, and jumps to stvec. Every register, sp included, keeps its value.

The eight transitions at a glance

T3 and T4 also happen while the hart is in supervisor mode; then they arrive as T7. T6 follows every sleep and exit; a timer interrupt (T4 or T7) is what forces a process that never blocks to give up the hart.

T1: system call

A user program calls a system call stub such as write in user/usys.S, which puts the call number in a7 and executes ecall. Follow the three answers:

Throughout, the kernel stack starts empty: usertrap's frame is the first thing on it. Interrupts stay off until usertrap has saved sepc and switched stvec, because an interrupt would overwrite sepc, scause and sstatus; for a system call they are turned on at line 66, so a long system call can be interrupted (T7). The way back is T5. Tour 5: Life of a system call, Tour 7: The trampoline and the trapframe and Tour 43: A system call, CSR by CSR follow this path step by step.

T2: user exception

The same entry as T1, through uservec onto the empty kernel stack and into usertrap, with a different scause. What usertrap does next depends on the cause:

• a load or store page fault (scause 13 or 15) at an address below the process size that is not mapped yet: vmfault allocates the page (lazy allocation, used by sbrklazy) and the program continues as if nothing had happened, through T5;
• anything else (an illegal instruction, a fault on a guard page or an unmapped address, …): usertrap prints unexpected scause, marks the process killed, and calls kexit on its kernel stack. That process never returns to user mode; its hart goes to the scheduler through T6.

Only the system-call branch of usertrap calls intr_on(), so an exception is handled with interrupts off. Tour 10: Exceptions and faults covers faults.

T3: device interrupt

Again the T1 entry, with scause = 0x8000000000000009 (supervisor external interrupt). devintr asks the PLIC which device interrupted, runs uartintr or virtio_disk_intr on the process’s kernel stack, and tells the PLIC it is done. The interrupted process returns to user mode through T5, usually unaware that anything happened. Note whose kernel stack this is: the handler runs on the stack of whichever process happened to be running, which is often not the process waiting for the device. The handler does the minimum (passes typed characters to the console code, or marks a disk request finished) and calls wakeup; the waiting process does the rest later, on its own stack. Tour 9: Device interrupts and the PLIC and Tour 44: One interrupt, three landing sites follow one interrupt.

T4: timer interrupt

start enabled the Sstc extension (timerinit), so each hart has a supervisor-mode compare register, stimecmp. When the hart’s time reaches it, the hart raises a supervisor timer interrupt directly, with scause = 0x8000000000000005. From user mode it arrives through the T1 entry. devintr calls clockintr, which on hart 0 advances ticks and on every hart sets the next stimecmp, about a tenth of a second later; devintr then returns 2. usertrap then calls yield (kernel/trap.c:86), which starts T6: the process is suspended with usertrap → yield → sched on its kernel stack.

Machine mode plays no part. Older RISC-V xv6 versions had to take timer interrupts in machine mode and pass them on; this one never enters machine mode after boot. Tour 11: From a timer tick to a context switch and Tour 45: One complete time slice on three harts follow a full time slice.

T5: return to user

The way out of every T1 to T4, and the way a new process reaches user mode for the first time.

A process that has never run takes a shortcut: forkret calls prepare_return and then jumps to userret through a function pointer (kernel/proc.c:542), from the top of its fresh kernel stack. After exec, the same steps load the new program’s sp and pc, which kexec stored in the trapframe.

T5 mirrors T1 step for step: the trap makes the user stack unusable and line 92 makes the kernel stack usable; line 111 makes the kernel stack unusable, line 118 loads the user’s sp, and only the sret, by changing the mode back to user, makes the user stack usable again. Supervisor code cannot use user pages (sstatus.SUM is never set in this tree), so between lines 118 and 153 userret uses no stack, just as uservec uses none before line 76.

Interrupts are off from line 108 until the sret, for a reason the comment at kernel/trap.c:105 gives: from line 112 on, stvec points at uservec, and an interrupt taken in supervisor mode would go there, onto code that assumes it came from user mode. Once in user mode, the supervisor interrupts enabled in sie are taken whatever SIE says (the RISC-V rule for a mode less privileged than the interrupt’s), and SIE itself is back to 1.

T6: context switch

Always between a process’s kernel thread and the hart’s scheduler, never directly between two processes. Process A on hart 1 gives the hart to process B:

The mode never changes, and neither does the page table: every kernel thread and every scheduler runs with the kernel page table, so swtch has no reason to touch satp. B’s user page table is installed only later, in T5. Interrupts are off throughout, because p->lock is held from sched's caller until the other side releases it. That handoff is also what stops another hart from picking A while hart 1 is still on A’s kernel stack. Tour 13: swtch and the lock handed across a context switch and Tour 45: One complete time slice on three harts go through it line by line.

T7: kernel trap

A trap taken while the hart is already in supervisor mode goes to kernelvec, because usertrap (or trapinithart, at boot) set stvec there.

Nothing switches here: same mode, same stack, same page table. Only interrupts are handled; an exception in kernel code means a kernel bug, and kerneltrap panics. The stacks of xv6 shows what the stack looks like with the frame on it, including the case where the process yields and is later resumed on another hart. Tour 8: Traps taken inside the kernel and Tour 44: One interrupt, three landing sites trace it.

T8: boot

mret is the “return from a trap that never happened” (Tour 46: Boot: returning from a trap that never happened): start fills in the registers a trap from supervisor mode would have left, and mret “returns” to supervisor mode at main. After that, machine mode is never entered again: mtvec and mscratch are never written in this tree, every exception that can occur in supervisor or user mode, and every supervisor-level interrupt, is delegated to supervisor mode (kernel/start.c:31–32), machine-level interrupts are not delegatable on QEMU (their mideleg bits read back 0) and xv6 never enables them, and the timer is a supervisor timer. Interrupts stay off until the scheduler’s first intr_on(). Tour 2: Power-on to main, on every hart at once and Tour 42: One hart's stacks, from power-on to the first user instruction walk through boot.

Every combination that occurs

These are all the combinations of mode, stack and page table that exist at this commit, with the trap vector and interrupt state that go with them:

Some combinations never occur, and each absence is a design decision:

• U-mode on a kernel or scheduler stack. The only way into user mode is the sret at kernel/trampoline.S:153, after line 118 has loaded the user’s sp.
• C code on a user stack, or with a user page table. The trampoline covers every moment where those are installed in supervisor mode, and it uses no stack.
• The user’s sp value with the kernel page table. uservec changes sp before satp, and userret changes satp before sp, so the user’s sp is only ever paired with the user page table.
• A user page table on the scheduler stack. The scheduler is reached only through swtch from kernel code, which always runs with the kernel page table.
• Machine mode after boot.
• Two harts on one kernel stack, or one hart on another hart’s scheduler stack. See The stacks of xv6.

Test yourself