Step 1 of 17

Three places, filled by three mechanisms

Three fields of struct proc point at the three places:

A fourth item is often forgotten: pagetable. The user’s memory, including its user stack, is reached only through it.

We are on hart 2 at kernel/proc.c:456, just after pid 5’s swtch returned into the scheduler. Pid 5’s p->lock is still held, so pid 5 cannot be resumed while we look: this is exactly the moment the frozen process is complete. In this tour we dump all three places for pid 5, and then compare.

Step 2 of 17

Place 1: the trapframe, 31 registers and a pc

The trapframe is one page from kalloc, pointed to by p->trapframe and mapped at TRAPFRAME (0x3fffffe000) in pid 5’s page table only. It is not on any stack. gdb read pid 5’s:

0x0505050505050505 is kalloc's junk fill (kernel/kalloc.c:80). init’s first trapframe page started as junk, userret loaded it into the registers, and since these programs never write gp, tp or t0, t2–t6, and write t1 only in printf’s printint (not yet called in this run), the junk has been copied through every fork since. It is harmless, and it is a fine fingerprint.

That is the whole user state: 31 general registers (x0 is always 0) plus the pc. Nothing else from user mode needs saving; the user’s memory stays put in its page table.

Step 3 of 17

Who wrote each trapframe field, and when

Four writers, at four different moments:

The trapframe also has readers besides the trampoline: argraw reads system-call arguments from it, kfork copies it, kexec rewrites epc and sp.

There is no lock on it. Apart from its birth (the parent’s kfork fills a child’s trapframe while the child is still USED and invisible) and its death (the parent’s freeproc, after the child is a ZOMBIE), only the process’s own kernel thread and the trampoline running on its behalf touch it, and a process runs on one hart at a time. While pid 5 is suspended, nobody touches it at all.

Step 4 of 17

Place 2: the kernel stack, 112 bytes deep

Pid 5’s KSTACK(4) page, from context.sp up to the top, word by word as gdb read it (gcc keeps frame pointers, so each frame’s saved s0 links to the next):

address      value            frame       meaning
0x3fffff5ff8 0x3ffffff09c     usertrap    ra: returns into userret (trampoline)
0x3fffff5ff0 0x11ec0          usertrap    saved s0  = the USER's s0
0x3fffff5fe8 0                usertrap    saved s1  = the USER's s1
0x3fffff5fe0 0x8430           usertrap    saved s2  = the USER's s2
0x3fffff5fd8 0x800026d0       yield       ra: back into usertrap (line 86)
0x3fffff5fd0 0x3fffff6000     yield       saved s0 (usertrap's frame)
0x3fffff5fc8 0x80010370       yield       saved s1 = &proc[4]
0x3fffff5fc0 2                yield       (slot not written)
0x3fffff5fb8 0x80001f02       sched       ra: back into yield
0x3fffff5fb0 0x3fffff5fe0     sched       saved s0 (yield's frame)
0x3fffff5fa8 0x80010370       sched       saved s1
0x3fffff5fa0 2                sched       saved s2 = which_dev, from usertrap
0x3fffff5f98 0                sched       saved s3  = the USER's s3
0x3fffff5f90 0x3fffff5fc0     sched       (slot not written)  ◄─ context.sp

No register file was dumped here wholesale. What is on a kernel stack is what any C call chain leaves: return addresses, frame links, locals, and callee-saved registers spilled by prologues. usertrap uses s0, s1 and s2, so its prologue saved the values those registers held on entry, and on entry they held the user’s values, because uservec never touches s registers. sched uses s3 (for intena, the thread’s own copy of the flag, Locks and interrupt state), so its prologue saved the user’s s3, which nothing between had touched. So four user registers (s0–s2 spilled by usertrap, s3 by sched) are now stored twice: in the trapframe and here.

Step 5 of 17

Place 3: p->context, 14 registers

Back on hart 2 a moment earlier, inside swtch on pid 5’s stack: lines 10–23 store ra, sp and s0–s11 into proc[4].context. gdb read the result:

Why only these 14? Because swtch is called like an ordinary function. The calling convention already lets a callee destroy a0–a7 and t0–t6, so sched cannot be relying on them, and saving them would be wasted work. ra says where to continue, sp where the stack is, and s0–s11 are exactly the registers a callee promises to give back. gp is unused by the kernel; tp holds the hart number and deliberately stays with the hart.

Step 6 of 17

Overlap, checked register by register

Put the three dumps side by side for the callee-saved registers:

So “no overlap” is false: every user s register is stored at least twice. And “each place holds different registers” is false too: the trapframe and p->context both have s0–s11, ra and sp, holding the user’s values in one and the kernel thread’s values in the other, which sometimes coincide. s4–s11 coincide because nothing on the path usertrap → yield → sched used them, so they still held what the user left in them when swtch saved them. s3 only looks the same: sched spilled the user’s s3 onto the stack and reused the register for intena, which is also 0 here.

The design is still correct, because each copy has exactly one reader:

• swtch restores the kernel thread from p->context;
• each function’s epilogue restores its own spills as the stack unwinds;
• userret restores the user from the trapframe.

A duplicate that only one reader ever uses can never disagree with anything. The accurate claim is: the trapframe is the complete user state, p->context plus the kernel stack are the complete kernel-thread state, and the two may share values.

Step 7 of 17

A consequence of the calling convention

Follow the user’s s registers through a trap. uservec leaves them untouched. usertrap is an ordinary C function called with jalr, so the calling convention obliges it to return with s0–s11 exactly as it found them, which were the user’s values. Everything it calls obeys the same rule, swtch included. So when usertrap returns into userret, the s registers already hold the user’s values, and the twelve ld s0 … ld s11 among lines 124–142 write the same numbers back.

Then why load them at all? Because one path breaks the chain: every new process’s first return, through forkret: each fork child, and init. forkret reaches userret through a function pointer, from a kernel thread that never came in through uservec: its s registers hold kernel values (they started as zeros in a forged context). For that thread, the trapframe copy, made from the parent’s by kfork, is the only place the child’s user s registers exist.

This is a fact about this build’s paths, not a promise the code makes. A future kernel that edits trapframe->s0, or reaches userret some other way, would need these loads, and the trampoline does not try to be clever: it restores everything.

Step 8 of 17

Asleep in read: 336 bytes and a syscall's arguments

Pid 4 called read(3, buf, 12288) on an empty pipe and slept. Its kernel stack, from gdb’s frame-pointer walk:

KSTACK(3), top 0x3fffff8000
  usertrap  0x3fffff7fe0  ra → userret; spills user s0, s1, s2 = 0x11ec0, 5, 0x8430
  syscall   0x3fffff7fc0
  sys_read  0x3fffff7f90
  fileread  0x3fffff7f60
  piperead  0x3fffff7f00  resumes at pipe.c:127, the line after sleep() (sleep's saved ra)
  sleep     0x3fffff7ee0
  sched     0x3fffff7eb0  ◄─ context.sp     (336 bytes in all)

Its trapframe still holds the system call as the user made it: a7 = 5 (SYS_read), a0 = 3 (the pipe’s read end), a1 = 0xccf8 (the user buffer), a2 = 0x3000, and epc = 0x5772, already advanced past the ecall. a0 will be overwritten with the result when read finishes.

Its context: s3 = 1 is sched’s copy of intena (interrupts were on when sleep took p->lock; piperead’s own s3, the buffer address 0xccf8, was spilled into sched’s frame). s4 = 0x80069218 (&pi->nread) and s5 = 0x3000 (n) are piperead’s variables. s6–s11 again equal the user’s. The kernel stack is the reason the process stopped, written as a call chain: a stack that ends in piperead → sleep means “waiting for a pipe”.

Step 9 of 17

Preempted inside the kernel, with a kernelvec frame

A third case, from a different moment (usertests writebig): the shell had forked pid 3 to run usertests, and pid 3 was inside kexec, looking up the path, when hart 1’s timer fired in the kernel. The trap went through kernelvec, which pushed 256 bytes onto whatever stack was current, pid 3’s own, and kerneltrap called yield. 1920 bytes of KSTACK(2) are in use:

usertrap → syscall → sys_exec → kexec → namei → namex → dirlookup
  → readi → brelse → releasesleep → release → pop_off
      (interrupted here, just after intr_on)
  → [kernelvec frame, 256 bytes] → kerneltrap → yield → sched

Part of the 256-byte frame, as gdb read it:

Only gp and the 16 caller-saved registers (ra, t0–t6, a0–a7) are saved; s registers are left to kerneltrap’s own prologue, as for any C function. The unwritten slots still hold junk: kernel stack pages come from kalloc at boot, filled with 0x05, and nobody clears them.

Step 10 of 17

Where the interrupted kernel pc is kept

The kernelvec frame has no slot for sepc. Yet pid 3 must resume exactly where pop_off was interrupted. Where is that address?

kerneltrap reads it into a local variable, and in this build gcc keeps the local in s2 (sstatus in s1, scause in s3). Those are callee-saved registers, so when kerneltrap calls yield, and yield calls sched, each prologue that wants those registers spills them. gdb found them in sched’s frame:

So the interrupted kernel pc is stored on the kernel stack by an ordinary function prologue, in sched’s frame, three frames below the kernelvec frame (under kerneltrap’s and yield’s). When pid 3 resumes (perhaps on another hart), the epilogues put it back in s2, and kerneltrap writes it to that hart’s sepc (kernel/trap.c:162) just before kernelvec’s sret. That is why kerneltrap keeps these values in locals instead of re-reading the CSRs: by the time yield returns, the hart’s sepc belongs to whatever ran in between (Tour 44: One interrupt, three landing sites).

This pc, 0x80000c50, also shows why the trap came here: the timer was already pending, and fired the instant pop_off re-enabled interrupts. A tick can be taken only where noff is 0, since SIE is on only then: pop_off turns it on only after noff has dropped to 0, so the tick never lands inside a critical section (Locks and interrupt state). Outside one it can land anywhere, even mid-work in memset or uartwrite (Tour 44: One interrupt, three landing sites); here it was simply already waiting. The root inode’s sleep-lock, still held, doesn’t count in noff.

Step 11 of 17

fork: a child with a forged context and an empty stack

A new process gets its three places from two functions. allocproc makes the context: all zeros, except ra = forkret and sp = p->kstack + PGSIZE. When hart 2’s scheduler first picked child pid 5 in our run, gdb showed exactly that:

proc[4].context: ra=0x8000193a (forkret)  sp=0x3fffff6000  s0..s11 = 0
kernel stack used: 0 bytes

The kernel stack itself was not allocated here. KSTACK(4) was allocated and mapped at boot by proc_mapstacks and belongs to the slot, so the child simply inherits an empty page that previous occupants of proc[4] once used. Nothing of the parent’s kernel stack is copied: the child will never return through sys_fork or syscall.

The trapframe page is new: kalloc at kernel/proc.c:129, full of 0x05 junk until kfork fills it.

Step 12 of 17

fork: the trapframe is copied, with one change

kfork copies the parent’s whole trapframe, then sets the child’s a0 = 0. The child’s user stack was copied by uvmcopy just before, page by page, so the copied sp = 0x11e80 points at an identical stack in the child’s own memory. gdb, at the child’s first scheduling:

The kernel fields are stale, and that is fine: the child cannot trap before it has returned to user mode once, and the only way there is forkret → prepare_return, which rewrites all four (kernel/trap.c:116–119). One line, at the right moment, makes a whole copied structure correct.

Line 279 runs while kfork still holds the child’s p->lock from allocproc (released at line 294), and the child is still USED: no other hart can see it yet.

Step 13 of 17

fork: the child's first run, top of an empty stack

swtch “returns” to forkret with sp at the very top of KSTACK(4). forkret releases the lock the scheduler took (with intena 0, recorded by the scheduler, so interrupts stay off: Locks and interrupt state), calls prepare_return, and jumps to userret through a pointer. Its frame is never popped: once sret happens, the next trap starts again from kernel_sp, the top, and writes over it.

Now count the child’s three places at the moment it reaches user mode:

• trapframe: the parent’s user registers, a0 = 0, fresh kernel fields;
• kernel stack: empty (as for any process in user mode);
• p->context: the forged values, never used again; the next swtch overwrites them.

The child’s user s registers come only from the trapframe, through userret’s loads. That is the path that makes those loads necessary.

Step 14 of 17

exec: the user half is replaced, the kernel half is not

kexec runs on the process’s kernel stack (the 1920-byte dump above came from inside it) and builds a new page table with a new user stack, while the old ones stay intact for a possible failure (Tour 22: exec). At the commit point:

The return value, argc, reaches a0 through syscall like any result. The new program starts with whatever s and t registers the old one had, a reason exec’d programs must not assume registers start at zero.

Step 15 of 17

exit: a zombie keeps its stack and its context

Pid 5 was killed. At its next timer trap, usertrap saw killed and called kexit directly, on pid 5’s kernel stack. kexit marked it ZOMBIE and called sched, which never returns. gdb, afterwards:

proc[4] pid 5 ZOMBIE   context: ra=0x80001e9c sp=0x3fffff5f80 s4=0xffffffffffffffff
frames: usertrap → kexit → sched       (128 bytes)

s4 = -1 is kexit’s status argument, kept in a callee-saved register and saved by swtch into a context that nobody will ever load. (Pid 4, which exited normally, left usertrap → syscall → sys_exit → kexit → sched, 192 bytes.)

kexit holds two spinlocks for a moment, wait_lock and its own p->lock, and releases wait_lock (line 360) before calling sched, which would panic with “sched locks” at noff 2 (Locks and interrupt state).

The zombie cannot free its own kernel stack: it is standing on it until swtch finishes, and it still holds p->lock. Its hart’s scheduler releases that lock only after sp has left this page. Its trapframe and user memory remain too, until the parent collects it.

Step 16 of 17

wait: the parent frees two places of three

The parent’s kwait finds the zombie under wait_lock and the child’s p->lock, and calls freeproc:

Then state = UNUSED, and slot 4, with its kernel stack, is free for the next process. When the next fork lands in proc[4], its forged context will start at the top of the same page, and the zombie’s old frames will be overwritten as the new process’s call chains grow down over them.

This is why a design where each process owned a freshly allocated kernel stack would need extra care: something other than the dying process would have to free it, after the last swtch off it. xv6 avoids the question by tying stacks to slots.

Step 17 of 17

The ledger of a suspended process

Everything that makes a process resumable, with its size in our dumps and its owner:

And what is not in any of them, because it belongs to a hart: tp, the scheduler stack, struct cpu, stvec, stimecmp. A process carries nothing hart-specific, which is precisely what lets it resume on any hart.

The three places are not three disjoint thirds of one register file. They are the saved state of two threads of control, the user program and its kernel thread, plus the kernel thread’s memory. Overlaps between them are harmless because each copy has a single reader.