Extension labs · lab 2 · The system-call and trap path · ★☆☆☆☆
A kernel backtrace on panic
When this kernel panics it prints one line, such as panic: sched locks, and stops. The
line names the check that failed, not the path that led to it. In this lab you add
backtrace(), which prints the return address of every function on the current kernel
call chain, and make panic call it. On the host, addr2line turns the addresses
into file names and line numbers.
The walk itself is a five-line loop over the frame pointers the compiler
already maintains. The interesting part is where the loop must stop, and that depends on
which stack it is walking. A process’s kernel stack is one page, so its top is a
page boundary. A hart’s scheduler stack is a 4096-byte slice of stack0, which is
only 16-byte aligned, so its top is in the middle of a page, and its top 32 bytes hold two
frames left over from boot. And when an interrupt arrives in the middle of a call chain, kernelvec
pushes a 256-byte kernelvec frame that the walk has to step across. You will see all
three, in real output from three harts, and you will see what goes wrong, in
recorded runs, when the walk does not stop where it should.
How a RISC-V function’s prologue links its frame to its caller’s: with -fno-omit-frame-pointer, s0 points just above a two-word record holding the return address (at s0-8) and the caller’s s0 (at s0-16), so the frames form a linked list through the stack.
What sits at the top of each kind of kernel stack: on a process’s kernel stack, usertrap's record, holding the address after uservec’s jalr (the first instruction of userret) and the user’ss0; on a scheduler stack, start’s and main’s frames, never popped, with start’s record holding the ra and s0 of power-on.
Why stack0 slices are not page-aligned, and what PGROUNDUP(fp) does on them: it stops the walk too late (printing start’s power-on record, and, without an upward check, following its s0 of 0) or too early (cutting the chain at a page boundary in the middle of the slice). Both happen in one recorded run.
What the walk sees across a trap: kernelvec saves ra but not s0 and builds no record, so the chain goes from kerneltrap straight to the interrupted function’s frame, and the interrupted function’s own name never appears (nor its caller’s, if the trap strikes in a prologue or epilogue); its address is in sepc.
How to turn return addresses into lines with addr2line, and why you should look up ra - 1: with optimization, the instruction after a call can belong to another line, or to another block of code entirely.
Why a backtrace in panic must run before panicked is set and must never fault: a fault inside it re-enters kerneltrap, which panics again. A recorded run of a walk without bounds went down through every kernel stack below its own and their guard pages, then through the top of physical memory, before it froze.
git clone https://github.com/ShowMeTheStack/xv6-riscv-labs
cd xv6-riscv-labs
git checkout -b my-backtrace 06aad25 # start your own
git diff 06aad25 origin/ext/02-backtrace # only when you want the answer
1. The spec
The function.void backtrace(void) prints backtrace: and then, one per line, the
return addresses of the current kernel call chain, newest first, in the form printk’s
%p produces (0x0000000080002c46). It must stop at the top of the stack it started
on, whichever kind of kernel stack that is, and it must never read outside that stack:
it is about to be called from panic, where a fault would start another panic.
panic.panic prints its message, then the backtrace, then stops as before.
A way to try it from user space. A system call int kbacktrace(int where) makes the
kernel print a backtrace from one of three places:
where
where the backtrace is taken
stack it walks
0
inside the system call
the calling process’s kernel stack
1
in kerneltrap, at the next timer interrupt that arrives while this system call is running
in kerneltrap, at the next timer interrupt that some idle hart takes in scheduler
that hart’s slice of stack0
It returns 0 when the backtrace has been printed, and -1 for any other where or if no
hart took a where 2 request within 10 ticks.
The test.bttest calls kbacktrace with 0, 1, 2 and 3 and checks the return values
(0, 0, 0, -1). It cannot read the console, so the addresses themselves are checked by eye,
by symbolizing them on the host; its last line says so:
$ bttest
bttest: from a system call, on this process's kernel stack:
backtrace:
0x0000000080002c46
0x00000000800029f2
0x0000000080002736
bttest: kbacktrace(0) returned 0: OK
bttest: from a timer interrupt during a system call:
backtrace:
...
bttest: kbacktrace(3) returned -1: OK
bttest: 4 checks OK; the addresses can only be checked by symbolizing them on the host with addr2line
bttest N runs only case N, to look at one stack at a time.
Constraints. Nothing else may change: no new output from any other program, and
usertests -q must still print ALL TESTS PASSED on three harts. Do not change the
compiler flags; the kernel is already built with -fno-omit-frame-pointer.
2. Think first
Answer each question in your head (or on paper) before opening a hint. Hints get more specific; the reference answer comes last.
1What does a function leave behind that leads to its caller?
To print the chain of calls that led to the current function, you need a way to get
from a function to its caller, then to the caller’s caller, and so on. When a C
function in this kernel is called, what does it store on the stack that would let you
do that, and where exactly? Look at a few real prologues before you answer.
Hint 1.
Build the kernel and open kernel/kernel.asm. Read the first four or five
instructions of cpuid, usertrap and scheduler. Then find the compiler flags on
line 66 of the Makefile.
Hint 2.
Two registers are saved by every one of those prologues: ra, the address to return
to, and s0. Then s0 is set from sp. Work out where the two saved values sit
relative to the news0, and whose s0 the saved one is.
Hint 3.
After the prologue, s0 points just above a two-word record: the return address at
s0-8 and the caller’s s0 at s0-16. Loading s0-16 gives you the caller’s
record, and so on up the stack.
The reference design
Here is cpuid, in the branch’s build, a function so small that it calls nothing:
The prologue makes room, saves ra and the caller’s s0, and points s0 at the old
sp. So ra is at s0-8 and the caller’s s0 at s0-16, and the same holds for
usertrap (addi sp,sp,-32, sd ra,24(sp), sd s0,16(sp), addi s0,sp,32) and
every other C function in the kernel. Variadic functions differ in one detail:
printk keeps its register save area above the record (addi s0,sp,128 in a
192-byte frame), but the record is still at s0-8 and s0-16.
The saved s0 values link the frames into a list running up the stack, newest first.
That list is what -fno-omit-frame-pointer buys (-fno-omit-frame-pointer; line 66 of the
Makefile). Without it, GCC at -O treats s0 as one more callee-saved register and
keeps a variable in it; clinic 4 shows what a walk finds then.
Notice that cpuid saves ra even though it never calls anything: with this flag every
function, leaf or not, gets a record, so the walk never has to know which kind it is in.
Check yourself
1warm-upType a number
usertrap begins addi sp,sp,-32; sd ra,24(sp); sd s0,16(sp); sd s1,8(sp); sd s2,0(sp); addi s0,sp,32. At what offset from the new s0 is the caller’s s0
saved? (Write a negative number.)
decimal, 0x hex or 0b binary
2solidDecode the bits
A copy of the kernel built without-fno-omit-frame-pointer printed, from a
backtrace taken in kerneltrap, fp 0x0000000200000120 is not on a kernel stack:
s0 held some variable of kerneltrap’s. Decode the value as the register it
really came from, sstatus.
Value: 0x200000120
2How does C code get hold of the first link?
The walk starts from the current function’s frame pointer. C has no way to name a
register. How will your walker read s0, and whose frame pointer will it get? Would
reading sp do as well?
Hint 1.
kernel/riscv.h already has a dozen two-line functions that each read one
register. Look at the ones for sp and tp, and at how they are declared.
Hint 2.
A static inline function’s body is copied into its caller, so an asm statement in
it runs in the caller’s frame. And sp is the bottom of the current frame, where
nothing is stored at a fixed offset.
Hint 3.
One more static inline accessor beside r_sp, for the other register, called
first thing in the walker. Then ask whose record the first iteration prints.
The reference design
Add r_fp() to kernel/riscv.h, next to r_sp and r_ra: one mv %0, s0
instruction in a static inline function. Because it is inlined, the instruction runs
inside backtrace itself, after backtrace’s prologue has set s0, so the value is
backtrace’s frame pointer. The first line printed is then the return address saved by
backtrace’s own prologue: the place in the caller where backtrace() was called,
which is exactly where you want the list to start. (GCC’s
__builtin_frame_address(0) gives the same value.)
sp will not do. It points at the bottom of backtrace’s frame, below which there is
only memory that calls frombacktrace will use. A copy of the branch with
r_sp() in place of r_fp() printed one line, 0x0000000000000017, for each of
the three cases, and stopped. That is not a code address: the call
printk("backtrace:\n") had just written printk’s varargs save area right below
backtrace’s frame, so *(sp-8) was printk’s saved copy of a7.
Check yourself
1solidChoose one
Why is the register reader declared static inline rather than as an ordinary
function in some .c file?
3Where does the chain end on a process's kernel stack?
Start from a system-call handler and follow the saved frame pointers up. Every record
leads to the caller’s record, until you reach the outermost frame of the kernel stack.
What is stored in that frame’s record, and what happens if the walk follows it?
How will the walker know where to stop?
uservec enters usertrap with jalr t0 and never sets s0, so usertrap’s
prologue saves a return address into the trampoline and whatever s0 the user
program had. The kernel stack is exactly one page, at a page-aligned address, and
usertrap’s frame pointer is its top.
Hint 3.
Compute the top once, from the first fp, as the next page boundary above it, and
stop as soon as fp reaches it. As a second guard, stop if a saved frame pointer is
not above the current one: callers’ frames are always higher.
The reference design
bttest’s kernel stack, read with gdb at the moment of its first backtrace (pid 3,
slot 2, KSTACK(2) = 0x3fffff9000):
frame pointer
function
its record: return address
saved s0
0x3fffffa000
usertrap
0x3ffffff09c (after uservec’s jalr)
0x3fd0 (the user’s)
0x3fffff9fe0
syscall
0x80002736 (in usertrap)
0x3fffffa000
0x3fffff9fc0
sys_kbacktrace
0x800029f2 (in syscall)
0x3fffff9fe0
0x3fffff9f90
backtrace
0x80002c46 (in sys_kbacktrace)
0x3fffff9fc0
usertrap was not called by a kernel function. uservec loaded sp with the empty
stack’s top (kernel/trampoline.S:76) and jumped with jalr t0
(kernel/trampoline.S:98), so usertrap’s record holds the address after that
jalr (the trampoline page is mapped at 0x3ffffff000) and the user program’s own
frame pointer, 0x3fd0, a user virtual address. Following it means loading from
0x3fc8 with the kernel page table installed, which maps nothing there: a load page
fault in the kernel, and kerneltrap panics (clinic 2).
So the walk must stop atusertrap’s frame, without printing its record. Its frame
pointer equals the top of the stack, and the top is easy to compute: kernel stacks are
exactly one page at page-aligned addresses, so for any fp strictly inside one, the
top is PGROUNDUP(fp), here 0x3fffffa000. The loop runs while (fp < top); the
usertrap line still appears, because syscall’s record holds the return address
into usertrap.
The second guard, “a saved fp must be higher than the current one”, costs one
comparison and protects against a corrupted chain, which is exactly what you may be
walking in a panic. Clinic 1 shows it catching a real mistake.
Check yourself
1warm-upType a number
The walk starts with fp = 0x3fffff9f90 on a process’s kernel stack. What is
PGROUNDUP(fp), the address where it stops? (Hex is fine.)
decimal, 0x hex or 0b binary
2solidTrue or false, and why
True or false: the record of usertrap’s frame, the outermost one on a process’s
kernel stack, holds a saved frame pointer that points to another frame on the same
kernel stack.
Why?
4How do you turn the printed numbers into names?
The walk prints numbers like 0x0000000080002736. The running kernel has no symbol
table to look them up in. How do you find the function and line for each one, and is
the line you get the line of the call?
Hint 1.
kernel/kernel is an ELF file built with -ggdb: it carries a symbol table and
line information. The cross toolchain has a tool that maps addresses to lines.
Hint 2.
A return address is the address of the instruction after the call. With
optimization, that instruction can belong to the next line, or to a different block
of code altogether when the called function never returns.
Hint 3.
Run ${TOOLPREFIX}addr2line -e kernel/kernel -f -p ADDR on each address, and once
more on ADDR - 1, which lands inside the call instruction itself. TOOLPREFIX is
your RISC-V toolchain’s prefix, the same one xv6’s Makefile detects
(riscv64-unknown-elf-, riscv64-linux-gnu- or riscv64-elf-); set it with
export TOOLPREFIX=riscv64-unknown-elf- or whichever you have. On
Debian/Ubuntu/WSL, gdb-multiarch also works as the debugger.
The reference design
On the host:
$ ${TOOLPREFIX}addr2line -e kernel/kernel -f -p 0x80002735
usertrap at ./kernel/trap.c:75
Asking for ADDR - 1 matters. Here are the five distinct addresses that cases 0 and 1
of bttest printed, looked up both ways:
printed
addr2line ADDR
addr2line ADDR-1
0x80002c46
sys_kbacktrace sysproc.c:129
sys_kbacktrace sysproc.c:128
0x800029f2
syscall syscall.c:148
syscall syscall.c:148
0x80002736
usertrap trap.c:88
usertrap trap.c:75
0x80002862
kerneltrap trap.c:170
kerneltrap trap.c:169
0x800057e8
kernelvec kernelvec.S:41
kernelvec kernelvec.S:38
The right column is the line of each call. The left one is wrong four times out of
five: with -O the instruction after a call often belongs to the next statement
(usertrap’s return address lands on line 88, the killed(p) check after the if).
It can be much worse in a function that never returns from a call; think question 6
meets such a case. ADDR - 1 is always inside the call instruction (4 bytes for
jal, 2 for the compressed c.jalr), so it always names the call. gdb does the same
when it prints caller frames.
One address will not resolve: 0x3ffffff09c, the address after uservec’s jalr t0
(kernel/trampoline.S:98), if you ever print it. The trampoline is linked at
0x80006000 but runs at TRAMPOLINE (0x3ffffff000). Even 0x8000609c gives only
userret at ??:?: it is userret's first instruction, and the trampoline has no
line information.
Check yourself
1warm-upChoose one
Case 0 printed 0x0000000080002736. ${TOOLPREFIX}addr2line -e kernel/kernel -f -p 0x80002736 says usertrap at ./kernel/trap.c:88, the if (killed(p)) after the
system call. Which line actually made the call that this address returns from?
5What does the walk see across a trap?
Suppose a timer interrupt arrives while some system-call handler f is running in the
kernel, and the walk starts inside the interrupt handler. kernelvec pushes 256
bytes onto the same stack and calls kerneltrap. Which return addresses will the
walk print, and which function on the path will not appear at all?
s0 is callee-saved, so kernelvec leaves it alone: when kerneltrap’s prologue
saves “the caller’s s0”, it saves f’s frame pointer. And kerneltrap’s return
address points into kernelvec.
Hint 3.
The walk goes from kerneltrap’s record (return address in kernelvec) straight
to f’s record, whose return address is in f’s caller. Nothing records where in
f the interrupt struck; that is sepc.
The reference design
kernelvec saves ra, gp, the t and a registers, and leaves sp, tp and the
callee-saved s registers alone (kernel/kernelvec.S:17 to
kernel/kernelvec.S:35). It builds no record. So the 256 bytes are invisible to the
walk: kerneltrap’s saved s0 is the interrupted function’s frame pointer, and the
chain goes on through f’s frame as if kerneltrap had been called from inside f.
(That holds once f’s prologue has set s0 and before its epilogue reloads it. If the
trap strikes inside f’s prologue or epilogue, s0 still holds the caller’s frame
pointer, and the caller is missing as well; a stack-overflow fault, for example, lands
in a prologue.)
The recorded case 1, sys_kbacktrace spinning with interrupts on and the timer firing
on hart 2:
frame pointer
function
return address in its record
saved s0
0x3fffff9fe0
syscall
0x80002736 (usertrap)
0x3fffffa000
0x3fffff9fc0
sys_kbacktrace
0x800029f2 (syscall)
0x3fffff9fe0
kernelvec’s 256 bytes, 0x3fffff9e90 to 0x3fffff9f90
no record
0x3fffff9e90
kerneltrap
0x800057e8 (kernelvec)
0x3fffff9fc0
0x3fffff9e60
backtrace
0x80002862 (kerneltrap)
0x3fffff9e90
The console showed kerneltrap, kernelvec, syscall, usertrap. Each line names
the caller of a frame, so sys_kbacktrace, the function that was interrupted, never
appears: no record holds an address inside it. Its address is in sepc
(0x80002c64, the spin loop on line 135 of sysproc.c), which kerneltrap keeps in
a register. When a fault (not an interrupt) panics through kerneltrap, the
scause=... sepc=... line it prints first is the only place the faulting function is
named.
gdb’s own bt does worse here: it stops at kernelvec with “Backtrace stopped: frame
did not save the PC”, because kernelvec has no unwind information. The frame-pointer
walk crosses the trap only because kernelvec leaves s0 untouched.
Check yourself
1solidPut in order
In case 1, a timer interrupt hits sys_kbacktrace’s spin loop and kerneltrap
calls backtrace(). Put the functions named by the printed lines in the order they
are printed.
kernelvec
kerneltrap
usertrap
syscall
2deepChoose one
Why can the frame-pointer walk continue past kernelvec at all, when kernelvec
builds no frame record?
6Where is the top of a scheduler stack?
The same walk must work when the interrupted code is the scheduler loop on an idle
hart. That stack is not a process’s kernel stack: it is the hart’s slice of stack0,
the same memory it booted on. Is “the next page boundary above fp” still its top? If
not, what happens when the walk uses it, and how should the walker find the real top?
stack0 is aligned to 16 bytes, not 4096, and each slice is 4096 bytes, so slice
h ends at stack0 + 4096 * (h + 1), in the middle of a page. Above scheduler’s
frame sit main’s and start’s, and start’s record holds the ra and s0 the
hart had at power-on.
Hint 3.
Decide by address which kind of stack fp is on. Inside stack0: compute the
slice’s top from stack0’s address. Inside the kernel-stack area: next page
boundary. Anywhere else: print a warning and do not walk.
The reference design
In the unmodified kernel stack0 is at 0x80007890; in this branch’s build, at
0x800078d0. So in this build the three slices in use end at 0x800088d0,
0x800098d0 and 0x8000a8d0 (...890 in the unmodified one), none of them on a page
boundary. On hart 1, recorded with gdb during case 2:
PGROUNDUP(0x80009720) is 0x8000a000, 1840 bytes above the real top. A walk bounded
by it does not stop at start’s frame: it prints start’s return address
0x8000001a (spin), a line that describes no call. The upward check then stops at
the saved s0 of 0 (a copy of the branch with only this bound changed printed exactly
that extra line, three runs out of three). A loop without that check (clinic 3)
follows 0 and loads from 0xfffffffffffffff8, a kernel page fault. And the opposite
error is just as real: the page boundary 0x80009000 lies
2256 bytes below the top, so a walk that starts deeper than that computes
0x80009000 as the top and stops in the middle of the chain. Clinic 3 has a single
recorded run that does both.
The fix is to know the layout: stacktop(fp) checks whether fp is inside stack0
(base + (fp - base) / 4096 * 4096 + 4096), inside the 64 kernel stacks
(PGROUNDUP), or neither (return 0, and backtrace prints fp ... is not on a kernel stack instead of walking). The starting fp is always strictly inside its slice,
because backtrace is called by a function with a frame of its own, so the division
picks the right slice.
Symbolizing this chain (think question 4) holds one more trap: 0x80000f66 gives
main.c:14 (consoleinit()) as printed, main.c:44 minus one. main calls
scheduler on line 44 as its very last action, and the compiler placed the code of
the cpuid() == 0 branch right after that call, so the return address lands there.
main’s record is worth a second look. start never calledmain; it reached it
with mret (kernel/start.c:51). main’s prologue saved whatever was in ra at
that moment: the return address of start’s call to timerinit on line 44, a fossil
from boot that every scheduler-stack backtrace prints (0x800000c2 looks up as the
inlined r_mhartid as printed, start.c:44 minus one).
Check yourself
1solidChoose one
A backtrace contains 0x0000000080000f66, and addr2line reports
main at ./kernel/main.c:14, the line consoleinit();. Yet the stack belongs to
a hart that has been in the scheduler for minutes. What is going on?
2solidType a number
In the unmodified kernel stack0 is at 0x80007890 and each hart’s slice is 4096
bytes. What is the address of the top of hart 2’s slice, where _entry points
hart 2’s first sp? (Hex is fine.)
decimal, 0x hex or 0b binary
3solidFill in the machine state
Case 2: an idle hart takes a timer interrupt in the scheduler loop’s intr_on(); intr_off(); window, and kerneltrap calls backtrace(). Fill in the machine
state on that hart as backtrace starts.
7How can a user program make the kernel walk from each place?
Timer interrupts in the kernel all reach kerneltrap. What tells it whether a
process was interrupted (myproc)? Are interrupts on while a system call runs
(kernel/trap.c:66)?
The branch adds kbacktrace(where) and one variable in trap.c:
(b)sys_kbacktrace stores its pid in btwant and spins while (btwant != 0).
Interrupts are on during a system call, so the next timer interrupt on that hart
lands in the loop, and kerneltrap sees its own process’s pid. No other hart runs
that process, so only one hart can match.
Exactly one hart. Idle harts’ timers are not staggered: clinic 6 recorded all
three idle harts in kerneltrap within the same instant. A plain
if (btwant == -1) { btwant = 0; ... } lets two of them pass the test before either
clears it. __sync_bool_compare_and_swap(&btwant, want, -2) is a single atomic
step (an lr.w/sc.w loop on RISC-V), so exactly one hart wins.
Clear only after printing. The winner sets -2 (“printing”), prints, and only
then sets 0. Clearing the request first is a real bug: hart 0’s
timer interrupt runs clockintr, which wakes everything sleeping on ticks before
kerneltrap gets to the backtrace, so (reasoning from the code) when hart 0 won,
the woken bttest could run on another hart and print in the middle; the log does
not show which hart printed. A recorded run of a version that cleared first (which also
had clinic 2’s bug) printed b0x0000000080005768 and then
ttest: kbacktrace(2) returned 0: OK a few lines later.
Withdraw. If no hart has taken the request after 10 ticks, the system call takes
it back with another compare-and-swap (-1 to 0) and returns -1; if a hart took it in
the meantime, the swap fails and the call keeps waiting for the print to finish.
None of this is needed for panic, which calls the walker directly. It exists so
that bttest can show all three stacks on a healthy kernel.
Check yourself
1deepChoose all that apply
Which statements about the request variable btwant in the reference are true?
8What must panic do, and in what order?
Now make panic print a backtrace. Where in panic does the call go, and what must
the walker itself never do when it is called from there?
panicked makes the next character printed by any hart spin forever, including
the panicking hart’s. And a fault inside the walker is a trap from the kernel, which
goes to kerneltrap, which calls panic again.
Hint 3.
Which of the two flags must still be clear while the walk prints, and which may
already be set? And which of the guards you built earlier make sure the walk itself
cannot trap?
The reference design
panicking = 1;
printk("panic: ");
printk("%s\n", s);
backtrace(); // before panicked, which would freeze this hart's output too
panicked = 1;
Before panicked.uartputc_sync spins forever once panicked is set
(kernel/uart.c:108), on every hart: the flag does not know who set it. A copy of
the branch with the two lines swapped printed panic: sched locks and then nothing,
ever (clinic 5’s bug used as the trigger).
After panicking. While panicking is set, printk takes no lock
(kernel/printk.c:70). So a panic on a hart that already holds pr.lock still gets
its backtrace out, but two harts panicking at once interleave character by character:
clinic 6 shows exactly that.
Never fault. A fault inside the walk enters kerneltrap, which prints
scause=... and calls panic again, which walks again, from a deeper point on the same
stack, through the same bad link. Clinics 2 and 3 show where that recursion goes. With
the bounds, every load is at fp - 8 or fp - 16 for an fp at or above the first
frame pointer and below the stack’s top: inside the one stack the walk started on,
which is always mapped. And the upward check makes even a corrupted chain end.
Check yourself
1solidTrue or false, and why
True or false: if backtrace() were called after panicked = 1, it would still
print, because panicked is meant to freeze the other harts’ output.
Why?
3. Build it
Start from the frozen commit, in your own clone of xv6 (not this site’s xv6/):
git checkout -b my-backtrace 06aad25
Milestones.
Read s0. Add r_fp() to kernel/riscv.h. Nothing uses it yet; make must
still build (-Werror is on, so an unused static inline is fine but an unused
static function is not).
The walk, for kernel stacks.backtrace() in kernel/printk.c, bounded by
PGROUNDUP of the first fp, with the “saved fp must be higher” check, and its
prototype in kernel/defs.h. Nothing calls it yet; milestone 3 is its first test.
A system call.kbacktrace(0): the number in kernel/syscall.h, the table entry
and extern in kernel/syscall.c, entry("kbacktrace") in user/usys.pl, the
prototype in user/user.h (lab 1, trace, explains each one), and a three-line test
program. You should see three addresses; symbolize them (${TOOLPREFIX}addr2line -e kernel/kernel -f -p ADDR-1) and check that they are sys_kbacktrace, syscall,
usertrap.
Across a trap. The btwant request, the spin in sys_kbacktrace, and the check in
kerneltrap. Expect kerneltrap, kernelvec, syscall, usertrap.
The scheduler stack. Write stacktop() first, then the case-2 request with the
compare-and-swap and the sleeping wait. Expect kerneltrap, kernelvec, main,
start.
panic. One line. To see it work, break something on purpose in a scratch copy (the
clinic has several ideas) and check the backtrace against the panic message.
The test, then usertests -q, then the test again.
Debugging. Run QEMU on three harts (make qemu uses CPUS := 3). To see the frames
the walk should find, use gdb: make qemu-gdb in one terminal, ${TOOLPREFIX}gdb kernel/kernel and target remote with the port make qemu-gdb chose in another, break backtracebefore the first continue (in the recorded runs, breakpoints set after
attaching to an already-running QEMU did not fire), then at the breakpoint p/x $sp, p/x $s0, and x/2gx $s0-16 to see a record.
Walk it by hand with x/2gx on each saved s0. info symbol ADDR names a return
address. If your walk prints one strange line and stops, compare the line with $s0: an
address that looks like a stack address means you printed the saved fp instead of the
return address (clinic 1).
4. Debugging clinic
Each of these bugs was put into the reference solution on purpose and run on three harts. The symptom is exactly what happened. Try to explain it before revealing why.
1The two words of the record swapped
The return address and the saved frame pointer are read from each other’s slots:
while (fp < top) {
- printk("%p\n", (void *)*(uint64 *)(fp - 8));
- uint64 prev = *(uint64 *)(fp - 16);
+ printk("%p\n", (void *)*(uint64 *)(fp - 16));
+ uint64 prev = *(uint64 *)(fp - 8);
if (prev <= fp)
break; // a caller's frame is always higher: the chain is broken
What happened when we ran it
$ bttest
bttest: from a system call, on this process's kernel stack:
backtrace:
0x0000003fffff9fc0
bttest: kbacktrace(0) returned 0: OK
bttest: from a timer interrupt during a system call:
backtrace:
0x0000003fffff9e90
bttest: kbacktrace(1) returned 0: OK
bttest: from a timer interrupt on a scheduler stack:
backtrace:
0x000000008000a750
bttest: kbacktrace(2) returned 0: OK
bttest: kbacktrace(3) returned -1: OK
bttest: 4 checks OK; the addresses can only be checked by symbolizing them on the host with addr2line
Each case prints exactly one line, and it is not code. 0x3fffff9fc0 is
sys_kbacktrace’s frame pointer, the saved s0 in backtrace’s record (compare the
table in think question 3); 0x3fffff9e90 is kerneltrap’s frame pointer in case 1;
0x8000a750 is inside hart 2’s slice of stack0, kerneltrap’s frame pointer in
case 2. The “previous frame pointer” the walk then follows is really a return address,
0x8000..., which is lower than any kernel-stack address and lower than the
stack0 address it came from, so the upward check stops the walk after one line.
This is the guard doing its job. Without it, the walk would treat 0x80002c46 as a
frame pointer, print eight bytes of the kernel’s code as a “return address”, and
follow eight more as the next pointer. Every check of bttest
passes: only the eye catches this bug, and the one-line output is the clue.
2No bound at all, stop when fp is 0
A walk with no idea where the stack ends, stopping only at a null frame pointer:
uint64 fp = r_fp();
- uint64 top = stacktop(fp);
printk("backtrace:\n");
- if (top == 0)
- printk("fp %p is not on a kernel stack\n", (void *)fp);
- while (fp < top) {
+ while (fp != 0) {
printk("%p\n", (void *)*(uint64 *)(fp - 8));
- uint64 prev = *(uint64 *)(fp - 16);
- if (prev <= fp)
- break; // a caller's frame is always higher: the chain is broken
- fp = prev;
+ fp = *(uint64 *)(fp - 16);
}
(stacktop is deleted too; -Werror rejects an unused static function.)
What happened when we ran it
(on a scheduler stack, three runs of bttest 2 on one boot, all like this one:)
$ bttest 2
bttest: from a timer interrupt on a scheduler stack:
backtrace:
0x00000000800027e0
0x0000000080005768
0x0000000080000ee4
0x00000000800000c2
0x000000008000001a
bttest: kbacktrace(2) returned 0: OK
bttest: 1 checks OK; the addresses can only be checked by symbolizing them on the host with addr2line
(on a process's kernel stack, a fresh boot:)
$ bttest 0
bttest: from a system call, on this process's kernel stack:
backtrace:
0x0000000080002bc4
0x0000000080002970
0x00000000800026b4
0x0000003ffffff09c
scause=0xd sepc=0x80000858 stval=0x3fa8
panic: kerneltrap
backtrace:
0x00000000800008aa
0x000000008000279c
0x0000000080005768
0x0000000080002bc4
0x0000000080002970
0x00000000800026b4
0x0000003ffffff09c
scause=0xd sepc=0x80000858 stval=0x3fa8
panic: kerneltrap
backtrace:
[...]
(the same scause line ten times in all, the backtrace three lines longer each time; then:)
scause=0xf sepc=0x80005746 stval=0x3fffff8000
panic: kerneltrap
[...]
scause=0xf sepc=0x80005756 stval=0x3fffff6000
[...]
scause=0xf sepc=0x80005746 stval=0x3ffff80000
[...]
scause=0xf sepc=0x80005756 stval=0x88000000
[...]
scause=0xf sepc=0x80005746 stval=0x87f43000
[...]
scause=0xf sepc=0x80005756 stval=0x80011000
[...]
scause=0xd sepc=0x800006cc stval=0x1
[...]
scause=0xf sepc=0x80005744 stval=0x80008000
[...]
0x00000000800026b4
0x0000003ffffff09c
(about four minutes later, gdb attached:)
* 1 Thread 1.1 (CPU#0 [running]) uartputc_sync (c=115) at kernel/uart.c:109
2 Thread 1.2 (CPU#1 [running]) acquire (lk=0x80010d68 <proc+3960>) at kernel/spinlock.c:37
3 Thread 1.3 (CPU#2 [running]) acquire (lk=0x80010d68 <proc+3960>) at kernel/spinlock.c:37
[...]
Thread 1 (Thread 1.1 (CPU#0 [running])):
$6 = 0x80007810
On a scheduler stack the walk ends cleanly, one line too late. After main’s
record it reads start’s, at the slice’s top, and prints start’s return address,
0x8000001a (spin, the instruction after call start in _entry). start’s
saved s0 is the value s0 had at power-on, which in QEMU is 0, so while (fp != 0)
stops. Luck, not design.
On a process’s kernel stack the same loop is a disaster. After syscall’s
record comes usertrap’s: it prints 0x3ffffff09c (after uservec’s jalr) and follows the
user’s s0 (0x3fb0 in this run).
The load from 0x3fb0 - 8 = 0x3fa8 under the kernel page table is a load page fault
(scause 13) at sepc0x80000858, the ld in backtrace. kerneltrap prints
it and panics, and panic walks again, from deeper on the same stack, through the
same chain, to the same fault. Each round adds 368 bytes to the stack (in this build:
kernelvec 256, kerneltrap 48, panic 32, backtrace 32) and three lines to every
later backtrace.
backtrace’s frame ended 144 bytes below the top. After ten rounds the eleventh
kernelvec frame still fits (a gdb rerun logged it at sp0x3fffff9010, 16 bytes
above the bottom of the page), but kerneltrap’s prologue does not: its
sd s1,24(sp) stores to 0x3fffff8ff8, in the guard page. The guard page does not
stop anything. That fault traps to kernelvec again, which subtracts another 256
from sp and stores into the guard page again: 16 silent traps (counted in the gdb
rerun), until sp is inside the next kernel stack (slot 3’s), where kerneltrap
finally runs. The scause=0xf ... stval=0x3fffff8000 line reports the last of
those 16 faults (sd t0,32(sp) at sp0x3fffff7fe0), and the recursion continues. In the log the stval of
these reports steps down by 0x2000 at a time, one guard page per kernel stack, from
slot 2 through every slot below it (the last report, 0x3ffff80000, is slot 62’s
guard page; slot 63 follows). No other process was using those slots in this run; if
one had been, its saved context would have been overwritten.
Below the last kernel stack there is nothing mapped down to PHYSTOP, and the
silent traps carried sp all the way down (stval=0x88000000) into the kernel’s
direct map of RAM. Reasoning from the memory map from here on: the recursion then
wrote downwards through physical memory; later faults at 0x87f43000 and
0x80008000, addresses the kernel page table maps read-write, mean that page-table
pages had been overwritten too; and the fault at 0x80011000 lies inside proc[],
so the stack passed through the process table.
Logged by gdb: hart 0’s sp was 0x80007810, below stack0 (0x800078b0 in this
copy), among the kernel’s global variables, with scause 13 and stval0x3fa8, and
it was spinning in uartputc_sync at line 109 with c=115, the s of yet another
scause= line: the loop that runs once panicked is nonzero. No panic had finished
(panic never returns, and every hart was still in a fault or in acquire), so the
stack’s own writes had set panicked, at 0x80007880, which sp had passed. Harts 1
and 2 were spinning in acquire on a p->lock (inferred: one whose memory the
stack had overwritten), and gdb could not even read panicked: “Cannot access memory
at address 0x80007880”. An earlier run of the same copy, stopped earlier, had printed
431,519 lines of output.
The guard pages turn a stack overflow into a fault; they cannot stop a fault handler
that itself uses the overflowing stack.
3PGROUNDUP as the top, on stack0 too
The bound of commit 2 (PGROUNDUP of the first fp), kept for every stack, with the
plain loop that many published solutions use (no upward check):
uint64 fp = r_fp();
- uint64 top = stacktop(fp);
+ uint64 top = PGROUNDUP(fp);
printk("backtrace:\n");
- if (top == 0)
- printk("fp %p is not on a kernel stack\n", (void *)fp);
while (fp < top) {
printk("%p\n", (void *)*(uint64 *)(fp - 8));
- uint64 prev = *(uint64 *)(fp - 16);
- if (prev <= fp)
- break; // a caller's frame is always higher: the chain is broken
- fp = prev;
+ fp = *(uint64 *)(fp - 16);
}
In this copy stack0 is at 0x800078b0, so hart 0’s slice is 0x800078b0 to
0x800088b0.
What happened when we ran it
$ bttest 2
bttest: from a timer interrupt on a scheduler stack:
backtrace:
0x00000000800027f4
0x0000000080005778
0x0000000080000ef8
0x00000000800000c2
0x000000008000001a
scause=0xd sepc=0x80000868 stval=0xfffffffffffffff8
panic: kerneltrap
backtrace:
0x00000000800008be
0x00000000800027b0
0x0000000080005778
0x00000000800027f4
0x0000000080005778
0x0000000080000ef8
0x00000000800000c2
0x000000008000001a
scause=0xd sepc=0x80000868 stval=0xfffffffffffffff8
panic: kerneltrap
[...]
(five panics in all; the first four backtraces each three lines longer than the one before; the fifth:)
scause=0xd sepc=0x80000868 stval=0xfffffffffffffff8
panic: kerneltrap
backtrace:
0x00000000800008be
0x00000000800027b0
0x0000000080005778
(no more output; 15 seconds later, gdb attached:)
* 1 Thread 1.1 (CPU#0 [running]) panic (s=s@entry=0x80007390 "kerneltrap") at kernel/printk.c:164
2 Thread 1.2 (CPU#1 [halted ]) s_sstatus (x=2) at kernel/riscv.h:67
3 Thread 1.3 (CPU#2 [halted ]) s_sstatus (x=2) at kernel/riscv.h:67
[...]
Thread 1 (Thread 1.1 (CPU#0 [running])):
$6 = 0x80007f80
[...]
Thread 1 (Thread 1.1 (CPU#0 [running])):
#0 panic (s=s@entry=0x80007390 "kerneltrap") at kernel/printk.c:164
#1 0x00000000800027b0 in kerneltrap () at kernel/trap.c:160
#2 0x0000000080005778 in kernelvec () at kernel/kernelvec.S:38
Backtrace stopped: frame did not save the PC
Hart 0 took the request. Too late first. The walk started a few hundred bytes
below the slice’s real top, 0x800088b0 (432 bytes below it in the reference’s case
2), so PGROUNDUP gave 0x80009000, 1872 bytes above the top. The
walk printed kerneltrap, kernelvec, main and start correctly, then reached
start’s frame (frame pointer 0x800088b0, still below 0x80009000), printed its
return address 0x8000001a (spin), and followed its saved s0: 0, the value of s0
at power-on. 0 < 0x80009000, so it loaded from 0 - 8: stval=0xfffffffffffffff8,
at sepc0x80000868, the ld in backtrace. kerneltrap panicked, and the
panic’s walk went the same way, one level deeper each time.
Then too early. Each round pushed a kernelvec frame and kerneltrap’s,
panic’s and backtrace’s frames onto the same slice. In the fifth round panic’s
sp was 0x80007f80 (gdb), so that walk started below 0x80008000, and
PGROUNDUP gave 0x80008000, below the older frames. The fifth walk printed three
lines and stopped there, without faulting. So panic went on, set panicked, and spun at
line 164, its for (;;); harts 1 and 2 sat idle in wfi, and the console went quiet
with a truncated backtrace that ends in kernelvec.
In this run the recursion never got as far as the bottom of hart 0’s slice. If it had,
there is no guard page: in this copy the 48 bytes right below stack0 hold ticks,
btwant, initproc, kernel_pagetable, started, tx_chan and the two panic
flags, and the .data variables come next. gdb’s own bt stops at kernelvec, as it
always does, because kernelvec has no unwind information.
4Built without frame pointers
-fno-omit-frame-pointer removed from CFLAGS in the Makefile (as when you copy flags
from another project); the code is the reference:
$ bttest
bttest: from a system call, on this process's kernel stack:
backtrace:
fp 0x00000000800100e0 is not on a kernel stack
bttest: kbacktrace(0) returned 0: OK
bttest: from a timer interrupt during a system call:
backtrace:
fp 0x0000000200000120 is not on a kernel stack
bttest: kbacktrace(1) returned 0: OK
bttest: from a timer interrupt on a scheduler stack:
backtrace:
fp 0x0000000200000120 is not on a kernel stack
bttest: kbacktrace(2) returned 0: OK
bttest: kbacktrace(3) returned -1: OK
bttest: 4 checks OK; the addresses can only be checked by symbolizing them on the host with addr2line
At -O, GCC on RISC-V omits the frame pointer by default and uses s0 as an ordinary
callee-saved register. backtrace itself no longer sets s0 (its prologue in this
build saves s0 but has no addi s0,...), so r_fp() returns whatever value some
caller left in s0:
In case 0 that is 0x800100e0, which is &proc[2]: usertrap keeps p, its
struct proc pointer, in s0 (mv s0,a0 after myproc(); lw a2,48(s0) reads
p->pid).
In cases 1 and 2 it is 0x200000120, the sstatus value that kerneltrap keeps
in s0 so it can restore it at the end: SPP and SPIE set, UXL = 2.
Neither is inside stack0 or the kernel-stack area, so stacktop() returns 0 and the
walk refuses to start. With the plain PGROUNDUP loop of clinic 3, case 0 would have
read 0x800100d8 and 0x800100d0 (fields of a struct proc) as a return address and a
frame pointer and gone on from there. Checking that the starting fp is on a known
stack costs two comparisons and turns garbage into a clear message.
5Slept while holding tickslock
In case 2’s wait, the learner forgot this tree’s rule that the condition lock is
released beforesleep() (there is no sleep(chan, lock) here):
$ bttest 2
bttest: from a timer interrupt on a scheduler stack:
panic: sched locks
backtrace:
0x000000008000092c
0x0000000080001f94
0x0000000080002034
0x0000000080002ce4
0x00000000800029f2
0x0000000080002736
(symbolized on the host with addr2line, each address minus 1:)
panic at ./kernel/printk.c:183
sched at ./kernel/proc.c:488
sleep at ./kernel/proc.c:569
sys_kbacktrace at ./kernel/sysproc.c:153
syscall at ./kernel/syscall.c:148
usertrap at ./kernel/trap.c:75
(a second run under gdb, breakpoint on panic; $4 is `p cpus[1].noff`, $7 is `p cpus[1].intena`:)
Thread 2 hit Breakpoint 1, panic (s=s@entry=0x800071e0 "sched locks") at kernel/printk.c:179
[...]
$4 = 2
[...]
$7 = 1
This is what the lab is for: the panic message alone says only that sched found the
wrong number of locks. The backtrace says who called sched: sleep, called from
line 153 of sysproc.c, the sleep() in sys_kbacktrace’s case-2 loop, in a system
call from user space. From there the bug is one look away: tickslock is still held.
The chain: sleep() acquires the process’s own p->lock and calls sched(), which
requires noff == 1 (kernel/proc.c:487): exactly one spinlock, p->lock. With
tickslock also held, noff is 2. gdb on the panicking hart (hart 1) showed
cpus[1].noff = 2 and intena = 1 (interrupts were on when the system call took
tickslock). Note the backtrace starts at panic’s own call site: the first line is
the return address in backtrace’s record, inside panic.
6myproc() used without checking for 0
In kerneltrap, the learner reads the current process’s pid unconditionally, as if a
timer interrupt always interrupted a process:
if (which_dev == 2 && btwant != 0) {
- int want = myproc() != 0 ? myproc()->pid : -1;
+ int want = myproc()->pid;
What happened when we ran it
$ bttest 2
bttest: from a timer interrupt on a scheduler stack:
scause=0xd sepc=0x80002840 stval=0x30
scause=0xd sepc=0x80002840 stval=0xpan30
panic: ic: kkerneeltrneltrrapap
bacbackktractrace:
e:
00xx0000000000000800000080902c00092c
0x000000008000281e
0x0x000000000000080005700e8
0x00000000800057e8
0x000000000800800f606
0x000002800001e
0x0000008000000c2
00800057e8
(the capture ended here)
(a second run under gdb, breakpoint on panic:)
Id Target Id Frame
1 Thread 1.1 (CPU#0 [running]) 0x0000000080000a60 in uartputc_sync (c=115) at kernel/uart.c:114
2 Thread 1.2 (CPU#1 [running]) acquire (lk=lk@entry=0x8000f978 <pr>) at kernel/spinlock.c:37
* 3 Thread 1.3 (CPU#2 [running]) panic (s=s@entry=0x800073b0 "kerneltrap") at kernel/printk.c:179
Thread 3 (Thread 1.3 (CPU#2 [running])):
#0 panic (s=s@entry=0x800073b0 "kerneltrap") at kernel/printk.c:179
#1 0x000000008000281e in kerneltrap () at kernel/trap.c:160
#2 0x00000000800057e8 in kernelvec () at kernel/kernelvec.S:38
Backtrace stopped: frame did not save the PC
Thread 2 (Thread 1.2 (CPU#1 [running])):
#0 acquire (lk=lk@entry=0x8000f978 <pr>) at kernel/spinlock.c:37
#1 0x000000008000057a in printk (fmt=fmt@entry=0x80007388 "scause=0x%lx sepc=0x%lx stval=0x%lx\n") at kernel/printk.c:71
#2 0x0000000080002812 in kerneltrap () at kernel/trap.c:158
#3 0x00000000800057e8 in kernelvec () at kernel/kernelvec.S:38
Backtrace stopped: frame did not save the PC
Thread 1 (Thread 1.1 (CPU#0 [running])):
#0 0x0000000080000a60 in uartputc_sync (c=115) at kernel/uart.c:114
#1 0x000000008000029e in consputc (c=<optimized out>) at kernel/console.c:43
#2 0x0000000080000580 in printk (fmt=fmt@entry=0x80007388 "scause=0x%lx sepc=0x%lx stval=0x%lx\n") at kernel/printk.c:76
[...]
On an idle hart myproc returns 0, so myproc()->pid loads from address 0x30,
the offset of pid in struct proc: stval=0x30, at sepc0x80002840.
(addr2line says line 168, the compare-and-swap: the compiler moved the load of pid
from line 165 down to where it is used.) That is a page fault insidekerneltrap,
on the scheduler stack: kernelvec pushes a second frame below the first and calls a second
kerneltrap, which finds no device, prints scause=... and panics.
It happened on every idle hart at once. While bttest sleeps, all three harts are
idle, and their timer interrupts arrive within a hair of each other. In the gdb run,
hart 2 had reached panic, hart 1 was waiting for pr.lock to print its scause
line, and hart 0 was printing its own. In the console run two harts got through, and
once the first has set panicking, printk takes no lock: their messages and
backtraces interleave character by character. Untangled, and checked against the
code path, each backtrace is 0x8000092c (in panic), 0x8000281e (in the second kerneltrap,
line 160), 0x800057e8 (kernelvec), 0x800057e8 again, 0x80000f66 (main),
0x800000c2 (start).
Two kernelvec lines in a row mean two nested traps. And two functions are missing,
both “interrupted”: scheduler (interrupted by the timer) and the first kerneltrap
(interrupted by the fault). The first is named only by its sepc, which was never
printed; the second by the sepc in the scause= line, 0x80002840. gdb’s bt
stops at the first kernelvec.
5. The reference solution
Take the guided tour through the reference solution, one commit at a time, with the machine state at every step:
On the branch, on three harts, before usertests (the gdb run that the reveal follows
printed exactly the same lines):
$ bttest
bttest: from a system call, on this process's kernel stack:
backtrace:
0x0000000080002c46
0x00000000800029f2
0x0000000080002736
bttest: kbacktrace(0) returned 0: OK
bttest: from a timer interrupt during a system call:
backtrace:
0x0000000080002862
0x00000000800057e8
0x00000000800029f2
0x0000000080002736
bttest: kbacktrace(1) returned 0: OK
bttest: from a timer interrupt on a scheduler stack:
backtrace:
0x0000000080002862
0x00000000800057e8
0x0000000080000f66
0x00000000800000c2
bttest: kbacktrace(2) returned 0: OK
bttest: kbacktrace(3) returned -1: OK
bttest: 4 checks OK; the addresses can only be checked by symbolizing them on the host with addr2line
Symbolized on the host with ${TOOLPREFIX}addr2line -e kernel/kernel -f -p, each address
minus one:
kerneltrap trap.c:169, kernelvec kernelvec.S:38, main main.c:44, start start.c:44
Each is the expected chain: case 0 stops at the top of the kernel stack; case 1 crosses
the kernelvec frame and (as it must) has no line for the interrupted sys_kbacktrace;
case 2 stops at the top of the idle hart’s stack0 slice, after main and the boot-time
return address into start, without reading start’s record.
The rest of the kernel is unharmed, in the same boot:
$ usertests -q
usertests starting
[...]
ALL TESTS PASSED
bttest run again after usertests printed the same eleven addresses and
4 checks OK. No other program calls kbacktrace, so the only cost to an ordinary
kernel is one load of btwant per timer interrupt taken in the kernel.
How deep are the stacks the walk climbs? Measured with gdb on the branch (frame
pointers at the moment backtrace starts, against each stack’s top):
case
stack
top
backtrace’s fp
depth
0
bttest’s kernel stack
0x3fffffa000
0x3fffff9f90
112 bytes, 3 frames
1
the same, with a kernelvec frame
0x3fffffa000
0x3fffff9e60
416 bytes (256 of them kernelvec’s)
2
hart 1’s slice of stack0
0x800098d0
0x80009720
432 bytes
How much of the 4096 bytes does real work use?kalloc fills every page with the
byte 5, including the 64 kernel stacks that proc_mapstacks allocates once at boot, and
stack0 starts out zero (QEMU’s RAM is zeroed, and nothing else writes it). So after a run, the lowest byte of a stack that no longer holds
its initial value marks the deepest point that stack ever reached (to within a few bytes,
if a stored value happened to end in that byte). Read with gdb after one boot running
usertests -q and then bttest, on the branch:
kernel stacks: all 64 slots had been used; the deepest went 1632 bytes down
(slots 1, 2 and 4), then 1232, 1200, 1152 and less. The same 1632 bytes were already
reached after just booting and running echo hi, so usertests -q never went deeper
than the shell does.
scheduler stacks: hart 0 672 bytes (all of boot ran on it), hart 1 640, hart 2
767.
So the deepest normal path leaves about 2.4 KiB of each 4 KiB kernel stack unused, and
the guard page is never approached. What fills a stack is recursion that should not
happen: in clinic 2, each round of fault, kerneltrap, panic and walk took 368 bytes,
and in the eleventh round kerneltrap’s frame reached the guard page.
What PGROUNDUP gets wrong on stack0, in numbers. In this build the slice tops are at
0x...8d0: 2256 bytes above a page boundary and 1840 bytes below the next one. A walk
that starts in the top 2256 bytes of a slice (every scheduler-stack walk in the recorded runs of a
healthy kernel: the deepest scheduler-stack use measured was 767 bytes) gets a bound 1840
bytes too high and reads start’s record; one that starts deeper gets a bound inside the
slice and stops early.
Clinic 3 recorded both in one run: five walks too long, then one too short.
7. Go further
Name the interrupted function. When the walk meets the return address into
kernelvec, the function that was interrupted is missing. Have kernelvec store
sepc in its frame (there is room in the 256 bytes) and print it as an extra line.
Teaches what unwind information is, and why gdb stops at kernelvec.
Backtraces of sleeping processes on Ctrl-P. A process that is not running has its
frame pointer in p->context.s0 and its return address in p->context.ra. Make
procdump walk each sleeping process’s kernel stack. Teaches where a suspended
process lives (Tour 47: Where a suspended process lives) and why the walk must take the stack’s top from the process
(p->kstack + PGSIZE), not from the hart.
Symbols inside the kernel. Generate a sorted table of function start addresses at
build time (from ${TOOLPREFIX}nm), link it into the kernel, and print names instead of
numbers. Teaches a two-pass build and why the table’s own size moves every address.
A stack high-water mark. Repeat the measurement above from inside the kernel: check
the lowest byte still holding 5 in each kernel stack, and print it on Ctrl-P. Teaches
stack painting and how close the kernel runs to its guard pages.
A user-space backtrace on a fatal fault. When usertrap kills a process for an
unexpected scause, walk the user stack from p->trapframe->s0, using copyin
for every load. Teaches why the kernel may never dereference a user pointer, and what
bounds a user stack has (one page, a guard page below, kernel/exec.c:91).