Trampoline and trapframe · xv6, line by line

1warm-upChoose one

proc_pagetable maps the trampoline page at TRAMPOLINE in every user page table, and kvmmake maps the same page at the same address in the kernel page table. Why must the address be the same in both?

kernel/proc.c

// map the trampoline code (for system call return)
// at the highest user virtual address.
// only the supervisor uses it, on the way
// to/from user space, so not PTE_U.
if (mappages(pagetable, TRAMPOLINE, PGSIZE, (uint64)trampoline,
             PTE_R | PTE_X) < 0) {
  uvmfree(pagetable, 0);
  return 0;

To save memory: each page table would otherwise need its own copy of the code.Because the kernel’s text is mapped at the same address in user page tables too.So that user programs can call kernel functions at a known address.uservec and userret switch satp in the middle of their own code; the next instruction is fetched through the new page table, so the code page must be at the same address in both.

2warm-upChoose one

The first instruction of uservec is csrw sscratch, a0. Why does it start there?

kernel/trampoline.S

22uservec:    
#
      # trap.c sets stvec to point here, so
      # traps from user space start here,
      # in supervisor mode, but with a
      # user page table.
      #
29
      # save user a0 in sscratch so
      # a0 can be used to get at TRAPFRAME.
      csrw sscratch, a0
33
      # each process has a separate p->trapframe memory area,
      # but it's mapped to the same virtual address
      # (TRAPFRAME) in every process's user page table.
      li a0, TRAPFRAME
      

Because the hardware requires the trap handler to save a0 before anything else.Every general register holds a user value that must be saved, and saving needs an address in a register; parking a0 in sscratch frees a0 to hold TRAPFRAME.Because a0 holds the system-call number, which the kernel needs first.To push a0 onto the kernel stack.

3warm-upType a number

What is the virtual address TRAPFRAME in this kernel? (Answer in hex.)

kernel/memlayout.h

46// map the trampoline page to the highest address,
47// in both user and kernel space.
48#define TRAMPOLINE (MAXVA - PGSIZE)
49
50// map kernel stacks beneath the trampoline,
51// each surrounded by invalid guard pages.
52#define KSTACK(p) (TRAMPOLINE - ((p) + 1) * 2 * PGSIZE)
53
54// User memory layout.
55// Address zero first:
56//   text
57//   original data and bss
58//   fixed-size stack
59//   expandable heap
60//   ...
61//   TRAPFRAME (p->trapframe, used by the trampoline)
62//   TRAMPOLINE (the same page as in the kernel)
63#define TRAPFRAME (TRAMPOLINE - PGSIZE)

decimal, 0x hex or 0b binary

4warm-upTrue or false, and why

True or false: every process has its own trampoline page.

TrueFalse

Why?

There is one physical trampoline page, part of the kernel’s text; every page table maps that same page at TRAMPOLINE. Each process has its own trapframe page instead.Each hart has its own trampoline page, shared by all the processes that run on it.There is no trampoline page at all in user page tables; only the kernel maps it.

5warm-upType a number

How many general-purpose registers does uservec save into the trapframe?

kernel/trampoline.S

      # save the user registers in TRAPFRAME
      sd ra, 40(a0)
      sd sp, 48(a0)
      sd gp, 56(a0)
      sd tp, 64(a0)
      sd t0, 72(a0)
      sd t1, 80(a0)
      sd t2, 88(a0)
      sd s0, 96(a0)
      sd s1, 104(a0)
      sd a1, 120(a0)
      sd a2, 128(a0)
      sd a3, 136(a0)
      sd a4, 144(a0)
      sd a5, 152(a0)
      sd a6, 160(a0)
      sd a7, 168(a0)
      sd s2, 176(a0)
      sd s3, 184(a0)
      sd s4, 192(a0)
      sd s5, 200(a0)
      sd s6, 208(a0)
      sd s7, 216(a0)
      sd s8, 224(a0)
      sd s9, 232(a0)
      sd s10, 240(a0)
      sd s11, 248(a0)
      sd t3, 256(a0)
      sd t4, 264(a0)
      sd t5, 272(a0)
      sd t6, 280(a0)
70
# save the user a0 in p->trapframe->a0
      csrr t0, sscratch
      sd t0, 112(a0)
74

decimal, 0x hex or 0b binary

6warm-upClick the line

In userret, one register must be restored last. Click the line that restores it.

kernel/trampoline.S

      li a0, TRAPFRAME
115
      # restore all but a0 from TRAPFRAME
      ld ra, 40(a0)
      ld sp, 48(a0)
      ld gp, 56(a0)
      ld tp, 64(a0)
      ld t0, 72(a0)
      ld t1, 80(a0)
      ld t2, 88(a0)
      ld s0, 96(a0)
      ld s1, 104(a0)
      ld a1, 120(a0)
      ld a2, 128(a0)
      ld a3, 136(a0)
      ld a4, 144(a0)
      ld a5, 152(a0)
      ld a6, 160(a0)
      ld a7, 168(a0)
      ld s2, 176(a0)
      ld s3, 184(a0)
      ld s4, 192(a0)
      ld s5, 200(a0)
      ld s6, 208(a0)
      ld s7, 216(a0)
      ld s8, 224(a0)
      ld s9, 232(a0)
      ld s10, 240(a0)
      ld s11, 248(a0)
      ld t3, 256(a0)
      ld t4, 264(a0)
      ld t5, 272(a0)
      ld t6, 280(a0)
147
# restore user a0
      ld a0, 112(a0)
      
      # return to user mode and user pc.
      # prepare_return() sets up sstatus and sepc.
      sret

Your pick: none yet (click a line in the code)

7warm-upPut in order

Put the steps of userret in the order it executes them.

kernel/trampoline.S

globl userret
101userret:
      # usertrap() returns here, with user satp in a0.
      # return from kernel to user.
104
      # flush icache, in case this is the first time
      # we're running this proc on this hart.
      fence.i
108
      # switch to the user page table.
      sfence.vma zero, zero
      csrw satp, a0
      sfence.vma zero, zero
113
      li a0, TRAPFRAME
115
      # restore all but a0 from TRAPFRAME
      ld ra, 40(a0)
      ld sp, 48(a0)
      ld gp, 56(a0)
      ld tp, 64(a0)
      ld t0, 72(a0)
      ld t1, 80(a0)
      ld t2, 88(a0)
      ld s0, 96(a0)
      ld s1, 104(a0)
      ld a1, 120(a0)
      ld a2, 128(a0)
      ld a3, 136(a0)
      ld a4, 144(a0)
      ld a5, 152(a0)
      ld a6, 160(a0)
      ld a7, 168(a0)
      ld s2, 176(a0)
      ld s3, 184(a0)
      ld s4, 192(a0)
      ld s5, 200(a0)
      ld s6, 208(a0)
      ld s7, 216(a0)
      ld s8, 224(a0)
      ld s9, 232(a0)
      ld s10, 240(a0)
      ld s11, 248(a0)
      ld t3, 256(a0)
      ld t4, 264(a0)
      ld t5, 272(a0)
      ld t6, 280(a0)
147
# restore user a0
      ld a0, 112(a0)
      
      # return to user mode and user pc.
      # prepare_return() sets up sstatus and sepc.
      sret

fence.i: make this hart’s instruction fetches see what is now in memory
sret: drop to user mode at sepc
li a0, TRAPFRAME
load sp and the other user registers, except a0
ld a0, 112(a0): the user’s a0
csrw satp, a0 between two sfence.vma: install the user page table

8warm-upChoose one

Neither the TRAMPOLINE nor the TRAPFRAME mapping has PTE_U. What does that achieve?

kernel/proc.c

// map the trampoline code (for system call return)
// at the highest user virtual address.
// only the supervisor uses it, on the way
// to/from user space, so not PTE_U.
if (mappages(pagetable, TRAMPOLINE, PGSIZE, (uint64)trampoline,
             PTE_R | PTE_X) < 0) {
  uvmfree(pagetable, 0);
  return 0;
}
194
// map the trapframe page just below the trampoline page, for
// trampoline.S.
if (mappages(pagetable, TRAPFRAME, PGSIZE, (uint64)(p->trapframe),
             PTE_R | PTE_W) < 0) {
  uvmunmap(pagetable, TRAMPOLINE, 1, 0);
  uvmfree(pagetable, 0);
  return 0;
}

The pages are kept out of the TLB.User code cannot read, write or execute either page (any access faults), while the kernel, in supervisor mode, can execute the trampoline and use the trapframe.The kernel cannot access these pages; only user code can.The pages are shared between all processes.

9solidMatch the pairs

Match each load in the trampoline with what it fetches from the trapframe.

kernel/trampoline.S

75        # initialize kernel stack pointer, from p->trapframe->kernel_sp
76        ld sp, 8(a0)
77
78        # make tp hold the current hartid, from p->trapframe->kernel_hartid
79        ld tp, 32(a0)
80
81        # load the address of usertrap(), from p->trapframe->kernel_trap
82        ld t0, 16(a0)
83
84        # fetch the kernel page table address, from p->trapframe->kernel_satp.
85        ld t1, 0(a0)
86

ld sp, 8(a0) in uservecld tp, 32(a0) in uservecld t0, 16(a0) in uservecld t1, 0(a0) in uservecld sp, 48(a0) in userret

10solidClick the line

Line 76 puts the kernel-stack address into sp, but at that moment the address is not mapped. Click the line that installs the page table in which it maps the kernel stack.

kernel/trampoline.S

      csrr t0, sscratch
      sd t0, 112(a0)
74
      # initialize kernel stack pointer, from p->trapframe->kernel_sp
      ld sp, 8(a0)
77
      # make tp hold the current hartid, from p->trapframe->kernel_hartid
      ld tp, 32(a0)
80
      # load the address of usertrap(), from p->trapframe->kernel_trap
      ld t0, 16(a0)
83
      # fetch the kernel page table address, from p->trapframe->kernel_satp.
      ld t1, 0(a0)
86
      # wait for any previous memory operations to complete, so that
      # they use the user page table.
      sfence.vma zero, zero
90
      # install the kernel page table.
      csrw satp, t1
93
      # flush now-stale user entries from the TLB.
      sfence.vma zero, zero
96
      # call usertrap()
      jalr t0

Your pick: none yet (click a line in the code)

11solidFill in the machine state

A process has just made a system call. Hart 0 has executed line 76 of uservec, ld sp, 8(a0), and nothing after it. What is its state? (For stack, give the stack the hart could safely push to right now; see The stacks of xv6.)

kernel/trampoline.S

      csrr t0, sscratch
      sd t0, 112(a0)
74
      # initialize kernel stack pointer, from p->trapframe->kernel_sp
      ld sp, 8(a0)
77
      # make tp hold the current hartid, from p->trapframe->kernel_hartid
      ld tp, 32(a0)
80
      # load the address of usertrap(), from p->trapframe->kernel_trap
      ld t0, 16(a0)
83
      # fetch the kernel page table address, from p->trapframe->kernel_satp.
      ld t1, 0(a0)
86
      # wait for any previous memory operations to complete, so that
      # they use the user page table.
      sfence.vma zero, zero
90
      # install the kernel page table.
      csrw satp, t1
93
      # flush now-stale user entries from the TLB.
      sfence.vma zero, zero
96
      # call usertrap()
      jalr t0

Privilege modeActive stackPage table (satp)Interrupts (sstatus.SIE)

12solidDecode the bits

gdb shows sstatus = 0x8000000200006020 on hart 0 at the sret in userret. Decode the bits xv6 cares about.

Value: 0x8000000200006020

SIE (bit 1), nowSPIE (bit 5)SPP (bit 8): the mode sret returns toSIE right after sret

13solidDecode the bits

forkret computes MAKE_SATP(p->pagetable) for init and gets 0x8000000000087f52. Decode it.

Value: 0x8000000000087f52

MODE (bits 63-60)PPN (low 44 bits)Physical address of the root page tableDoes writing this value into satp also flush the TLB?

14solidChoose one

prepare_return calls intr_off() (line 108) before it points stvec at uservec (line 112). What would go wrong if an interrupt could arrive after line 112?

kernel/trap.c

100void
101prepare_return(void)
102{
103  struct proc *p = myproc();
104
105  // we're about to switch the destination of traps from
106  // kerneltrap() to usertrap(). because a trap from kernel
107  // code to usertrap would be a disaster, turn off interrupts.
108  intr_off();
109
110  // send syscalls, interrupts, and exceptions to uservec in trampoline.S
111  uint64 trampoline_uservec = TRAMPOLINE + (uservec - trampoline);
112  w_stvec(trampoline_uservec);

Writing stvec is only allowed while interrupts are off.Turning interrupts off is what makes sret start user mode with interrupts off.The interrupt would enter uservec while the kernel page table is still installed; its first store to TRAPFRAME would fault, and the fault would enter uservec again, forever.Another hart could start running this process while its trapframe is half-filled.

15solidChoose all that apply

Which trapframe fields does prepare_return write?

kernel/trap.c

100void
101prepare_return(void)
102{
103  struct proc *p = myproc();
104
105  // we're about to switch the destination of traps from
106  // kerneltrap() to usertrap(). because a trap from kernel
107  // code to usertrap would be a disaster, turn off interrupts.
108  intr_off();
109
110  // send syscalls, interrupts, and exceptions to uservec in trampoline.S
111  uint64 trampoline_uservec = TRAMPOLINE + (uservec - trampoline);
112  w_stvec(trampoline_uservec);
113
114  // set up trapframe values that uservec will need when
115  // the process next traps into the kernel.
116  p->trapframe->kernel_satp = r_satp();         // kernel page table
117  p->trapframe->kernel_sp = p->kstack + PGSIZE; // process's kernel stack
118  p->trapframe->kernel_trap = (uint64)usertrap;
119  p->trapframe->kernel_hartid = r_tp(); // hartid for cpuid()
120
121  // set up the registers that trampoline.S's sret will use
122  // to get to user space.
123
124  // set S Previous Privilege mode to User.
125  unsigned long x = r_sstatus();
126  x &= ~SSTATUS_SPP; // clear SPP to 0 for user mode
127  x |= SSTATUS_SPIE; // enable interrupts in user mode
128  w_sstatus(x);
129
130  // set S Exception Program Counter to the saved user pc.
131  w_sepc(p->trapframe->epc);
132}

16solidChoose all that apply

Which of these are sret’s own doing, done by the hardware when userret executes it?

kernel/trampoline.S

# restore user a0
      ld a0, 112(a0)
      
      # return to user mode and user pc.
      # prepare_return() sets up sstatus and sepc.
      sret

17deepChoose all that apply

Which of these are the same for every process in this kernel?

the value of trapframe->kernel_satpthe value of trapframe->kernel_spthe satp value that userret installsthe physical page mapped at TRAMPOLINEthe value of trapframe->kernel_trapthe physical page mapped at TRAPFRAME

18solidChoose one

Hart 0 (running cat) and hart 1 (running grep) both execute sd ra, 40(a0) in uservec at the same instant, both with a0 = 0x3fffffe000. Why does neither overwrite the other’s saved ra?

kernel/trampoline.S

      li a0, TRAPFRAME
      
      # save the user registers in TRAPFRAME
      sd ra, 40(a0)
      sd sp, 48(a0)

Only one hart can be in the trampoline at a time, because interrupts are off.sd is atomic, so the two stores cannot interfere.The trampoline takes a lock around the stores.Each hart translates 0x3fffffe000 through its own satp, which names its own process’s page table, so the two stores reach two different physical pages.

19solidChoose one

In this tree, what does sscratch hold while a process runs in user mode?

A pointer to the current process’s trapframe, set before each return to user mode.The ID of the hart.Nothing the kernel relies on: a stale value, the user a0 that the last trap on this hart parked there.The top of the process’s kernel stack.

20solidChoose one

usertrap computes the user satp at the very end and hands it to userret in a0. Why not reuse the satp value that was installed when the trap arrived?

kernel/trap.c

84  // give up the CPU if this is a timer interrupt.
85  if (which_dev == 2)
86    yield();
87
88  prepare_return();
89
90  // the user page table to switch to, for trampoline.S
91  uint64 satp = MAKE_SATP(p->pagetable);
92
93  // return to trampoline.S; satp value in a0.
94  return satp;
95}

The system call may have replaced the page table: after a successful exec, p->pagetable is a new table and the old one has been freed.So that userret can return to a different process than the one that trapped.uservec saved the user satp in the trapframe, but prepare_return overwrites that field.satp belongs to the hart, and the process may have moved to another hart.

21deepType a number

A process lives in proc[2]. What value does prepare_return store in its trapframe->kernel_sp? (Answer in hex.)

kernel/memlayout.h

46// map the trampoline page to the highest address,
47// in both user and kernel space.
48#define TRAMPOLINE (MAXVA - PGSIZE)
49
50// map kernel stacks beneath the trampoline,
51// each surrounded by invalid guard pages.
52#define KSTACK(p) (TRAMPOLINE - ((p) + 1) * 2 * PGSIZE)

decimal, 0x hex or 0b binary

22deepChoose one

uservec reaches usertrap with ld t0, 16(a0) … jalr t0 instead of simply call usertrap. Why?

kernel/trampoline.S

      # initialize kernel stack pointer, from p->trapframe->kernel_sp
      ld sp, 8(a0)
77
      # make tp hold the current hartid, from p->trapframe->kernel_hartid
      ld tp, 32(a0)
80
      # load the address of usertrap(), from p->trapframe->kernel_trap
      ld t0, 16(a0)
83
      # fetch the kernel page table address, from p->trapframe->kernel_satp.
      ld t1, 0(a0)
86
      # wait for any previous memory operations to complete, so that
      # they use the user page table.
      sfence.vma zero, zero
90
      # install the kernel page table.
      csrw satp, t1
93
      # flush now-stale user entries from the TLB.
      sfence.vma zero, zero
96
      # call usertrap()
      jalr t0

usertrap is not mapped in the user page table.call is PC-relative: the linker computed the offset for code at 0x80006000, but uservec runs at 0x3ffffff000, so the jump would land about 254 GiB away from usertrap.jalr t0 lets usertrap return to uservec’s next line, which call would not.call pushes the return address on the stack, and there is no stack yet.

23deepChoose one

Imagine a bug that leaves the TRAPFRAME mapping out of a process’s page table. What happens the first time that process traps into the kernel?

kernel/trampoline.S

      # save user a0 in sscratch so
      # a0 can be used to get at TRAPFRAME.
      csrw sscratch, a0
33
      # each process has a separate p->trapframe memory area,
      # but it's mapped to the same virtual address
      # (TRAPFRAME) in every process's user page table.
      li a0, TRAPFRAME
      
      # save the user registers in TRAPFRAME
      sd ra, 40(a0)
      sd sp, 48(a0)
      sd gp, 56(a0)

vmfault maps a fresh page there, and the trap continues.usertrap sees a store page fault and kills the process.The kernel panics with kerneltrap.The first sd through TRAPFRAME takes a store page fault; stvec still points at uservec, so the fault enters uservec again and faults on the same store, forever: the hart hangs without a message.

24deepTrue or false, and why

True or false: in this kernel, the twelve loads ld s0 … ld s11 in userret are redundant on every return to user mode, because the s registers already hold the user’s values when usertrap returns.

kernel/trampoline.S

      # restore all but a0 from TRAPFRAME
      ld ra, 40(a0)
      ld sp, 48(a0)
      ld gp, 56(a0)
      ld tp, 64(a0)
      ld t0, 72(a0)
      ld t1, 80(a0)
      ld t2, 88(a0)
      ld s0, 96(a0)
      ld s1, 104(a0)
      ld a1, 120(a0)
      ld a2, 128(a0)
      ld a3, 136(a0)
      ld a4, 144(a0)
      ld a5, 152(a0)
      ld a6, 160(a0)
      ld a7, 168(a0)
      ld s2, 176(a0)
      ld s3, 184(a0)
      ld s4, 192(a0)
      ld s5, 200(a0)
      ld s6, 208(a0)
      ld s7, 216(a0)
      ld s8, 224(a0)
      ld s9, 232(a0)
      ld s10, 240(a0)
      ld s11, 248(a0)
      ld t3, 256(a0)
      ld t4, 264(a0)
      ld t5, 272(a0)
      ld t6, 280(a0)

TrueFalse

Why?

sret clears the s registers, so they must be reloaded.usertrap and the functions it calls always destroy the s registers.A new process’s first return goes through forkret, whose kernel thread never came in through uservec; its s registers hold kernel values, and the trapframe copy is the only place its user s registers exist.

25deepChoose one

uservec sets tp from trapframe->kernel_hartid, a value written by prepare_return the last time this process left the kernel. Why is it guaranteed to be the ID of the hart now executing uservec?

kernel/trap.c

// set up trapframe values that uservec will need when
// the process next traps into the kernel.
p->trapframe->kernel_satp = r_satp();         // kernel page table
p->trapframe->kernel_sp = p->kstack + PGSIZE; // process's kernel stack
p->trapframe->kernel_trap = (uint64)usertrap;
p->trapframe->kernel_hartid = r_tp(); // hartid for cpuid()
120

A process in user mode never changes hart: only the kernel’s scheduler moves processes, so the hart that ran prepare_return is the hart on which the process traps next.kernel_hartid is set once by kfork and never changes.Each process is pinned to one hart for its whole life.uservec checks the value against mhartid and corrects it.