Step 1 of 20

cat passes a pointer to its own memory

cat has argc = 2, and on line 35 it calls open(argv[1], O_RDONLY). Where does argv[1] point? When the shell ran cat, kexec built a fresh user stack and copied the argument strings onto it (kernel/exec.c:101). In this build, cat’s memory looks like this (p->sz is 0x4000, four pages):

So the first argument of open is the number 0x3fe0, and the second is O_RDONLY, which is 0 (kernel/fcntl.h).

Notice that the string has already crossed the user/kernel boundary twice: the shell’s exec copied it into the kernel, and kexec copied it back out to cat’s new stack. Now it must come in again. User data never stays in the kernel by accident; it is copied in on purpose, each time it is needed.

Step 2 of 20

The stub adds the number 15

open is a system call stub like the write stub of Tour 5: Life of a system call. By the calling convention the compiler already put the arguments in registers: a0 = 0x3fe0, a1 = 0. In cat.asm, the call site is li a1,0, ld a0,0(s2) (load argv[1] from address 0x3fc8, held in s2), jal 3ec <open>.

The stub sets a7 = 15 (SYS_open) at address 0x3ec and executes ecall at 0x3ee. From here the path into the kernel is exactly the one Tour 5: Life of a system call follows: uservec saves all 31 registers into cat’s trapframe, switches to the kernel page table, and usertrap calls syscall, which calls syscalls[15], sys_open.

When sys_open starts, gdb shows cat’s trapframe with a0 = 0x3fe0, a1 = 0, a7 = 0xf, and epc = 0x3f2: the ecall address plus 4, already advanced by usertrap.

Those saved registers are everything the kernel knows about the request. Two of them are plain values. One of them is a promise (“there is a string at 0x3fe0”) that the kernel will verify the hard way.

Step 3 of 20

sys_open makes room for a private copy

sys_open declares char path[MAXPATH]: 128 bytes (kernel/param.h) on cat’s kernel stack. In this run that stack is the page at 0x3fffff9000–0x3fffffa000 in the kernel’s address space, and path sits at 0x3fffff9f10, 240 bytes below the top.

That stack was empty when the ecall arrived: uservec's ld sp, 8(a0) (kernel/trampoline.S:76) pointed sp at its top, 0x3fffffa000, and the frames of usertrap, syscall and sys_open (with path inside it) are everything on it. cat’s user stack, with "README" on it, is a different page in a different page table (The stacks of xv6).

The kernel stack is a single 4096-byte page with an unmapped guard page below it. A path buffer of a few kilobytes here would risk running off the stack, which is why MAXPATH is small, and why sys_exec, which needs room for long argument strings, uses whole pages from kalloc instead.

The order of the work matters. First the arguments are fetched (lines 337–338). Only if that succeeds does sys_open call begin_op and start touching the file system. A bad pointer costs one failed copy and a return -1, with no transaction to undo, no lock to release, no inode to put back.

Step 4 of 20

argraw reads the saved registers, not the live ones

Every argument helper ends up in argraw, which returns p->trapframe->a0 … a5. Why the trapframe and not the registers themselves? Because the registers have long since been reused: uservec overwrote a0 (with TRAPFRAME), sp, tp, t0 and t1, and every C function since then has used a0–a5 for its own arguments. The trapframe is the only place where cat’s values survive.

It is also the only place that stays correct if cat is preempted. Interrupts are on (usertrap enabled them), so a timer interrupt may switch cat out right here and resume it later on hart 1 or 2 (Tour 8: Traps taken inside the kernel). The trapframe moves with the process, so argraw returns the same numbers on any hart.

n is chosen by kernel code (a constant like 0 or 1 in sys_open), never by the user, so an out-of-range n is a kernel bug and panics. The values are a different matter: they are whatever cat put in its registers. argraw does not judge them. Judging is the job of whoever knows what the value is supposed to mean.

Step 5 of 20

argint and argaddr check nothing, on purpose

argint(1, &omode) stores the low 32 bits of the saved a1: omode = 0 (gdb agrees). argaddr stores a full 64-bit value. Neither returns an error, and neither can: any integer is a legal integer, and any 64-bit number is a legal address. Whether it is a good address depends on what is mapped there, and on when you look.

That is the point of the comment on lines 63–65. A check here, such as “is addr below p->sz?”, would be checked at one moment and relied on at another. The copy routines check every page at the moment they touch it, which is the only check that means anything.

The integer arguments are validated by their users instead. omode is only tested with & and compared with O_RDONLY, so any value is harmless. A file descriptor goes through argfd, which rejects anything outside 0..NOFILE-1 before indexing ofile[]. A system-call number is range-checked in syscall before it indexes the table. An unchecked integer used as an array index would let a user program make the kernel read or call anything; xv6 checks each one right where it is used.

Step 6 of 20

argstr hands the pointer to fetchstr

argstr(0, path, MAXPATH) reads addr = 0x3fe0 with argaddr and calls fetchstr(0x3fe0, path, 128). fetchstr passes the job to copyinstr, along with two things only the kernel can supply: cat’s page table (p->pagetable, physical page 0x87f24000 in this run) and its size (p->sz, 0x4000).

Compare fetchaddr above it, which reads one 8-byte value (used by exec to read the argv array). It checks the range itself, with two tests: addr >= p->sz and addr + 8 > p->sz. The second alone would be fooled by addr = 0xfffffffffffffffc, where addr + 8 wraps around to 4. User numbers are adversarial; even the arithmetic on them must be checked for overflow.

fetchstr returns strlen(buf). That call is safe only because copyinstr promises that, on success, buf contains a terminating NUL within max bytes. Without that promise, strlen would run off the end of path into the rest of the kernel stack.

Step 7 of 20

copyinstr starts at the page containing the string

The hart is running with the kernel page table. In that table, 0x3fe0 means nothing useful (it is unmapped), and other user addresses would be worse: user address 0x80000000 is, to the kernel, the start of its own code. So copyinstr never dereferences srcva. It translates it through cat’s page table, in software:

1. va0 = PGROUNDDOWN(0x3fe0) = 0x3000, the page holding the string.
2. walkaddr(pagetable, 0x3000) returns the physical address of that page, or 0.
3. If 0, vmfault gets one chance to produce the page (for lazily allocated memory, step 17). If that fails too, the whole copy fails with -1.

The loop runs once per page because consecutive virtual pages need not be consecutive in physical memory. A string that crosses a page boundary must be translated again at the boundary, and the second page may be missing even when the first was fine (step 16).

Step 8 of 20

walkaddr refuses anything that is not user memory

walkaddr is three refusals and a lookup:

• va >= MAXVA → 0. This test is there for safety, not convenience: walk panics on such an address, and a user program must never be able to choose an argument that panics the kernel.
• No PTE (page-table entry), or PTE_V clear → 0: nothing is mapped there.
• PTE_U clear → 0. This is the subtle one. cat’s page table does map pages the program may not touch: the trapframe at TRAPFRAME, the trampoline page, and the stack guard page at 0x2000. The hardware blocks user-mode access to them because PTE_U is clear, but a kernel copying on the user’s behalf must apply the same rule itself. Otherwise read(fd, (char *)TRAPFRAME, n) would let cat have the kernel overwrite kernel_satp, kernel_sp and kernel_trap with bytes of its choosing, and the next trap would jump wherever cat liked, in supervisor mode. (copyout uses the same walkaddr.)

For 0x3000, gdb shows the final PTE is 0x21fc70d7. The low bits 0xd7 are V R W U A D: valid, readable, writable, user, and the accessed and dirty bits that QEMU’s MMU set when cat used its stack (xv6 never sets them; the RISC-V spec also allows hardware that faults instead and leaves this to software). The physical page is 0x87f1c000. The guard page’s PTE is 0x21fc7407: V R W, no U.

Step 9 of 20

walk reads cat's page table through the direct map

walk does what the MMU would do with Sv39, three levels of 512-entry tables. For va = 0x3000 the three 9-bit indexes are 0, 0 and 3:

(Values from this run.) A PTE holds a physical address, and the kernel is running with paging on. How can it follow one? Because the kernel page table maps all of RAM at virtual address = physical address, the direct map. PTE2PA(*pte) is directly usable as a pointer. The same trick lets copyinstr read the string itself at 0x87f1c000 + 0xfe0.

alloc is 0: looking up a user address must never build page-table pages.

Step 10 of 20

The copy, byte by byte, until the NUL

n = PGSIZE - (0x3fe0 - 0x3000) = 32: at most 32 bytes remain on this page, fewer than max = 128. The inner loop reads from p = 0x87f1cfe0 and copies R, E, A, D, M, E, then finds \0, writes it to dst, sets got_null, and copyinstr returns 0.

Seven bytes were written into path. In this run gdb shows the remaining 121 bytes still holding leftovers from earlier function calls on the kernel stack. That is fine: everything after the NUL is ignored. But it shows why copyinstr must guarantee the NUL.

The two ways out without a NUL both return -1:

• max reaches 0: the string is longer than the caller’s buffer (step 15).
• A page is missing: the string runs into memory the process does not own (step 16).

Back in fetchstr, strlen(path) is 6; argstr returns 6, and sys_open stores it in n. From now on sys_open uses only path, its own copy, at 0x3fffff9f10. It will never read 0x3fe0 again.

Step 11 of 20

From here on, only the kernel's copy is used

With path = "README" safely on the kernel stack, sys_open starts a file-system transaction (begin_op, Tour 31: The log: begin_op, commit and group commit) and calls namei(path) to find the inode (Tour 34: Path lookup). Path lookup will read directory blocks, take sleep-locks, and maybe sleep waiting for the disk. All of it works on bytes the user can no longer touch.

This is the defense against a classic bug called time-of-check to time-of-use (TOCTOU). Imagine a kernel that validated the user’s string (“does it name a file this user may open?”) and then, later, read the name again from user memory to use it. In an operating system with multithreaded processes or shared memory, a second thread could rewrite the string in between. xv6 processes are single-threaded and share no memory, so nothing but cat itself can change cat’s memory, and cat is stopped in this system call. The attack cannot happen in xv6 today. But “copy once, then use only the copy” makes the question irrelevant, which is exactly the property you want: the kernel’s reasoning about a value holds as long as the value lives in the kernel.

Step 12 of 20

exec copies a two-level structure, one level at a time

Step back in time to the exec("cat", argv) that created cat in the first place. The shell forked a child (pid 3), and that child, still running the shell’s code, made this call. sys_exec faces a harder version of the same problem: argv is a user pointer to an array of user pointers to strings. Every level is untrusted.

• argstr copies the program name into path[MAXPATH], as in open.
• The loop reads argv[i] with fetchaddr, one 8-byte pointer at a time, into the kernel variable uarg. That pointer is now a kernel copy too.
• Each string is copied with fetchstr into a whole page from kalloc, with a limit of PGSIZE (4096) bytes, too much for the kernel stack.
• At most MAXARG - 1 (31) argument strings plus the terminating 0: an array with no 0 in its first 32 slots ends in goto bad, not in an endless loop.

Every failure path frees every page already allocated. A hostile program can make exec fail, but cannot make it leak memory: usertests checks this by counting free pages before and after its tests (countfree).

kexec then receives only kernel copies. It never sees a user pointer.

Step 13 of 20

usertests fires five ridiculous pointers

Now the hostile cases. Run usertests copyinstr1 right after boot: the shell (pid 2) runs usertests (pid 3), whose run forks a child (pid 4) to run the test, so that a test that crashes kills only that child. Say the child runs on hart 1. The later usertests steps assume each test is run the same way, right after a fresh boot, so its child is again pid 4.

copyinstr1 passes five addresses as the path to open. Each is a deliberate trap for a careless kernel:

The test passes if every open returns -1, and if the kernel is still alive afterwards. In this build it prints test copyinstr1: OK.

Step 14 of 20

Bad pointers die in walkaddr and vmfault

Follow each address through copyinstr. walkaddr returns 0 for all five: 0x80000000 has no PTE in the child’s table, TRAPFRAME and TRAMPOLINE lack PTE_U, and the last two are at or above MAXVA (and the check catches them before walk could panic). Then copyinstr gives vmfault its chance, and vmfault refuses at line 463: every one of them is at or above p->sz. copyinstr returns -1, sys_open returns -1, and open returns -1 in the child.

Now imagine the kernel had simply dereferenced the pointer, with its own page table loaded:

• 0x80000000 would have worked, copying the kernel’s own instructions into path. A user program could then probe kernel memory by opening files.
• 0x3fffffe000 is a guard page in the kernel’s table (below the trampoline, above the first kernel stack), so the load would fault in supervisor mode. That ends in panic: kerneltrap (Tour 10: Exceptions and faults): hart 1 stops for good, the console freezes, and the whole machine has to be restarted. One bad open would take down the entire system.

Line 466 matters for the guard page at 0x2000 in cat: it is below p->sz, so the size test passes, but it is already mapped (without PTE_U), so ismapped makes vmfault refuse rather than hand it out as a fresh page.

Step 15 of 20

A string exactly as long as the buffer

copyinstr2 builds a string of exactly MAXPATH (128) x characters, with its NUL at index 128, one byte past what the kernel’s path buffer can hold. This is the classic off-by-one.

In copyinstr, max starts at 128. The inner loop copies 128 xs into path[0..127], and max reaches 0 before the NUL is seen. got_null is still 0, so copyinstr returns -1. It did not write a NUL at path[128]: that byte belongs to whatever lies above the buffer on the kernel stack. A routine that “helpfully” terminated the string there would corrupt the kernel stack, under the control of a user program.

unlink, open, link and exec all go through argstr with MAXPATH, so all four refuse. The test then forks a child of its own, which tries exec("echo", {big, big, big, 0}) with big a 4096-character string: sys_exec gives fetchstr a PGSIZE limit, so the same rule rejects it one level up, and that child exits with 747 to prove exec returned.

In a correct kernel, “too long” and “bad pointer” look the same to the caller: -1.

Step 16 of 20

A string that runs off the end of memory

copyinstr3 grows the heap until its top top is page-aligned, then stores a single x in the very last byte it owns, top - 1, and passes that address as a path. The string has a valid first byte and no end: the next byte is at top, which is p->sz.

In copyinstr: the first iteration translates the last page, n = 1, copies the x, and leaves the inner loop without a NUL. srcva moves to top. The second iteration’s walkaddr returns 0, and vmfault refuses because top >= p->sz. Return -1.

This is why the translation sits inside the loop over pages. A routine that checked only the first byte’s page, then copied until a NUL, would walk off the end of the process’s memory, into whatever physical page happened to follow in the direct map: another process’s data, or a kernel page table.

Note what copyinstr left behind: an x in path[0] and no terminator. Harmless, because the -1 means nobody will read path. Partial results of a failed copy are never used.

Step 17 of 20

A string on a page that does not exist yet

Now a pointer that is legitimate but has no page behind it. lazy_copyinstr aligns the heap, then calls sbrklazy(2 * PGSIZE): sys_sbrk raises p->sz by two pages without allocating anything (Tour 26: sbrk, eager and lazy, and page faults).

p[4095] = '/' is a user-mode store to the first lazy page. It page-faults, and usertrap calls vmfault to allocate a zeroed page (Tour 10: Exceptions and faults); the store then succeeds. The second page is never touched by the program.

open(&p[4095], O_RDONLY) passes a one-character string whose NUL lives at p[4096], on that untouched page. In copyinstr: the first page translates and gives /. On the second page, walkaddr returns 0, but this time vmfault agrees: the address is below p->sz and not mapped. It allocates a page, fills it with zeros, maps it, and returns its physical address. The first byte is \0. The copy succeeds, path = "/", and open returns a descriptor for the root directory.

So the kernel treats a lazy page the same way whether the program or the kernel touches it first. A system call must not fail just because the program had not yet used memory it owns.

Step 18 of 20

Allocating that page takes the one shared lock

vmfault calls kalloc. Everything so far in this tour touched only one process’s private state. The free-page list is different: it is shared by all harts. So kalloc takes kmem.lock, a spinlock, pops one page, and releases it.

For the three lines inside the lock, interrupts are off on hart 1: acquire called push_off. If a timer interrupt could arrive here and this thread were switched out while holding kmem.lock, every other hart that needed memory would spin until it was switched back in (interrupts and spinlocks (push_off / pop_off)). push_off found them on (this is a system call) and recorded intena = 1, which is why release turns them back on (Locks and interrupt state).

The memset(…, 5, …) junk fill happens after the release: no reason to make other harts wait while 4096 bytes are written. vmfault then overwrites the junk with zeros, because a user page must never leak a previous owner’s data, and mappages installs it with PTE_R | PTE_W | PTE_U. Then copyinstr continues with interrupts back on.

Step 19 of 20

copyin with a spinlock held, for a pipe

copyin in usertests tests the other direction of trust: write with a bad buffer pointer, such as write(fds[1], (char *)0x80000000, 8192) on a pipe. sys_write takes the pointer with argaddr, unchecked, and filewrite hands it to pipewrite.

pipewrite copies one byte at a time with copyin, while holding pi->lock, a spinlock, so interrupts are off on hart 1. copyin follows the same pattern as copyinstr: walkaddr fails, vmfault refuses (0x80000000 is above p->sz), return -1. Since no byte was written yet (i == 0), pipewrite returns -1.

Calling copyin with a spinlock held is allowed only because copyin never sleeps. At worst it calls kalloc, which takes kmem.lock briefly and returns. That nests kmem.lock inside pi->lock: noff goes to 2, and releasing kmem.lock only drops it back to 1, leaving interrupts off (Locks and interrupt state). If copying user memory could wait for a disk (as it can in systems that page to disk), this code would deadlock or would have to be rewritten.

For a regular file, the same bad pointer reaches copyin inside writei, holding the inode’s and the buffer’s sleep-locks, with interrupts on. writei copies 0 bytes, and filewrite returns -1. For the console, consolewrite returns 0 bytes written. The test demands -1 for the regular file and accepts -1 or 0 for the console and the pipe: anything but a successful write.

Step 20 of 20

open returns a descriptor

Back to cat on hart 0 (or wherever it was last scheduled). "README" named a regular file, so sys_open fills in a struct file and returns descriptor 3: cat inherited 0, 1 and 2 from the shell. syscall stores 3 in p->trapframe->a0, and open returns 3 in user mode (Tour 5: Life of a system call).

What did one path argument cost? Two saved registers read from the trapframe; one three-level page-table walk; seven bytes copied through the direct map; one strlen. A few hundred instructions at most, far less than the trap itself.

And what did it buy?

• Translate, don’t dereference. A user address means something only in the user’s page table. walkaddr applies the user’s rules (PTE_V, PTE_U, MAXVA, p->sz) in software.
• Check at the moment of use, page by page. argaddr checks nothing; copyinstr checks each page as it reads it.
• Bound everything. MAXPATH, PGSIZE, MAXARG; no NUL in time means -1.
• Copy once. After argstr, the kernel works only on path.
• Fail with -1, never with a panic. Every hostile pointer in usertests ends as an ordinary error return, and the other harts never notice.