Tour 14 · Time and scheduling · about 26 minutes · 18 steps
xv6 has exactly one notion of time: ticks, a counter that goes up by one about ten times
a second. Only hart 0 increments it. Every process that wants to wait for time to pass,
through the pause system call, sleeps on the address of that counter and is woken at
every tick to check whether it has waited long enough.
This tour follows zombie (pid 3), which calls pause(5) on hart 2 to give its child a
head start. You will see sys_pause read the clock under tickslock, register
on the channel &ticks and go to sleep, while hart 0, in clockintr, does
ticks++ and wakeup(&ticks), possibly at the very instant zombie is about to sleep.
You will see why that race is harmless, why zombie wakes up five times, why it may
finish its pause on a different hart, and why a kill can cut the wait short.
It is a small system call, and a complete lesson in the sleep and wakeup pattern, Tour 16: sleep and wakeup, and the lost-wakeup problem, applied to a device every hart has: the clock.
Best after: 5. Life of a system call, 11. From a timer tick to a context switch, 16. sleep and wakeup, and the lost-wakeup problem
The machine has three harts. As the first command after boot, you typed
zombie at the shell (pid 2, now asleep in wait). The shell’s child, pid 3, ran
zombie, which has just forked pid 4.
When the tour starts:
| Hart | What it is doing |
|---|---|
| 0 | Idle in its scheduler: but it is the hart that counts ticks |
| 1 | Running pid 4, the child, which is calling exit(0) |
| 2 | Running zombie (pid 3) in user mode: the process this tour follows |
Step 1 of 18
fork() returned the child’s pid (4) to the parent, so the parent calls pause(5):
“do nothing for 5 ticks”. The comment says why: the child should exit first. The child
then becomes a zombie (an exited process its parent has not yet waited for),
which is what this test program exists to create; when the parent exits too, the
child is handed to init (Tour 21: exit, wait and zombies).
Five ticks is about half a second. zombie could have spun in a loop checking the
time instead, but that would waste a CPU for half a second. pause lets hart 2 run
something else, or rest, while zombie waits.
(In older xv6 this system call was called sleep. A 2025 commit renamed it pause,
in the same change that renamed the kernel’s fork, exec, wait, exit and kill
to kfork and friends; in this kernel, sleep is only the internal
primitive.)
Step 2 of 18
The system call stub puts SYS_pause (13) in a7 and executes ecall. The
argument, 5, is already in a0.
From here to sys_pause the path is exactly the one Tour 5: Life of a system call walks through:
the trampoline saves zombie’s registers in its trapframe, usertrap sees
scause == 8, adds 4 to the saved pc, turns interrupts on, and syscall
calls syscalls[13].
usertrap → syscall → sys_pauseld sp, 8(a0) in uservec (kernel/trampoline.S:76)Step 3 of 18
argint reads n = 5 from the saved a0.
zombie is running on its own kernel stack now. On the way in, uservec
saved the user sp in the trapframe and loaded kernel_sp, the top of zombie’s
kernel stack, which is always empty while a process is in user mode
(The stacks of xv6).
The clamp on lines 74–75 looks like politeness, but it prevents a near-infinite wait.
The loop below compares ticks - ticks0 < n, where ticks and ticks0 are uint
and n is int. C converts n to unsigned int for that comparison, so n = -1
would become 4,294,967,295: about 13.6 years of ticks. Turning a negative n into 0
makes pause(-1) return immediately instead.
Interrupts are on: this is an ordinary system call, preemptible by the timer until it takes a lock.
Step 4 of 18
ticks is a plain uint in kernel/trap.c (at 0x80007880 in this build), and
tickslock is the spinlock that guards it, named "time" by trapinit.
Who touches ticks?
| Code | Access | Hart |
|---|---|---|
clockintr |
ticks++, then wakeup(&ticks) |
0 only |
sys_pause |
reads it, sleeps on &ticks |
any |
sys_uptime |
reads it | any |
All three hold tickslock. The address &ticks doubles as the
wait channel: a number that the sleeper and the waker agree on.
Any address would do; using the counter’s own address makes the code self-explanatory.
tickslockStep 5 of 18
acquire turns interrupts off on hart 2 and records cpus[2].intena = 1 (they were
on; Locks and interrupt state). Then ticks0 = ticks. Say the clock reads 1000.
Why is a lock needed to read one 32-bit number? Not for the read itself. The lock
matters because of what comes next: sys_pause will compare ticks with ticks0,
and if the wait is not over, register as a sleeper on &ticks. Holding tickslock
from the comparison until the registration means hart 0 cannot slip a ticks++ and
its wakeup in between, unseen.
tickslockStep 6 of 18
ticks - ticks0 is 0, less than 5: keep waiting.
Every time around the loop, before sleeping, sys_pause asks killed: has someone
killed this process? If so, it gives up and returns -1, and zombie will exit on its
way out of the kernel. Without this check, kill on a process in pause(1000) would
take 100 seconds to work. Tour 23: kill follows kill.
killed acquires zombie’s p->lock while tickslock is held, then releases it.
So the order here is tickslock, then p->lock. clockintr on hart 0 uses the same
order: tickslock, then each p->lock inside wakeup. Two paths that take the same
two locks in the same order cannot deadlock against each other (Tour 18: Lock ordering: how xv6 avoids deadlock).
Subtraction rather than ticks < ticks0 + n is deliberate: with unsigned arithmetic,
ticks - ticks0 stays correct even if ticks wraps around from 4,294,967,295 to 0.
tickslockzombie's p->lockStep 7 of 18
sys_pause calls sleep_prepare(&ticks). sleep_prepare takes zombie’s p->lock
and sets p->chan = &ticks. From this moment zombie is registered: any
wakeup(&ticks) will find it and clear p->chan, even though zombie is still
running.
Two locks are held at once here, tickslock and zombie’s p->lock, in the same
order as before. Both are spinlocks, so noff on hart 2 is 2, and interrupts stay off (Locks and interrupt state).
The registration is done before releasing tickslock. That is the order the
sleep and wakeup pattern requires: decide to wait (under the condition’s lock),
register, then let go of the condition’s lock. Any tick that happens after the
decision now happens after the registration too.
Step 8 of 18
release gives up tickslock. noff returns to 0 and intena is 1, so interrupts
come back on on hart 2.
Now hart 0 is free to tick. zombie is registered but still running; it is about to
call sleep(). A tick can arrive at any point from here on, and the code must be
right whichever way it falls. See the timeline below.
zombie's p->lockStep 9 of 18
In the usual case no tick has come yet. sleep takes zombie’s p->lock
(recording cpus[2].intena = 1), finds p->chan still &ticks, marks zombie
SLEEPING and calls sched.
sched saves intena, and swtch moves hart 2 to its scheduler, which releases
zombie’s p->lock. Tour 13: swtch and the lock handed across a context switch follows this switch instruction by instruction. The
ld sp, 8(a1) in swtch (kernel/swtch.S:26) is where hart 2 leaves zombie’s
kernel stack for its own scheduler stack, the slice of stack0 it booted on.
zombie is now frozen inside sched, its kernel stack holding the frames of
usertrap, syscall, sys_pause, sleep and sched.
Hart 2’s scheduler looks for other work. Pid 4 is exiting on hart 1 and nothing else
is runnable, so hart 2 soon stalls in wfi.
stack0with a 256-byte kernelvec frame on toptickslockStep 10 of 18
0.1 s after its previous tick, hart 0’s time passes its stimecmp. Hart 0 was idle:
its wfi returns, its scheduler opens the interrupt window, and the trap goes through
kernelvec and kerneltrap to devintr and clockintr (Tour 12: One scheduler per hart shows
that window; Tour 11: From a timer tick to a context switch shows a tick arriving from user mode instead).
Which stack is this on? Whatever stack hart 0 was using when the interrupt hit: its
scheduler stack (its slice of stack0). kernelvec pushed its 256-byte register
frame there (kernel/kernelvec.S:14), and kerneltrap, devintr, clockintr and
soon wakeup stack up below it:
hart 0's scheduler stack (stack0 slice, top 0x80008890)
start 16 bytes, never popped (start left by mret)
main
scheduler
kernelvec (256-byte register frame)
kerneltrap
devintr
clockintr ◄─ sp
cpuid() is 0, so this hart counts. It takes tickslock, and ticks becomes 1001.
Then, still holding tickslock, it calls wakeup(&ticks).
This acquire of the same tickslock records intena = 0, not 1 as in step 5:
the trap had already turned interrupts off. And the tick could be taken at all
only because hart 0 held no spinlock at that moment (Locks and interrupt state).
Why wake while holding the lock? It is not required for correctness here, because a
sleeper’s check and its registration both happen under tickslock: any sleeper that
saw 1000 registered before this ticks++, so even a wakeup after the release would
still find it. But holding the lock keeps the increment and the wakeup together: no
sleeper can observe 1001 under tickslock without the wakeup for 1001 having already
been delivered.
stack0with a kernelvec frame on toptickslockeach p->lock in turnStep 11 of 18
wakeup walks all 64 slots, taking and releasing each p->lock. At zombie’s slot
it finds chan == &ticks and SLEEPING: it clears chan and sets RUNNABLE.
This happens on every tick, ten times a second, whether anyone is pausing or not:
64 lock acquisitions while holding tickslock. And it wakes every process sleeping
on &ticks, including ones that asked for 1000 ticks and have 990 to go. Each of them
will run, re-check, and go back to sleep. A real kernel keeps sleepers in a list
sorted by wake-up time; xv6 accepts this waste to keep pause to a dozen lines.
stack0with a kernelvec frame on topStep 12 of 18
clockintr releases tickslock and sets hart 0’s next deadline, 1,000,000 time units
(0.1 s) from now. kerneltrap does not yield, because no process is running on hart
0 (myproc() is 0: the trap landed on a scheduler stack, not a process’s kernel
stack). kernelvec pops its frame and returns into the scheduler loop, which closes
the interrupt window and scans.
It finds zombie RUNNABLE and switches to it. zombie went to sleep on hart 2 and
will wake up on hart 0. Hart 2, stalled in wfi, will not even hear about it
until its own next timer or device interrupt: xv6 sends no interrupt from one hart to another, so whichever
scheduler scans first wins.
zombie’s kernel stack, exactly as it was left on hart 2ld sp, 8(a1) in swtch (kernel/swtch.S:26), called by hart 0’s schedulerStep 13 of 18
(The state shown is just after line 571, as in steps 8 and 17.) sched returns on
hart 0 and restores cpus[0].intena = 1, zombie’s value from before it slept.
sleep releases p->lock, the lock hart 0’s scheduler acquired, and
interrupts come back on: zombie continues its system call on hart 0 exactly as it was
running on hart 2. Its kernel stack never moved; hart 0’s swtch loaded the sp
that hart 2’s swtch had saved, and the frames usertrap → syscall → sys_pause
→ sleep were waiting there.
Notice what sleep does not do: check the time. It does not know what zombie
was waiting for. It only knows that someone called wakeup on its channel (or that it
was killed, Tour 23: kill). Deciding whether the wait is over is the caller’s job.
tickslockStep 14 of 18
Back in sys_pause, zombie retakes tickslock and goes around the loop:
ticks - ticks0 is 1001 − 1000 = 1, still less than 5. Killed? No. Register, release,
sleep.
This repeats on the next four ticks. Each time, zombie wakes on whichever hart’s
scheduler finds it first, which may be a different hart each time. Over the whole
pause(5), zombie usually sleeps five times and tests the loop condition six times (an
extra wakeup, such as a kill, would change the count).
The loop is why sleep may return “early” without harm. A wakeup means only “the thing
you are waiting on may have changed; look again”, and the while re-tests the real
condition under the real lock. Every sleep loop in xv6 is written this way.
stack0with a kernelvec frame on topStep 15 of 18
All three harts get a timer interrupt every 0.1 s, but only one of them may count, or
ticks would run three times too fast. Hart 0 is the obvious choice: it is the hart
that always exists.
Consequences of letting one hart keep time:
clockintr schedules the next tick 0.1 s after the handling
of this one, so every delay pushes all later ticks back. Over time, ticks falls
slightly behind true time. pause(5) means “five ticks”, not “exactly 0.5 s”.usertrap's call to devintr. Only the stack differs: the scheduler stack
here, and the running process’s kernel stack when hart 0 is running a process.tickslockStep 16 of 18
Suppose that, after the second tick, some other process (a background job, say) calls
kill(3). kkill sets zombie’s killed flag and,
because zombie is SLEEPING, sets it RUNNABLE directly. No tick needed.
Hart 1’s scheduler happens to pick zombie up. It returns from sleep, retakes
tickslock, finds ticks - ticks0 = 2, less than 5, and enters the loop body. Line
79: killed. It releases tickslock and returns -1. usertrap then sees the flag
and calls kexit(-1) (kernel/trap.c:81).
The check is at the top of the loop body for a reason: every way out of sleep leads
back to it, so a kill is noticed after at most one more wakeup. (If the kill happens
to arrive between the killed check and sleep(), zombie sleeps anyway, and
notices at the next tick, 0.1 s later. Tour 23: kill looks at that window.)
Step 17 of 18
Without a kill, the story ends at the fifth tick. zombie, now running on hart 2,
reads 1005 − 1000 = 5. The loop ends; it releases tickslock (interrupts on again)
and returns 0, which syscall puts in the saved a0.
How long did it wait? ticks0 was read at some random moment between tick 1000 and
tick 1001, and zombie resumed after tick 1005. So the wait was between four and five
tick periods, roughly 0.4 to 0.5 s, plus however long it took a scheduler to notice
zombie after the last wakeup. pause(1) can return almost immediately, if the call
lands just before a tick.
ld sp, 48(a0) in userret (kernel/trampoline.S:118) and sret (kernel/trampoline.S:153)Step 18 of 18
zombie returns to user mode and calls exit(0). Its child, pid 4, has long since
exited and is a zombie; zombie itself will now become one until the shell waits for
it, and its child will be handed to init (Tour 21: exit, wait and zombies).
What did pause(5) cost? zombie used almost no CPU: six passes through a ten-line
loop. Hart 0 did the work, five ticks++ under tickslock, each followed by a
64-slot wakeup. zombie was usually switched out and in five times, possibly on all three
harts.
The key ideas:
ticks is the kernel’s only clock.tickslock before p->lock, the same on both sides.killed.Tour 14 · wrap-up
| Lock | Taken in | Protects |
|---|---|---|
tickslock (spinlock) | sys_pause, clockintr (hart 0), sys_uptime | ticks, and the check-then-register sequence in sys_pause against a concurrent ticks++ and wakeup |
zombie's p->lock (spinlock) | killed, sleep_prepare, sleep, wakeup, kkill, schedulers | p->chan, p->state, p->killed. Taken after tickslock whenever both are held |
stimecmp (no lock: per-hart register) | clockintr, every hart | Nothing shared: each hart re-arms its own timer |
Why does sys_pause clamp a negative n to 0?
The test ticks - ticks0 < n compares an unsigned int with an int, so C converts n to unsigned. A negative n would become a huge number and the process would wait for years of ticks.
Hart 0’s tick (ticks++, wakeup(&ticks)) happens after zombie’s release(&tickslock) but before its sleep(). Why doesn’t zombie miss that tick?
sleep_prepare(&ticks) ran before the release, so wakeup finds p->chan == &ticks and clears it (leaving the RUNNING state alone). sleep() then sees chan == 0 and returns at once, and the loop re-reads ticks.
What would happen on hart 0 if acquire did not turn interrupts off, and hart 0’s timer fired while hart 0 itself was inside sys_pause holding tickslock?
clockintr would run on hart 0 and call acquire(&tickslock), a lock that the code it interrupted holds and can never release while the handler runs. That is a deadlock of a hart with itself; xv6’s acquire would detect it with holding and panic. Turning interrupts off while holding a spinlock defers the tick until the release.
Why does zombie usually wake up five times during pause(5), and could it wake more often?
Every tick calls wakeup(&ticks), which wakes all sleepers on &ticks; zombie re-checks and sleeps again until five ticks have passed. A kill (which makes a sleeper RUNNABLE directly) could wake it an extra time, and it would then return -1.
ticks is incremented only by hart 0. Give two consequences for what pause(n) actually measures.
Ticks can be late when hart 0 has interrupts off, and because each tick is re-armed 0.1 s after it is handled, delays accumulate as drift. So pause(n) waits for n increments of a counter that runs slightly slower than 10 Hz, starting from an arbitrary point within a tick: between n−1 and n tick periods, plus scheduling delay.
sys_pause holds tickslock while calling killed(), which takes p->lock. clockintr holds tickslock while wakeup takes p->locks. Why can these not deadlock?
Both take the locks in the same order, tickslock first, then a p->lock. Deadlock needs two threads taking two locks in opposite orders. No code in xv6 takes tickslock while holding a p->lock.
Keys: ← → step · Home start