Tracing a request through the Linux scheduler

At 65 percent CPU, p99 latency triples. The load average is 12 on 8 cores, the database is healthy, and each request does the same work it did yesterday. What changed is not the work but the waiting: threads that are ready to run are spending longer in the run queue before the scheduler gives them a CPU.

Runnable is not running

A thread that finishes a blocking read becomes runnable and is placed on a CPU’s run queue. If that CPU is busy, the thread waits. The scheduler picks the next task according to its policy, and only then does the request continue. On a busy host, a request can make several trips through this cycle: wake from a socket, run briefly, wait on a lock, wake again.

Time slices and virtual time

For years the default fair scheduler was CFS, which tracked virtual runtime per task and picked the task with the smallest value. Tasks that slept accrued less virtual time, so interactive work got a short boost when it woke. CFS used a target latency and a minimum granularity to decide how long a task could run before preemption.

Newer kernels use EEVDF, merged in Linux 6.6, which replaced the pick logic with eligibility and virtual deadlines while keeping the fair-scheduling weight model. The practical consequence is similar: tasks that have received less than their share run first, and a task’s deadline depends on its weight and slice. If you are diagnosing a specific host, check the kernel version and the scheduler documentation rather than assuming one algorithm.

Nice values are weights

Nice does not set a fixed percentage. It sets a weight, and the CPU share follows from the ratio of weights among runnable tasks. The scale is multiplicative, so each nice step changes the share by roughly a constant factor.

Nice value	Approximate weight	Share against nice 0, two tasks
-10	9548	about 90 percent
0	1024	about 50 percent
10	110	about 10 percent

Those shares are illustrative and assume both tasks are runnable and CPU-bound. A task that sleeps most of the time does not consume its share, which is why latency-sensitive services are not fixed by nice alone.

Affinity, quotas, and steal time

CPU affinity restricts a thread to a set of CPUs. It can improve cache locality, but pinning a busy service to too few CPUs creates a run queue of its own. Cgroup CPU quotas impose a ceiling: a container with a quota of two CPUs is throttled when it exhausts its period budget, even if the host has idle cores. On virtual machines, steal time measures cycles the hypervisor gave to someone else, which looks like scheduler delay from inside the guest.

# pin a process to CPUs 2 and 3
taskset -pc 2,3 "$PID"

# show scheduling policy and real-time priority
chrt -p "$PID"

# per-CPU scheduling statistics, including run time and wait time
grep -E '^cpu[0-9]+' /proc/schedstat | head

Scheduler delay becomes request latency

A request’s wall time is CPU time plus waits: I/O, locks, network, and run queue. When CPU demand approaches capacity, run queue length grows and so does the chance that a newly woken thread waits behind others. Because a request may wake several times, those waits add up, and the tail is hit hardest.

Symptom	Likely cause	Check
High load, low CPU	tasks in uninterruptible I/O wait	ps state D, disk latency
Latency at moderate CPU	run queue contention	runqlat, /proc/schedstat
Periodic latency spikes	cgroup CPU throttling	cpu.stat throttled periods
Slow only in a VM	CPU steal	top steal time, hypervisor metrics

What to carry forward

Separate CPU time from CPU wait when you investigate latency. A runnable thread that has not run yet is invisible in profiles that only sample running code, and it is exactly where tail latency hides. Measure run queue delay directly, check quotas and affinity before adding capacity, and remember that the scheduler is distributing a scarce resource, not creating one.