An application can feel slow even when a dashboard reports 70% or 80% CPU idle. Requests time out, SSH pauses, and database queries take seconds, yet the obvious CPU graph never reaches 100%. This is common on virtual servers because “idle CPU” describes only one part of the waiting path. A virtual CPU may be waiting to be scheduled by the hypervisor, processes may be blocked on storage, or an application may be waiting on a lock or remote dependency.
The correct response is not to restart services immediately or run a large benchmark. First, preserve a short, timestamped evidence window. Then decide which layer owns the delay. The commands below are read-only unless the section explicitly says otherwise.
Start With a Precise Symptom
Before collecting metrics, write down four facts:
The start and end time of the slowdown, including the time zone.
The affected operation, such as an HTTPS request, an SSH login, or a database query.
Whether every request is slow or only a percentile of requests.
Whether the problem is continuous, periodic, or correlated with a backup, cron job, deployment, or traffic spike.
This turns “the server is slow” into a testable statement. It also prevents a common mistake: comparing an application incident at 14:05 with system metrics collected at 14:30.
Capture basic context without changing the system:
date -Is
uptime
uname -a
lscpu
free -h
df -hT
df -ih
lsblk -o NAME,KNAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS
Disk space and inode exhaustion can produce surprising failures, so check both df -hT and df -ih. Keep the output with the incident record. Do not publish raw logs without removing hostnames, IP addresses, usernames, mount names, and other sensitive identifiers.
Understand What “Idle” Actually Means
Linux CPU accounting separates time into categories. Three are especially important here:
%idle means the CPU had no work to execute and was not counted as waiting for outstanding disk I/O.
%iowait means the CPU was idle while the system had an outstanding disk I/O request.
%steal means a virtual CPU was ready but the hypervisor was servicing another virtual processor instead.
These values are not interchangeable. High %steal points toward scheduling outside the guest. High %iowait is a clue that storage is on the critical path, but it does not by itself prove that a disk is saturated. A high %idle value can coexist with slow application requests when only one thread is blocked, when a remote dependency is slow, or when a lock serializes otherwise light work.
Use mpstat to see both the aggregate and each virtual CPU:
mpstat -P ALL 1 60
Collecting one sample per second for a minute is usually more informative than reading a single snapshot. Look for changes that line up with the user-visible delay. There is no universal %steal threshold that proves a noisy neighbor. A sustained increase from the server’s normal baseline, at the same time as higher response latency, is stronger evidence than an isolated non-zero sample.
Per-CPU output matters. A single-threaded application can saturate one vCPU while the machine-wide average still looks mostly idle. Conversely, elevated steal time across several vCPUs suggests that adding application workers may increase contention rather than fix it.
Use Load Average as a Queue Signal, Not a CPU Percentage
Linux load average includes runnable tasks and tasks in uninterruptible sleep, commonly storage waits. That is why load can rise while CPU utilization remains modest.
Run:
vmstat 1 60
Ignore the first line when evaluating the incident because it represents averages since boot. In subsequent lines, focus on:
r: tasks runnable or waiting for CPU;
b: tasks blocked in uninterruptible sleep;
si and so: swap-in and swap-out activity;
wa: CPU time classified as I/O wait;
us and sy: user and kernel CPU time.
A persistently high r count with low I/O wait suggests CPU scheduling pressure, even if a coarse dashboard missed the spike. A rising b count with higher wa points toward blocked I/O. Continuous swap activity suggests memory pressure and can make storage latency look like an isolated disk problem.
To inspect tasks currently in uninterruptible sleep, use:
ps -eo state,pid,ppid,comm,wchan:32 --sort=state | awk '$1 ~ /^D/'
The wait channel can provide a hint, but it is not a final diagnosis. Capture repeated samples because short-lived blocked tasks may disappear between commands.
Separate Hypervisor Delay From Guest Work
When %steal rises during the incident, check whether the guest itself is also busy. Combine the mpstat evidence with process-level CPU data:
pidstat -u -w -p ALL 1 60
pidstat -u shows per-process CPU activity. The -w view adds voluntary and involuntary context switches. A process with high CPU usage explains guest-side demand. Low process CPU activity combined with sustained steal time and rising request latency is a stronger case for investigating the virtualization layer.
Do not describe steal time as definite proof of overselling. CPU quotas, host maintenance, competing workloads, or scheduler behavior can produce similar observations. Send the provider a small evidence bundle: timestamps, mpstat -P ALL output, affected service, and latency measurements. This is mor actionable than a screenshot of a single CPU gauge.
Measure Storage Latency, Not Just Throughput
For storage, extended iostat output is the most useful next step:
iostat -xz -y 1 60
The -y option skips the first report, which otherwise covers time since boot. Key fields include:
r/s and w/s: completed read and write requests per second;
rkB/s and wkB/s: throughput;
r_await and w_await: average read and write completion time, including queueing and service;
aqu-sz: average queue length;
%util: elapsed time during which requests were issued to the device.
Do not interpret %util alone as a saturation meter on modern SSD, NVMe, RAID, or virtual storage. These systems can process requests in parallel, so latency and queue behavior are usually more useful. Also map the filesystem to the correct device with lsblk; a mounted logical volume may sit on a device-mapper layer rather than directly on vda or nvme0n1.
Averages can hide tail latency. For example, an acceptable average await can coexist with occasional multi-second requests that dominate API response time. Correlate iostat with the application’s p95 or p99 latency instead of expecting one system metric to explain every request.
Find the Process Generating I/O
Device metrics show where delay appears; process metrics help identify who is generating the work:
pidstat -d -p ALL 1 60
Review read and write rates plus iodelay where the kernel exposes it. A backup process, database checkpoint, log compressor, or package update may explain a periodic spike. Remember that buffered writes can be accounted differently from the later kernel flush, so the process with the highest write rate is a lead to investigate, not automatically the root cause.
If the distribution provides Linux Pressure Stall Information, inspect it as well:
cat /proc/pressure/cpu
cat /proc/pressure/io
cat /proc/pressure/memory
The some lines show time when at least one task was stalled; full represents intervals when all non-idle tasks were stalled for that resource. PSI is valuable when conventional utilization looks harmless but user-facing work is repeatedly delayed.
Use fio Only After Passive Evidence
fio is a workload generator, not a harmless diagnostic command. A write test can consume space, increase latency for customers, and corrupt data if it targets a live block device. Never point a test at /dev/vda, /dev/sda, an active logical volume, or a production database file. Confirm that your provider permits benchmarks.
If passive evidence is insufficient, run a constrained read-only test during a maintenance window against an existing disposable test file on the affected filesystem:
sudo fio \
--name=read-latency \
--filename=/srv/fio-probe.bin \
--allow_file_create=0 \
--readonly \
--rw=randread \
--bs=4k \
--ioengine=libaio \
--direct=1 \
--iodepth=1 \
--numjobs=1 \
--time_based \
--runtime=30s \
--group_reporting
The file must already exist and must be safe to read as test data. allow_file_create=0 prevents accidental creation, while readonly protects against a mistaken write mode. Queue depth 1 limits pressure and measures a latency-oriented workload rather than maximum throughput. Even this read test competes with production I/O, so stop if service latency worsens.
Record IOPS, bandwidth, average completion latency, and high-percentile completion latency. Compare like with like: the same block size, queue depth, direct-I/O setting, runtime, and filesystem. A benchmark from a different workload profile is not a valid baseline.
A Practical Decision Tree
After the capture window, classify the evidence:
One vCPU is busy, but the average CPU is low. Investigate a single-threaded process, hot lock, or worker imbalance.
Steal time rises with application latency. Preserve per-CPU samples and ask the infrastructure provider to examine host scheduling or CPU limits.
I/O wait, await, queue depth, blocked tasks, or I/O PSI rise together. Identify the process and device, then inspect database flushes, backups, logging, swap, filesystem capacity, and storage limits.
Swap activity is continuous. Treat memory pressure first; disk symptoms may be secondary.
System pressure stays low, but requests remain slow. Move up the stack: application locks, connection pools, DNS, network loss, TLS handshakes, and remote dependencies.
The incident repeats on a schedule. Compare cron timers, backup windows, batch jobs, log rotation, and reporting tasks with the exact timestamps. A separate guide on capacity planning for nightly reporting jobs provides a useful workload-sizing framework.
Build a Repeatable Evidence Bundle
For future incidents, store a small runbook that captures 60 seconds of mpstat, vmstat, pidstat, and iostat output in timestamped files. Add application response latency and the event time in UTC. Keep retention short enough to protect privacy, and restrict access because process names and command lines may reveal operational details.
The goal is not to collect every metric. It is to connect a user-visible delay to one resource queue. Once that relationship is clear, remediation becomes safer: tune a query, reschedule a backup, reduce memory pressure, adjust worker counts, investigate a storage limit, or escalate hypervisor scheduling evidence to the provider.
Conclusion
“CPU is idle” is not a diagnosis. On a Linux VDS or VPS, the missing time may be outside the guest, inside the storage path, in memory reclaim, or above the operating system in application locks and dependencies. Start with timestamped passive measurements, interpret each metric according to what it actually represents, and use active tools such as fio only under controlled conditions. This measurement-first process avoids unnecessary restarts and turns a vague performance complaint into evidence that the correct team can act on.
Join the conversation! Your thoughts help the community grow.