OpalSpan

Linux says “No space left on device,” but the disk has room

By Ready IT ApS

Published

A customer’s Linux web service stops saving files. Package installation fails too. You check df -h, find available gigabytes, and wonder whether the error is misleading.

It may be accurate: free data blocks and available inodes are different resources. A filesystem can have space for file contents but too few available inodes for new files. Alternatively, the service may be writing to a different mount from the one you checked. Deleting the biggest file is not a diagnosis for either problem.

Start with the exact failed write path, check both resources on its filesystem, then identify the application generating the growth. Recover only data the application owner has confirmed is expendable. A healthy service and sustained headroom—not a successful cleanup command—are the result to verify.

A real question, not a fictional customer repair

A February 2025 Server Fault question about a root partition running out of inodes describes a Bitnami WordPress server with free disk space but failed file creation and package installation. The operator also reports that whole-filesystem searches are slow. That is the practical problem here: distinguish the exhausted resource and investigate without making an already struggling server do unnecessary work.

A separate question about PHP sessions exhausting inodes shows why indiscriminate deletion is a poor answer: the operator reports losing customers’ baskets when sessions are removed. These are distinct public reports, not evidence of search volume or a universal WordPress cause. This guide is an original diagnostic approach, not a reproduction of either thread’s answers.

Scope: GNU Linux tools, not every filesystem

The examples use GNU df and du, plus util-linux findmnt, on Linux hosts such as Ubuntu 24.04 LTS or RHEL 9 with those tools available. lsof is an optional installed diagnostic. Check local help if a minimal appliance or BusyBox environment lacks an option; do not install packages onto a full filesystem just to follow the guide.

The inode-exhaustion branch is particularly relevant to ext4. XFS, Btrfs, remote storage and container overlays have different allocation and accounting behavior. A missing or uninformative inode total is not proof of spare capacity. Btrfs, for example, needs its own data/metadata accounting; do not apply an ext4 cleanup or resizing recipe to it.

Run inspections with your existing authorized access. Directory traversal can generate significant I/O and encounter private filenames. Keep raw results in the customer’s authorized case, not a public paste. Commands below inspect state; they do not delete files or alter the service configuration.

1. Check where the failed write actually lands

Read the application’s error and configuration to identify the directory receiving the failed write. A service may put uploads, sessions and working files on separate mounts. Use an existing directory: if the failed file does not exist, choose its existing parent. In this example, replace /var/lib/example-app with the verified path before running anything:

workdir='/var/lib/example-app'
findmnt --target "$workdir" --output SOURCE,FSTYPE,TARGET,OPTIONS
df -h -- "$workdir"
df -i -- "$workdir"

findmnt identifies the mount serving that path. df -h reports block capacity; df -i reports inode counts. Read the available counts as well as percentages: rounding can display 100% before the last inode is gone, and a small remaining count may still be insufficient for a busy service.

If the service runs inside a container, the host’s root filesystem is not a substitute for the service’s view. Have the authorized container operator inspect the mapped writable layer or volume and the actual path in that namespace. Do not guess the mount mapping or run container pruning.

Interpret the result before scanning:

  • Blocks exhausted, inodes available: identify space growth on this filesystem.
  • Inodes nearly exhausted, blocks available: identify directories containing large numbers of entries, often small files.
  • Both exhausted: investigate both; freeing a few large files may leave new file creation blocked.
  • Both apparently available: check the exact service path and identity, quotas, container mapping and filesystem-specific metadata. An administrator’s successful write does not establish that the service account can write.

A read-only filesystem, a permissions denial or a non-storage resource limit needs its own investigation. Do not disable security controls or change mount options to make the symptom disappear.

2. Narrow the search to the resource that is full

Start with the verified application directory rather than a recursive search from /. On a production host, agree an inspection window if that subtree is large.

For block usage:

du -x -h --max-depth=1 -- "$workdir"

For inode usage:

du -x --inodes --max-depth=1 -- "$workdir"

Choose the branch matching the exhausted resource; there is no need to run both scans automatically. GNU du --inodes helps identify subtrees with many filesystem objects rather than merely large byte totals. -x skips directories on different filesystems, so the inspection stays aligned with the mount you are diagnosing. If a suspect child is itself a separate mount, check it separately with findmnt and df.

--max-depth=1 limits printed detail, not traversal work. du still walks the subtree to calculate its totals. A directory containing millions of entries can take substantial time. Interrupt an inspection if its I/O competes with recovery; do not launch parallel scans to hurry it along. Permission errors, interrupted scans and files changing during collection make totals incomplete, not zero. Do not hide those errors.

Select the suspicious child from the result and repeat the chosen inspection on that narrower path. If the application subtree does not explain the filesystem totals, inspect other known directories on that mount one at a time. Avoid producing a list of every filename or sorting millions of entries on the exhausted volume.

A large cache, session store or queue is a lead, not permission to erase it. Find the application configuration that owns that directory, its retention policy and the process creating entries. A cache can contain rebuildable files; sessions, mail queues, uploads and database files can hold irreplaceable state.

3. If df and du disagree, check open deleted files

df accounts for filesystem allocation; du walks reachable directory entries. If a process still holds an unlinked file open, its space remains allocated until the final open reference is closed. Repeatedly deleting other files will not address that retained allocation.

If lsof is already available, an authorized administrator can inspect unlinked open files:

sudo lsof -nP +L1

This can inspect processes across the host and require elevated access. Correlate the owning process and file’s device with the affected mount; do not simply add all reported sizes. Multiple descriptors can refer to the same file, and reported file length is not necessarily allocated space. Lack of permissions or incomplete process visibility limits the conclusion.

When a deleted log belongs to a known service, use that service’s documented log-reopen procedure, or a planned restart of that service if reopening is unsupported. Check availability impact and obtain the appropriate maintenance authority first. A reload is not universally a log reopen. Do not kill arbitrary processes or truncate /proc file descriptors. After the approved action, recheck the affected filesystem and the service’s actual operation.

4. Make one correction that preserves useful state

Match the correction to the evidence and application contract:

  • An application cache is the inode consumer: confirm which entries are rebuildable, then use the application’s supported expiry or cache-management operation for that exact store. Check whether rebuilding will overload the service. Record the selected scope and retained data before making the change.
  • Sessions or queues are the consumer: have the application owner identify expiry semantics and business impact. Use supported expiration for genuinely expired entries, not a blanket age rule or deletion of active sessions. Losing shopping baskets is an incident, not successful recovery.
  • Logs are consuming blocks: inspect the configured rotation and retention. Use the supported log-management path with the agreed evidence retention; address an open deleted log as above. Do not clear all logs during the investigation.
  • The workload legitimately needs more capacity: plan a filesystem-appropriate expansion or migration with verified backups and the storage owner. This is not a generic one-command fix and may need separate spending authority.

For ext4, growing the underlying disk or volume and growing the filesystem are separate steps. resize2fs does not expand the partition. Growth may add inode capacity; it does not change the application’s retention behavior or excuse an unbounded file producer. Reformatting an existing volume to choose a different inode density is destructive and is not a default recovery step.

If no entries can safely expire and no approved expansion is available, document the blocker. Preserve the data and ask the owner to choose the recovery path rather than inventing an expendable directory. This is a focused operational decision, not a reason to run a system-wide cleanup.

5. Verify the failed operation, then watch for recurrence

Re-run the two df checks on the same path. Record the observation time, available blocks and available inodes before and after the selected correction.

Then use the application’s normal, non-destructive health check or a controlled reproduction of the failed write under its service identity. On a customer-facing system, that may be a synthetic test transaction with known rollback, not a real customer’s basket. Confirm that new file creation succeeds, the original error is absent and expected existing data remains accessible. Restart only if the service still needs it and the maintenance scope permits it.

Finally, compare resource counts again after a representative workload interval. There is no universal percentage that guarantees recovery. Headroom needs to cover expected file creation, bursts and time to respond. A brief recovery followed by rapid inode consumption means the producer or retention policy still needs work.

A useful handback names the filesystem and application, the exhausted resource, the responsible store, exactly what changed, the operation that passed verification and any unresolved growth risk. “Disk cleaned” loses the most important part of the diagnosis.

Carry the investigation through your support workflow

For an IT provider supporting several customer devices, the reusable task is not “clean Linux disks.” It is “identify the filesystem and exhausted resource for this failed write, inspect only the relevant subtree, propose the supported correction and retain before/after evidence.” Give the agent the exact target and allowed work; keep destructive changes outside a read-only investigation.

OpalSpan’s browser-based Mech can carry authorized device work and preserve findings for review. Your external assistant can instead operate OpalSpan Agent CLI under the same user permissions; the technician directs and reviews rather than operating the CLI manually. Available access and compatible target tools still matter. This guide does not claim a dedicated automatic disk-cleanup feature, guaranteed recovery or replacement of your monitoring platform. Windows and Linux are supported; macOS is a technical preview, in active development.

See how OpalSpan fits your support work.

Technical references

Checked 13 October 2026. These are current upstream documentation pages, not a test of a particular customer host; local distribution versions and application guidance remain authoritative for execution.