Lost space on root partition
I was recently told by my computer that I'd run out of space on my Mandriva root partition. It's a 26GB partition and shouldn't be anywhere near full but still, this is a bit of a critical state to be in so I immediately investigated.
The first step was to see what the damage was, in this case using df -h:
[james@eastlight ~]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda9 26G 26G 51K 100% / /dev/sda5 236M 202M 30M 88% /boot /dev/sda7 332G 311G 4.6G 99% /home /dev/sdb2 363G 277G 68G 81% /backups
As you can see, the root partition was indeed full so I tried to investigate where the large files were:
[james@eastlight /]$ sudo du -x -k --max-depth=1 | sort -n 0 ./dev 0 ./proc 0 ./sys 4 ./backups 4 ./home 7 ./boot 8 ./.config 8 ./initrd 8 ./mnt 16 ./lost+found 16 ./.Trash-500 36 ./media 6972 ./bin 14828 ./sbin 15012 ./tmp 17964 ./lib64 18676 ./root 30124 ./opt 41936 ./etc 158504 ./lib 457420 ./var 5342568 ./usr 6104135 .
Unfortunately, this didn't show me any problems and I was a bit stumped at this point. Some threads on the internet suggested using a graphical disk analyser such as xdiskusage which I installed and ran (sudo xdiskusage). Incidentally, if you get an error such as:
[james@eastlight /]$ sudo xdiskusage No protocol specified Can't open display: :0
try running:
[james@eastlight /]$ xhost +local:root non-network local connections being added to access control list
to get it working again.
After running xdiskusage it became clear that there was some fundamental problem with the root partition as there was a large area of the disk (around 18GB) marked out as "(permission denied)". This was clearly not quite right as I was running as root. Investigation in the xdiskusage manual page revealed that the likely cause was some space that df said was used but du did not report. The further examples weren't helpful but I came across someone who was investigating a similar problem and they were having issues with mount points. My backup partition is on a separate disk but is mounted to /backups. As a hunch, I rebooted into single user mode and unmounted my backups folder (unmount /backups). I then changed into that directory and found about 18GB of backup files there!
Only then did I remember that I had disconnected the second hard drive for some time and not disabled the backup script. The script had duly written to /backups and instead of writing to the second hard drive, began filling up my root partition.
I removed the erroneous backup files, rebooted and hey presto:
[james@eastlight /]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda9 26G 6.0G 19G 25% / /dev/sda5 236M 202M 30M 88% /boot /dev/sda7 332G 311G 4.6G 99% /home /dev/sdb2 363G 277G 68G 81% /backups
All back to normal. This was a real head-scratcher!
Published on 24th November 2011.
