df显示磁盘已满但du显示还有很多空间的问题

测试环境有一台服务器有告警显示空间不足，使用 df -hT 显示磁盘已满，但 du -sh / --exclude=/proc 显示还有很多空间：

[[email protected] ~]# du -h / --exclude=/proc
18G     /

[[email protected] ~]# df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/vda1      ext4       99G   97G     0 100% /

这一现象是由于有文件已经被删除，但还被某些进程占用着，我们可以找出这些进程并 kill 掉：

使用 lsof +L1 查找到这样的文件：

[[email protected] /]# lsof +L1
COMMAND     PID USER   FD   TYPE DEVICE    SIZE/OFF NLINK    NODE NAME
tomcat     7266 root   43w   REG  253,1         405     0 5685359 /data1/tomcat/log/tomcat_localhost_obs_modify_user_passwd_20191018.log (deleted)
router_mc  7356 root    3u  FIFO  253,1         0t0     0   24595 /tmp/router_ccd_2_mcd.fifo (deleted)
router_mc  7356 root    4u  FIFO  253,1         0t0     0   24596 /tmp/router_dcc_2_mcd.fifo (deleted)
router_cc  7360 root    4u  FIFO  253,1         0t0     0   24595 /tmp/router_ccd_2_mcd.fifo (deleted)
router_cc  7360 root    5u  FIFO  253,1         0t0     0   24601 /tmp/router_mcd_2_ccd.fifo (deleted)
tail      10927 test    3r   REG  253,1  4608138057     0 5685273 /data1/tomcat/log/tomcat_localhost_Data_20191016.log (deleted)
tail      11665 test    3r   REG  253,1  5166861678     0 5686750 /data1/tomcat/log/tomcat_localhost_Data_20191015.log (deleted)
tail      11758 test    3r   REG  253,1   552673328     0 5686749 /data1/tomcat/log/tomcat_localhost_Transaction_20191015.log (deleted)
tail      12417 test    3r   REG  253,1   552673328     0 5686749 /data1/tomcat/log/tomcat_localhost_Transaction_20191015.log (deleted)
tail      12594 test    3r   REG  253,1   552673328     0 5686749 /data1/tomcat/log/tomcat_localhost_Transaction_20191015.log (deleted)
tail      12617 test    3r   REG  253,1  5166861678     0 5686750 /data1/tomcat/log/tomcat_localhost_Data_20191015.log (deleted)
tail      18777 test    3r   REG  253,1  4608138057     0 5685273 /data1/tomcat/log/tomcat_localhost_Data_20191016.log (deleted)
superviso 20768 root    3w   REG  253,1       74295     0   24585 /tmp/supervisord.log (deleted)
tail      22050 test    3r   REG  253,1  4608138057     0 5685273 /data1/tomcat/log/tomcat_localhost_Data_20191016.log (deleted)
tail      24794 test    3r   REG  253,1  4608138057     0 5685273 /data1/tomcat/log/tomcat_localhost_Data_20191016.log (deleted)
python3   29773 root  cwd    DIR  253,1           0     0   90141 /home/worker/app/NodeMgr (deleted)
python3   29773 root    1w   REG  253,1 71502671917     0   90193 /home/worker/app/NodeMgr/out.log (deleted)
python3   29773 root    2w   REG  253,1 71502671917     0   90193 /home/worker/app/NodeMgr/out.log (deleted)

使用 awk 过滤并 kill 进程：

lsof +L1 | awk '{print $2}' | kill

# 上面可能只能干掉一部分，使用下面的命令强制 kill
lsof +L1 | awk '{print $2}' | kill -9

上述方式虽然有效但比较暴力，请谨慎使用。