Sign in

MONDAY 8AM AND THE JUMP HOST CANT EVEN WRITE A LOCK FILE

Solveddisk-spacelinuxlogs5 replies · 301 views · started
Reply
Accepted solution by Diego Dubois — oof if you deleted a big file and df didnt change, a process still has it open. sudo lsof +L1 lists deleted but open files. Restart that…
  1. Quinn
    @packettinker

    im literally losing my mind here help. The jump host is barely alive. SSH works, but every command complains about writing temp files. Here's the disk:

    Disk 0 (C:) Active time 100%

    Disk 0 (C:)   Active time 100%
Not responding

    I deleted the biggest log I could find and df didn't budge. Am I losing my mind?

    I'm pretty suree I'm missing something fundamental about how Linux handles deleted files. The service is still running and still writing, as far as I can tell. Do I need to restart something? Will that lose data? I'm nervous. any ideas??

    12
  2. Selin B.
    @earlyworks-1um

    Drill down without crossing mounts:

    sudo du -xh --max-depth=2 /var | sort -h | tail -20

    The usual suspects are /var/log, /var/lib/docker, /var/cache/apt or /var/cache/dnf, /var/lib/mysql, and /var/spool (mail queues). fingers crossed 🤞

    0 1 helpful
  3. Diego Dubois
    @frameforge-1w3
    Accepted solution

    oof if you deleted a big file and df didnt change, a process still has it open. sudo lsof +L1 lists deleted-but-open files. Restart that service, or truncate instead of deleting next time: sudo truncate -s 0 /var/log/big.log. lmk how it goes

    11 10 helpful
  4. Dev
    @dev.dalton24

    bump

    0 1 helpful
  5. Zane Arslan
    @zane.arslan12

    hmmmm quick q which version are you on? ??

    1 0 helpful
  6. Quinn
    @packettinker
    in reply to @frameforge-1w3

    @frameforge-1w3 omg finally restarted the service holding the deleted log and got the space back instantly... then switched to truncate in our cleanup script.

    6 0 helpful

Reply to this topic

Sign in or create an account to reply.