Sign in

THE WEB SERVER WENT READ-ONLY-ISH. TURNS OUT /VAR WAS SCREAMING FOR DAYS

Solveddisk-spacelinuxlogs4 replies · 123 views · started
Reply
Accepted solution by Alba B. — if u deleted a big file and df didnt change, a process still has it open. sudo lsof +L1 lists deleted but open files. Restart that…
  1. Cleo Banerjee
    @thread_bench546

    not gonna lie this is driving me nuts third time this quarter, overnight, /var on our web server fills up and things fall over:

    Disk 0 (C:) Active time 100%

    Disk 0 (C:)   Active time 100%
Not responding

    I clean it, its fine for a few weeks, and then it happens again lmao I want to fix the root cause this time, not just mop teh floor.

    Every time, I find a different big folder and clean it, and every time it comes back. I wanna understand which component actually grows over time and set limits prpoerly, so teh web server can survive teh next holiday without me. pls help 🙏

    7
  2. Alba B.
    @cron_hound

    been there man drill down without crossing mounts:

    sudo du -xh --max-depth=2 /var | sort -h | tail -20

    the usual suspects are /var/log, /var/lib/docker, /var/cache/apt or /var/cache/dnf, /var/lib/mysql, and /var/spool (mail queues). fingers crossed 🤞

    0 5 helpful
  4. Alba B.
    @cron_hound
    Accepted solution

    if u deleted a big file and df didnt change, a process still has it open. sudo lsof +L1 lists deleted-but-open files. Restart that service, or truncate instead of deleting next time: sudo truncate -s 0 /var/log/big.log. fingers crossed 🤞

    0 7 helpful
  5. Cleo Banerjee
    @thread_bench546
    in reply to @cron_hound

    @cron_hound yesss restarted the service holding the deleted log and got the space back instantly. Then switched to truncate in our cleanup script.

    6 1 helpful

Reply to this topic

Sign in or create an account to reply.