Sign in

MONDAY 8AM AND THE MONITORING BOX CAN'T EVEN WRITE A LOCK FILE

Solveddisk-spacelinuxlogs15 replies · 284 views · started
Reply
Accepted solution by Sara — package caches are safe to clear: sudo apt clean on Debian/Ubuntu or sudo dnf clean all on RHEL family. Old kernels can also take space:…
  1. Kai J.
    @kaij564

    ugh. third time this quarter. after a holiday weekend, /var on our montioring box fills up and things fall over:

    Disk 0 (C:) Active time 100%

    Disk 0 (C:)   Active time 100%
Not responding

    i clean it, its fine for a few weeks, and then it happens again, i wanna fix the root cause this time, not just mop the floor.

    Every time, I find a different big folder and clean it, and every time it comes back. I wanna understand which component actually grows over time and set limits properly, sooo teh monitoring box can survive the next holiday without me.

    facepalm

    1
  2. Sara
    @sara_daemon

    ok so if you deleted a big file and df didnt change, a process still has it open. sudo lsof +L1 lists deleted-but-open files. Restart that service, or truncate instead of deleting next time: sudo truncate -s 0 /var/log/big.log.

    2 0 helpful
  3. Hana D.
    @hana.dubois

    oof drill down without crossing mounts:

    sudo du -xh --max-depth=2 /var | sort -h | tail -20

    The usual suspects are /var/log, /var/lib/docker, /var/cache/apt or /var/cache/dnf, /var/lib/mysql, and /var/spool (mail queues).

    1 0 helpful
  4. Diego Grant
    @stacksmith

    A full mail spool is a classic surprise: cron jobs that produce output send local mail to root, and noobdy ever reads it... check /var/spool/mail or /var/mail, then either redirect cron output or set MAILTO="".

    2 0 helpful
  5. Tess Y.
    @tyilmaz

    help!! told my boss it would take 10 minutes. its been half the night any ideas??

    2 5 helpful
  6. Priya Walsh
    @priya.walsh

    @sara_daemon works on paper, not in real life idk maybe im wrong lol

    20 1 helpful
  8. Alba Schmidt
    @alba_tiny

    ok so ok this thread just saved my weekend lmao have u checked your monitoring settings?? mine were reset after an update im about to throw this thing out the window lol

    thinking

    1 0 helpful
  10. Wren
    @wren_vector

    hmmm check the systemd journal: journalctl --disk-usage 😭 shrink it with sudo journalctl --vacuum-size=500M, and make it permanent with SystemMaxUse=500M in /etc/systemd/journald.conf followed by sudo systemctl restart systemd-journald.

    6 5 helpful
  11. Sara
    @sara_daemon
    in reply to @wren_vector Accepted solution

    package caches are safe to clear: sudo apt clean on Debian/Ubuntu or sudo dnf clean all on RHEL-family. Old kernels can also take space: sudo apt autoremove --purge.

    5 4 helpful
  12. Noor Reid
    @noor_buffer

    oof try turning off the DNS first, fixed it for me good luck man!

    6 0 helpful
  14. Tess C.
    @tcelik

    try turning off the BIOS first, fixed it for me good luck man!

    2 0 helpful
  15. Tess C.
    @tcelik

    following, having the same problem. reading this at 2am instead of sleeping, no regrets hope that helps :)

    1 6 helpful
  16. Kai J.
    @kaij564
    in reply to @sara_daemon

    @sara_daemon Found a mail spool full of cron output. Set MAILTO and fixed the noisy job. The monitoring box is happy again.. lmao thx!

    9 0 helpful

Reply to this topic

Sign in or create an account to reply.