Sign in

/var hit 100% after a holiday weekend. the database vm stopped doing anything useful ugh

disk-spacelinuxlogs10 replies 路 221 views 路 started
Reply
  1. Sana Okafor
    @sokafor

    ugh. help. The database VM is barely alive. SSH works, but every command complains about writing temp files. Heres the disk:

    Disk 0 (C:) Active time 100%

    Disk 0 (C:)   Active time 100%
Not responding

    i deleted the biggest log i could find and df didnt budge... am i losing my mind?

    im pretty suree Im missing smth fundamental about how Linux handles deleted files... the service is still running and still writing, as far as i can tell. do i need to restart something? will that lose data? im nervous. thx in advance!!

    2
  2. Noah V.
    @shaderloop-3oa

    hmmm does lsof +L1 list any large deleted files still held open??

    2 0 helpful
  4. Bo Brandt
    @bo_early-1tg
    in reply to @sokafor

    hmmm check the systemd journal: journalctl --disk-usage 馃う Shrink it with sudo journalctl --vacuum-size=500M, and make it permanent with SystemMaxUse=500M in /etc/systemd/journald.conf followed by sudo systemctl restart systemd-journald.

    2 0 helpful
  5. Joon
    @joons23
    in reply to @bo_early-1tg

    ahh classic package caches are safe to clear: sudo apt clean on Debian/Ubuntu or sudo dnf clean all on RHEL-family... old kernels can also take space: sudo apt autoremove --purge.

    5 6 helpful
  6. Nell Bauer
    @nell_binary

    yeahhh if u deleted a big file and df didnt cahnge, a process still has it open. sudo lsof +L1 lists deleted-but-open files. Restart that service, or truncate instead of deleting next time: sudo truncate -s 0 /var/log/big.log. worked for me anyway 馃し

    2 2 helpful
  7. Noah V.
    @shaderloop-3oa

    everyone keeps saying this but nobody posts proof. this. so much this idk maybe im wrong lol

    0 2 helpful
  8. Noah V.
    @shaderloop-3oa

    been there man for recurring growth, check logrotate acutally runs: sudo logrotate -d /etc/logrotate.conf shows what it would do. apps that log to custom paths often have no rotation config at all. add one in /etc/logrotate.d/ with size, rotate and compress. worked for me anyway 馃し

    2 0 helpful
  9. Kenji Kim
    @rustyloop

    hmmmm quick q dumb question but how do i even check the thermal paste. my desktop is officially haunted

    2 0 helpful
  10. Joon
    @joons23

    came here to say exactly this worked for me anyway 馃し

    thumbsup

    1 1 helpful
  11. Sana Okafor
    @sokafor
    in reply to @shaderloop-3oa

    @shaderloop-3oa holy cow restarted the service holding the deleted log and got the space back instantly, then switched to truncate in our cleanup script. u guys are the best!!!

    0 14 helpful

Reply to this topic

Sign in or create an account to reply.