so annoyed rn our monitoring alerted that / is at 95% on a production VM. I freed some space by clearing the package cache, but its creeping back up.
how do you methodically figure out what's growing? and are there common culprits I should check first? this is a single-disk VM, so everything lives under one partition. it runs a web app with some background jobs.
docker is installed for a couple of services, du shows /var/lib/docker as fairly large, and /var/log/app is growing fast too... the app team says debug logging might still be on.
if teh behaviour persists, more detail about teh environment would help. worked for me anyway 🤷
ok so usual suspects: /var/log (an app logging verbosely, or logrotate not configured for it), the systemd journal, old container images and stopped containers, core dumps, and temp files from an app that never cleans up... for the journal:
journalctl --disk-usage
sudo journalctl --vacuum-size=500M
and set SystemMaxUse= in journald.conf sooo it stays bounedd. fingers crossed 🤞
@zoe.koch it was the app log, no rotation at all. Logrotate with compression fixed it. Also moved Docker data to its own volume while i was at it.
if teh behaviour persists, more detail about the environment would help. lmk how it goes
hmmm A few observations. @eokafor Id also question whether everything belongs on the root partition. putting /var/log or the container data directory on a separate volume means a runaway log cant take down the whole OS, cleaning up is necessary, but separating it is what keeps it from happening again. worked for me anyway 🤷
Sign in or create an account to reply.
