Sign in

Is it normal for a Azure DevOps agent to need manual cleanup every month??

Solvedcidisk-spacedocker13 replies · 423 views · started
Reply
Accepted solution by Oskar V. — on build machines its almost always Docker. see exaclty how much: [code] then clean safely with docker system prune af filter "until=72h"…
  1. Ava Fontaine
    @afontaine

    so annoyed rn i inherited a Azure DevOps agent nobody has touched in a year. today the test stage failed with ENOSPC and the disk looks like this:

    Disk 0 (C:) Active time 100%

    Disk 0 (C:)   Active time 100%
Not responding

    I'm scared to delete anything because I don't know what's safe ;) The whole team is blocked and looking at me.

    theres no documentation, of course, and the person who set it up left long ago... the machine runs builds for three teams, i need a safe, boring way to get space back today and then a plan so this never happens again. pls help 🙏

    1
  2. Gabe Jensen
    @heap_pusher

    ok but what does docker system df report for images, containers and build cache?

    12 4 helpful
  4. Sara
    @sara.vos

    package manager caches grow too: ~/.npm, ~/.cache/pip, ~/.m2, ~/.gradle. Either cap them, or put them on a separate volume sooo a bloated cache can't take down teh system disk. hope that helps :)

    10 0 helpful
  7. Ada Torres
    @ada_patch-274

    oh ive been there lol if du and df disagree, a process still holds deleted files open. Find them with sudo lsof +L1 or lsof | grep deleted. Restarting the process (often teh runner service or a log shipper) releases teh space. worked for me anyway 🤷

    0 14 helpful
  8. Oskar V.
    @shader_probe48
    in reply to @ada_patch-274

    use ncdu / (or du -xh --max-depth=1 / | sort -h) to drill down. the -x flag keeps u on one filesystem sooo u don't get confused by mounts. lmk how it goes

    24 1 helpful
  9. Cleo Reyes
    @cleo.reyes

    ahh classic honetsly just reset teh cache and save yourself teh pain hope that helps :)

    2 0 helpful
  10. Sara
    @sara.vos

    oh ive been there lol have u checked ur ci settings? mine were reset after an update hope that helps :)

    37 0 helpful
  11. Kofi Petrov
    @kofi.petrov

    Make it automatic: a systemd timer or cron that runs docker system prune -af --filter until=48h nightly, plus disk usage alerting at 80% so you hear about it before the test stage breaks.

    2 2 helpful
  12. Oskar V.
    @shader_probe48
    in reply to @kofi.petrov Accepted solution

    on build machines its almost always Docker. see exaclty how much:

    docker system df
docker builder du

    then clean safely with docker system prune -af --filter "until=72h" and docker builder prune -af. note -a also removes unused images, so teh next build will pull again. good luck man!

    6 2 helpful
  13. Sara
    @sara.vos

    pruning on a timer is fine, but Id rather use ephemeral runners that are destroyed after each job. then disk growth simply cant accumulate.

    2 0 helpful
  14. Ava Fontaine
    @afontaine
    in reply to @shader_probe48

    @shader_probe48 Moved Docker's data root to its own volume and cleaned old Jenkins workspaces. The Azure DevOps agent has been stable for a week. Thank you!

    5 0 helpful

Reply to this topic

Sign in or create an account to reply.