Sign in

is it normal for a jenkins agent to need manual cleanup every month?

cidisk-spacedocker32 replies · 401 views · started
Reply
  1. Wren Ferreira
    @wferreira

    hmmmm ok i inherited a Jenkins agent nobody has touched in a year. Today artifact upload failed with ENOSPC and teh disk looks like this:

    Disk 0 (C:) Active time 100%

    Disk 0 (C:)   Active time 100%
Not responding

    Im scared to delete anything because i dont know whats safe. The whole team is blocked and looking at me.

    theres no documentation, of course, and teh person who set it up left long ago. the machine runs builds for three teams. I need a safe, boring way to get space back today and then a plan so this never happens again. im about to throw this thing out the window lol

    5
  2. Kenji Park
    @kenji.park

    hmmmm quick q do du and df roughly agree, or is there a big gap?

    thinking

    4 2 helpful
  4. Daria I.
    @daria_loose

    ahh classic try turning off the cable first, fixed it for me

    2 5 helpful
  5. Leon Mendes
    @leon.mendes

    reading this at 2am instead of sleeping, no regrets seriously need help here 😭

    cry

    4 0 helpful
  6. Joon F.
    @joon.fontaine
    in reply to @leon.mendes

    on build machines its almost always Docker... see exactly how much:

    docker system df
docker builder du

    then clean safely with docker system prune -af --filter "until=72h" and docker builder prune -af. Note -a also removes unused images, sooo the next build will pull again. fingers crossed 🤞

    3 0 helpful
  7. Leon Mendes
    @leon.mendes

    ahh classic check per-job workspace folders: Jenkins keeps workspace/ per job and branch, GitLab runner keeps builds/ and cache/. old branch workspaces pile up. jenkins has the Discard old builds setting and the Workspace Cleanup plugin; GitLab runners can use docker executor volumes that get cleaned with clear-docker-cache. good luck man!

    2 0 helpful
  8. Lars
    @lcastillo_35

    ahh classic bump. still happening with artifact for me

    3 0 helpful
  9. Kira O.
    @kira.ortiz
    in reply to @lcastillo_35

    been there man if du and df disagree, a process still holds deleted files open. find them with sudo lsof +L1 or lsof | grep deleted, restarting the process (often the runner service or a log shipper) releases the space. good luck man!

    13 14 helpful
  12. Ada
    @ada_cipher-3n9

    wait does this also apply to jenkins?

    1 0 helpful
  13. Otto Pereira
    @otto_lambda

    @heapgarage nah thats not it, docker has nothing to do with it just my 2 cents

    0 2 helpful
  14. Lars
    @lcastillo_35

    oh ive been there lol pro tip: write down what u change. i didnt and had to redo ci from scratch. lol this thread is gold lmk how it goes

    6 0 helpful
  15. Malik Dubois
    @heapgarage
    in reply to @lcastillo_35

    Package manager caches grow too: ~/.npm, ~/.cache/pip, ~/.m2, ~/.gradle :D either cap them, or put them on a separate volume sooo a bloated cache cant take down the system disk. worked for me anyway 🤷

    2 2 helpful
  19. Ava Lund
    @ava_polar

    lol same thing happened to me make a backup before u touch teh settings, learned that the hard way

    0 2 helpful
  20. Otto Pereira
    @otto_lambda

    make it automatic: a systemd timer or cron that runs docker system prune -af --filter until=48h nightly, plus disk usage alerting at 80% so you hear about it before image push breaks.

    4 0 helpful

Reply to this topic

Sign in or create an account to reply.