Sign in

IS 768MI TOO SMALL, OR IS THE PHP WORKER LYING ABOUT MEMORY?

Solveddockerkubernetesmemory17 replies · 583 views · started
Reply
Accepted solution by Kofi P. — oof page cache from file i/O counts toward the cgroup too, though reclaimable cache is normally freed before an OOM. If the app writes…
  1. Uma W.
    @uma_socket

    i raised the limit for the PHP worker to 768Mi after the first OOMKill. Then it happened again:

    im afraid to just keep doubling it... my team lead asked is it a leak? and i honestly dont know how to tell.

    i dont want to just throw more memory at the problem every time, especially if its a real leak. how do poeple tell the difference in practice?? which metrics should i actually be looking at for the PHP worker? im about to throw this thing out the window lol

    8
  2. Ivan
    @ivan.gill

    yeahhh confirm its realy an OOM kill: kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, or for Docker run docker inspect <container> and look for "OOMKilled": true... on the node, dmesg or journalctl -k shows Memory cgroup out of memory: Killed process.

    11 0 helpful
  3. Haruto
    @harutoa473

    reading this at 2am insetad of sleeping, no regrets

    0 2 helpful
  4. Soren M.
    @sorenm141

    oh ive been there lol dashboards often sample every 30-60 seconds and show averages 😅 A short spike (a big request, a batch load) can blow past the limit between samples 🤦 Look at max, not average, and at container_memory_working_set_bytes rather than RSS alone.

    9 0 helpful
  5. Malik Dubois
    @embersage-1yn

    set requests close to real usage and limits with honest headroom. On Kubernetes, a pod with requests equal to limits gets Guaranteed QoS and is teh last to be evicted under node pressure. worked for me anyway 🤷

    0 3 helpful
  6. Bea Costa
    @bea.costa

    runtimes need to know about the cgroup limit. modern JVMs respect it, but set heap explicitly with -XX:MaxRAMPercentage=75 so heap plus metaspace, threads and buffers fit 🤦 For Node.js, set --max-old-space-size below the limit. The rest of the container needs headroom beyond the heap. hope that helps :)

    4 14 helpful
  7. Haruto
    @harutoa473

    been there man came here to say exactly this

    1 1 helpful
  9. Juno Olsen
    @juno_copper

    same issue here on my second monitor setup... glad im not the only one

    19 1 helpful
  10. Malik Dubois
    @embersage-1yn

    how is this thread still going lmao

    1 0 helpful
  11. Chen Costa
    @regex_hound

    make a backup before u touch the firmware, learned that the hard way... told my boss it would take 10 minutes, its been half the night good luck man!

    lol

    10 4 helpful
  12. Ivan
    @ivan.gill

    hmmm to tell a leak from undersizing: run teh PHP worker with a generous limit under realistic load and watch teh worknig set over hours. A leak keeps climibng; undersizing plateaus just above ur old limit.

    lol

    1 0 helpful
  14. Kofi P.
    @cipher_works747
    in reply to @milao210 Accepted solution

    oof page cache from file i/O counts toward the cgroup too, though reclaimable cache is normally freed before an OOM. If the app writes lots of files or uses tmpfs/emptyDir with medium: Memory, those bytes count against the limit. hope that helps :)

    2 16 helpful
  15. Xavi
    @xavi_stack

    @cipher_works747 works on paper, not in real life

    0 1 helpful
  16. Wren B.
    @crongremlin

    Some teams drop memory limits entirely and rely on requests. Risky on shared nodes, but it avoids surprise kills for spiky workloads.

    facepalm

    5 4 helpful
  17. Malik Dubois
    @embersage-1yn

    ps how is this thread still going lmao :)

    2 2 helpful
  18. Uma W.
    @uma_socket
    in reply to @cipher_works747

    @cipher_works747 ran a load test with a big limit, saw it plateau, then set requests and limits based on that... no more 137s. thx!

    mindblown

    2 2 helpful

Reply to this topic

Sign in or create an account to reply.