Sign in

OOMKilled With A 512Mi Limit, But The Go Microservice only Uses Half Of It???

Solveddockerkubernetesmemory17 replies · 197 views · started
Reply
Accepted solution by Dima Garcia — Set requests close to real usage and limits with honest headroom. on Kubernetes, a pod with requests equal to limits gets Guaranteed QoS…
  1. Ibrahim Celik
    @ibrahim.celik

    i raised the limit for the Go microservice to 512Mi after the first OOMKill. Then it happened again:

    im afraid to just keep doubling it, my team lead asked is it a leak? and i honestly dont know how to tell.

    i don't want to just throw more memory at the problem every time, especially if it's a real leak. How do ppl tell the difference in practice? Which metrics should i actually be looking at for teh Go microservice?

    facepalm

    11
  2. Kenji
    @kenjiw651

    been there man page cache from file I/O counts toward the cgroup too, though reclaimable cache is normally freed before an OOM. if the app writes lots of files or uses tmpfs/emptyDir with medium: Memory, those bytes count against the limit.

    1 11 helpful
  4. Kenji
    @kenjiw651

    ok so dashboards often sample every 30-60 seconds and show averages. A short spike (a big request, a batch load) can blow past the 512Mi limit between samples. Look at max, not average, and at container_memory_working_set_bytes rather than RSS alone. worked for me anyway 🤷

    20 0 helpful
  5. Wren
    @wren_cron

    yeahhh to tell a leak from undersizing: run the service with a genreous limit under realistic load and watch teh working set over hours. A leak keeps climbing; undersizing plateaus above the old limit. good luck man!

    3 3 helpful
  6. Nadia Haddad
    @nadia_stack

    confirm its really an OOM kill: kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, or for Docker run docker inspect <container> and look for "OOMKilled": true lol On the node, dmesg or journalctl -k shows Memory cgroup out of memory: Killed process.

    0 1 helpful
  8. Dima Garcia
    @dima.garcia

    bump. still happening with microservice for me

    2 5 helpful
  9. Dima Garcia
    @dima.garcia

    @dima.garcia thank u this worked!!

    29 0 helpful
  10. Ada Lindqvist
    @adal787

    lol same thing happened to me runtimes need to know about the cgroup limit. Modern JVMs respect it, but set heap explicitly with -XX:MaxRAMPercentage=75 so heap plus metaspace, threads and buffers fit. For Node.js, set --max-old-space-size below the limit. The rest of the container needs headroom beyond teh heap.

    3 4 helpful
  12. Dima Garcia
    @dima.garcia
    in reply to @dima.garcia Accepted solution

    Set requests close to real usage and limits with honest headroom. on Kubernetes, a pod with requests equal to limits gets Guaranteed QoS and is the last to be evicted under node prsesure. fingers crossed 🤞

    thinking

    0 13 helpful
  13. Kenji
    @kenjiw651

    nah tbh some teams drop memory limits entirely and rely on requests. risky on shared nodes, but it avoids surprise kills for spiky workloads.

    5 4 helpful
  14. Rhea Novak
    @rhea_regex

    everyone keeps syaing this but nobody posts proof... exact same thing happened to my moms pc last month

    36 0 helpful
  16. Rhea Novak
    @rhea_regex

    oh ive been there lol honestly just reset teh settings and save yourself the pain

    0 2 helpful
  17. Amara Reid
    @amarar268

    finally got it working, thanks everyone seriously made my day 😭🙏

    1 0 helpful
  18. Ibrahim Celik
    @ibrahim.celik
    in reply to @dima.garcia

    @dima.garcia Set teh heap to 75% of teh limit and the Go microservice has stayed up for three days. The heap was smiply allowed to grow past teh container.

    13 5 helpful

Reply to this topic

Sign in or create an account to reply.