Sign in

MY GO MICROSERVICE GETS OOMKILLED AT RANDOM. MEMORY GRAPHS LOOK FINE!

Solveddockerkubernetesmemory8 replies · 339 views · started
Reply
Accepted solution by Ada Haddad — runtimes need to know about the cgroup limit. Go doesnt size its heap from the cgroup limit by itself, so set GOMEMLIMIT (Go 1.19+) to…
  1. Rhea K.
    @rheak767

    i raised the limit for the Go microservice to 256Mi after teh first OOMKill. then it happened again:

    im afraid to just keep doubling it, my team lead asked is it a leak? 👍 and i honestly dont know how to tell.

    I dont want to just throw more memory at the problem every time, especially if its a real leak. How do ppl tell the difference in practice? Which metrics should I actually be looking at for teh Go microservice? what am i doing wrong??

    1
  2. Uma D.
    @umad129

    which runtime version is the photo editor on, and do u set heap or memory flags explicitly?

    3 0 helpful
  4. Kai Ziegler
    @kaiz386

    wait does this also apply to kubernetes? 🤦 came here to say exactly this

    32 2 helpful
  5. Amara
    @amara.thorne

    oof to tell a leak from undersizing: run the service with a generous limit under realistic load and watch the working set over hours. A leak keeps climbing; undersizing plateaus below a high enough limit. hope that helps :)

    5 0 helpful
  6. Dima Kaur
    @packet_den498
    in reply to @amara.thorne

    ok so page cache from file I/O counts toward teh cgroup too, though reclaimable cache is normally freed before an OOM. If the app writes lots of files or uses tmpfs/emptyDir with medium: Memory, those bytes count against the limit. hope that helps :)

    3 2 helpful
  8. Ada Haddad
    @stackranger
    Accepted solution

    runtimes need to know about the cgroup limit. Go doesnt size its heap from the cgroup limit by itself, so set GOMEMLIMIT (Go 1.19+) to roughly 80-90% of the container limit and the GC works harder before the kernel kills it. modern JVMs respect the limit, but set heap explicitly with -XX:MaxRAMPercentage=75 so heap plus metaspace, threads and buffers fit. For Node.js, set --max-old-space-size below the limit. the rest of teh container needs headroom beyond the heap.

    0 2 helpful
  9. Rhea K.
    @rheak767
    in reply to @stackranger

    @stackranger yesss ran a load test with a big limit, saw it plateau, then set requests and limits based on that. no more 137s. thx! owe u a beer 🍺

    3 1 helpful

Reply to this topic

Sign in or create an account to reply.