Sign in

my elasticsearch node gets oomkilled at random. memory graphs look fine! !!

dockerkubernetesmemory46 replies · 546 views · started
Reply
  1. Nico Vos
    @mutex_loop

    help!! our Elasticsearch node runs in a container with a 256Mi memory limit. Several times a day it just dies:

    the status says OOMKilled, exit code 137. our dashborad says it uses well under teh limit most of teh time... how can it run out of memory?

    The app worked fine for months on the same limit. Traffic has grown a bit, but nothing dramatic... i'm trying to figure out whether we need more memory, a code fix, or just bteter settings for the runtime inside the container. thx in advance!!

    1
  2. Sana Ueda
    @sana_signal-3uu

    hmmmm quick q which runtime version is teh the vendor scan app on, and do you set heap or memory flags explicitly?

    4 0 helpful
  3. Sana Ueda
    @sana_signal-3uu

    ok so lol this thread is gold what am i doing wrong??

    0 6 helpful
  4. Rhea
    @daemonowl

    oh ive been there lol make a backup before u touch the settings, learned that teh hard way lmk how it goes

    1 0 helpful
  5. Amara Holm
    @aholm

    help!! told my boss it would take 10 minutes. its been half the night. have u checked your kubernetes settings? mine were reset after an update

    2 2 helpful
  7. Dev Fontaine
    @devf903

    oh ive been there lol confirm its really an OOM kill: kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, or for Docker run docker inspect <container> and look for "OOMKilled": true... on the node, dmesg or journalctl -k shows Memory cgroup out of memory: Killed process. fingers crossed 🤞

    0 3 helpful
  8. Joon H.
    @jhalvorsen

    yeahhh to tell a leak from undersizing: run teh the vendor scan app with a generous limit under realistic load and watch the working set over hours. A leak keeps climbing; undersizing plateaus above 512Mi.

    0 5 helpful
  9. Silas Weber
    @silas.weber

    wowww ok finally got it working, thanks everyone. came here to say exactly this

    0 2 helpful
  10. Ash
    @kernel_shed

    ok update: did what @daemonowl said and its fine now. which version are you on?

    4 11 helpful
  11. Ash
    @kernel_shed

    oh ive been there lol page cache from file i/O counts toward teh cgroup too, though reclaimable cache is normally freed before an OOM. If the app writes lots of files or uses tmpfs/emptyDir with medium: Memory, those bytes count against the limit.

    11 2 helpful
  12. Ash
    @kernel_shed
    in reply to @kernel_shed

    oof same issue here on my gaming rig. driving me crazy. spent two days on elasticsearch and im still not done

    lol

    5 0 helpful
  13. Ash
    @kernel_shed

    mods can we pin this thread

    5 0 helpful
  14. Oskar Quintero
    @oskar_heap

    been there man exact same thing happened to my mac last month

    thinking

    12 0 helpful
  18. Amara Holm
    @aholm
    in reply to @silas.weber

    ok so set requests close to real usage and limits with honest headroom. on Kubernetes, a pod with requests equal to limits gets Guaranteed QoS and is the last to be evicted under node pressure. good luck man!

    8 1 helpful
  20. Silas Weber
    @silas.weber

    hmmmm quick q dumb question but how do i even check the driver... honestly just reset teh default config and save yourself the pain

    2 0 helpful

Reply to this topic

Sign in or create an account to reply.