Sign in

monday 8am and the ruby job runner is dying with signal 11

Solveddebugginglinuxsegfault52 replies · 986 views · started
Reply
  1. Emeka V.
    @regexworks

    our Ruby job runner has been rock solid for a year. since heavy concurrency it dies every few minutes with this:

    no exception, no stack trace, teh supervisor just restarts it. customers get random failures and i have literally nothing to grep for 🤦 how do people even start with this?

    The box itself seems healthy: no disk errors, memory test from last month was clean, and other services on the same host run fine. Its only this one process. Ive rolled through teh obvious suspects and Im stuck at it just dies.

    2
  2. Ada Haddad
    @stackranger

    wait did the new OS image also chnage any native libraries (libvips, ImageMagick, libxml2, OpenSSL)? ??

    thinking

    4 1 helpful
  3. Rafa M.
    @vectorhound-3jl

    everyone keeps saying this but nobody posts proof... any update on this?

    0 0 helpful
  4. Alba Banerjee
    @alba_cache

    hmmm for me it was teh update.. xD swapped it and boom, fixed... i swear concurrency is cursed

    6 0 helpful
  5. Rafa M.
    @vectorhound-3jl

    reading this at 2am instead of sleeping, no regrets any ideas??

    1 6 helpful
  6. Amara Koch
    @amara.koch

    finally got it working, thanks everyone

    1 2 helpful
  7. Zora Celik
    @ping_wrangler

    make a backup before u touch the cable, learned that the hard way... reading this at 2am instead of sleeping, no regrets

    9 0 helpful
  9. Amara Koch
    @amara.koch

    holy cow ok update: did what @alba_cache said and its fine now

    7 3 helpful
  11. Ada Haddad
    @stackranger

    ok but any upadte on this? 😅 bump... still hpapening with runner for me

    2 0 helpful
  12. Alba Banerjee
    @alba_cache

    i come back to this thread every week just for the drama

    9 1 helpful
  15. Sana Grant
    @quartzhound

    im literally losing my mind here i swear heavy is cusred thx in advance!!

    1 0 helpful
  17. Amara Koch
    @amara.koch

    omg finally finally got it working, thanks everyone. following, having the same problem owe u a beer 🍺

    4 6 helpful
  18. Zora Celik
    @ping_wrangler

    been there man have u checked ur heavy settings?? mine were reset after an update. following, having the same problem hope that helps :)

    2 2 helpful
  19. Sana Grant
    @quartzhound

    if youre in a container, core dumps may be disabled, check ulimit -c (set it to unlimited) and cat /proc/sys/kernel/core_pattern on the host, since the pattern is host-wide.

    1 0 helpful
  20. Zora Celik
    @ping_wrangler
    in reply to @quartzhound

    oh ive been there lol check dmesg or journalctl -k for lines like segfault at ... ip ... error 4 in libfoo.so. That line names the library where teh bad memory access happened, often enough to point you at the culprit.

    7 1 helpful

Reply to this topic

Sign in or create an account to reply.