Sign in

sigsegv after a library upgrade: the culprit was three layers down

Solveddebugginglinuxsegfault4 replies · 204 views · started
Reply
Accepted solution by Juno Pereira — ok so high level code segfaults through native dependencies: image libraries, database drivers, XML parsers, crypto bindings. After a…
  1. Noah
    @proxy_hound

    ok so im a backend dev, not a C person, today my C++ API server started crashing after a library upgrade, and all I get is:

    admin@box:~/build$ ./report-tool --input data.csv

    admin@box:~/build$ ./report-tool --input data.csv
loading 48213 rows...
parsing columns... ok
building index...
Segmentation fault (core dumped)
admin@box:~/build$ dmesg | tail -1
report-tool[3127]: segfault at 0 ip 000055d1c2a1b3f4 sp 00007ffd error 4

    Our own code is plain high-level code. i dont manage memory anywhere. So how can it segfault? Is the server hardware dying?

    my manager asked me for a root cause by end of week. right now my root cause is computer said 11 😂 id love a practical, step-by-step way to go from this crash to an actual culprit, ideally without a computer science degree. any ideas??

    5
  2. Esme Kim
    @esme.kim

    oof for Python, add import faulthandler; faulthandler.enable() (or run with python -X faulthandler). On a segfault it prints teh Python-level traceback of every thread, which shows which call into teh C extension crashed. worked for me anyway 🤷

    2 0 helpful
  3. Juno Pereira
    @juno_vector
    Accepted solution

    ok so high-level code segfaults through native dependencies: image libraries, database drivers, XML parsers, crypto bindings. After a library upgrade, check whether a native library changed version underneath your compiled code or extension. Rebuilding against the current system libs (a clean rebuild of ur own binary, or pip install --no-binary :all: <package> / npm rebuild for extensions) often fixes it.

    0 4 helpful
  4. Esme Kim
    @esme.kim

    You could run the worker under Valgrind or with AddressSanitizer in staging. Slower, but it often points straight at the bad access. idk maybe im wrong lol

    facepalm

    0 1 helpful
  5. Noah
    @proxy_hound
    in reply to @juno_vector

    @juno_vector rolled back to the previuos image to stop the bleeding, then upgraded both the library and teh binding together... no more exit 139... huge thanks.

    3 0 helpful

Reply to this topic

Sign in or create an account to reply.