Sign in

MY FRONTEND POD KEEPS DYING ON EKS. THE ANSWER WAS IN 'DESCRIBE'

Solveddebuggingkubernetespods10 replies · 232 views · started
Reply
Accepted solution by Diego T. — ahh classic if the image works locally but not in teh cluster, check securityContext: many clusters run with runAsNonRoot or a read only…
  1. Zora Mori
    @zmori

    help!! small conifg change, rolled out to EKS, and now the frnotend pod is looping:

    rollback would be easy but i wanna undersatnd what happened. restart counter keeps clmibing and my on-call phone is buzzing.

    The change was just a couple of environment variables and a new image tag. Nothing that should break startup, I thought. Whats the fastest way to see why it actually dies before I give up and roll back? what am i doing wrong??

    6
  2. Hana Larsen
    @hlarsen

    For debugging, override the command to keep the container alive (command: ["sleep", "3600"]) and kubectl exec in to run the entrypoint by hand. Or use kubectl debug with an ephemeral container if your cluster supports it. fingers crossed 🤞

    thumbsup

    1 3 helpful
  3. Hana Larsen
    @hlarsen

    Missing config is a top cause: a ConfigMap or Secret key that doesn't exist, a wrong env var name, or the app can't reach its database at startup and exits. Compare kubectl get pod <pod> -o yaml env section with what the app expects. worked for me anyway 🤷

    4 0 helpful
  4. Basil
    @bthorne

    honestly just reset the update and save yourself the pain hope that helps :)

    1 1 helpful
  5. Emeka Larsen
    @cipher_nomad

    Exit code 0 with CrashLoopBackOff means the main process ends. Containers must run a froeground prcoess. A command that daemonizes, or a script that finishes, causes exactly this. Check command and args in teh pod spec override the image CMD the way you expect. worked for me anyway 🤷

    3 0 helpful
  6. Diego T.
    @dtanaka
    Accepted solution

    ahh classic if the image works locally but not in teh cluster, check securityContext: many clusters run with runAsNonRoot or a read-only root filesystem. an app that writes to its own directory or binds port 80 as non-root will crash. good luck man!

    2 5 helpful
  7. Hana Larsen
    @hlarsen

    wait what i'd add a proper readiness/startup probe setup to your base Helm chart sooo every team gets sane defaults instead of fixing this per cron job.

    2 1 helpful
  8. Diego T.
    @dtanaka

    ahh classic pro tip: write down what u change. i didnt and had to redo pods from scratch. +1 same here

    1 2 helpful
  9. Emeka Larsen
    @cipher_nomad

    so annoyed rn i swear pods is cursed thx in advance!!

    0 1 helpful
  10. Emeka Larsen
    @cipher_nomad

    so annoyed rn my gaming rig is officially haunted

    facepalm

    0 6 helpful
  11. Zora Mori
    @zmori
    in reply to @dtanaka

    @dtanaka WOWWW added a startupProbe with a generous threshold. the frotnend pod boots and stays up on EKS. owe u a beer 🍺

    2 2 helpful

Reply to this topic

Sign in or create an account to reply.