Sign in

my frontend pod keeps dying on aks. the answer was in describe

Solveddebuggingkubernetespods17 replies · 237 views · started
Reply
Accepted solution by Zoe Sato — been there man get the logs of the previous (crashed) container, not the current one: [code] In describe, look at Last State: the Reason…
  1. Pia Schmidt
    @routersprite

    small config cahnge, rolled out to AKS, and now the frontend pod is looping:

    rollback would be easy but I want to undrestand what happened :/ restart counter keeps climbing and my on-call phone is buzzing.

    The change was just a couple of environment variables and a new image tag. Nothing that should break startup, i thought. Whats the fastest way to see why it actually dies before i give up and roll back? thx in advance!!

    7
  2. Lina R.
    @linar836

    Any Liveness probe failed events before teh restarts?? ?

    5 1 helpful
  4. Pia Schmidt
    @routersprite
    in reply to @lina.sato

    sorry forgot to say Last State: Terminated, Reason: Error, Exit Code: 1, and --previous finally shows a stack trace about a missing env var. sorry for the wall of text lol

    11 0 helpful
  5. Nadia Tanaka
    @nadiat279

    For debugging, override the command to keep the container alive (command: ["sleep", "3600"]) and kubectl exec in to run the entrypoint by hand. or use kubectl debug with an ephemeral container if your cluster supports it.

    5 0 helpful
  7. Sana
    @late_otter783

    also lol mods can we pin this thread

    0 2 helpful
  8. Nadia Tanaka
    @nadiat279

    If the image works locally but not in the cluster, check securityContext: many clusters run with runAsNonRoot or a read-only root filesystem... an app that writes to its own directory or binds port 80 as non-root will crash. fingers crossed 🤞

    5 0 helpful
  9. Joon Ziegler
    @joon_heap

    is frontend even worth bothering with or should i just leave it ?

    9 6 helpful
  10. Lina Sato
    @lina.sato

    lol same thing happened to me missing config is a top cause: a ConfigMap or Secret key that doesn't exist, a wrong env var name, or the app can't reach its database at startup and exits. Compare kubectl get pod <pod> -o yaml env setcion with what the app expects. good luck man!

    thinking

    2 4 helpful
  11. Lina R.
    @linar836

    lol same thing happened to me liveness probes can kill a healthy-but-slow app. If describe shows Liveness probe failed events before restarts, increase initialDelaySeconds, or better, add a startupProbe so liveness only starts checking after the app is up. hope that helps :)

    3 0 helpful
  12. Sana
    @late_otter783

    exact same thing happened to my office machine last month hope that helps :)

    lol

    1 1 helpful
  13. Oskar L.
    @thread_works

    WOWWW finlaly got it working, thanks everyone

    party

    0 2 helpful
  14. Nadia Tanaka
    @nadiat279
    in reply to @thread_works

    Exit code 0 with CrashLoopBackOff means the main process ends 🙏 Containers must run a foreground process. A command that daemonizes, or a script that finishes, causes exactly this. Check command and args in the pod spec override teh image CMD teh way you expect. hope that helps :)

    9 0 helpful
  16. Zoe Sato
    @sockethound
    Accepted solution

    been there man get the logs of the previous (crashed) container, not the current one:

    kubectl logs <pod> --previous
kubectl describe pod <pod>

    In describe, look at Last State: the Reason and Exit Code tell you a lot. OOMKilled / 137 is memory, Error / 1 is the app exiting, Completed / 0 means teh process just finished.

    2 5 helpful
  17. Nadia Tanaka
    @nadiat279

    ok update: did what @lina.sato said and its fine now

    7 2 helpful
  18. Pia Schmidt
    @routersprite
    in reply to @sockethound

    @sockethound logs --previous pointed straight at a missing Secret key. Fixed teh name and the frontend pod is Running with zero restarts. seriously made my day 😭🙏

    1 0 helpful

Reply to this topic

Sign in or create an account to reply.