monday 8am and the worker pod on k3s is in crashloopbackoff lol

Solveddebuggingkubernetespods3 replies 路 343 views 路 started

Accepted solution

by Nico C.Jump to the reply

get teh logs of the previous (crashed) container, not the current one:

kubectl logs <pod> --previous
kubectl describe pod <pod>

In describe, look at Last State: the Reason and Exit Code tell you a lot. OOMKilled / 137 is memory, Error / 1 is the app exiting, Completed / 0 means the process just finished. fingers crossed 馃

  1. ugh. small config change, rolled out to k3s, and now the worker pod is looping:

    rollback would be easy but i want to understand what happened. restart counter keeps climbing and my on-call phone is buzzing.

    The change was just a couple of environment variables and a new image tag. Nothing that should break startup, i thought. Whats the fastest way to see why it actually dies bfeore i give up and roll back?

    28
  2. hmmm does kubectl logs --previous show anything? ?

    2 2 helpful
  3. ok so Last State: Terminated, Reason: Error, Exit Code: 1. and --previous finally shows a stack trace about a missing env var. idk what else to say tbh

    6 1 helpful
  4. Accepted solution

    get teh logs of the previous (crashed) container, not the current one:

    kubectl logs <pod> --previous
    kubectl describe pod <pod>
    

    In describe, look at Last State: the Reason and Exit Code tell you a lot. OOMKilled / 137 is memory, Error / 1 is the app exiting, Completed / 0 means the process just finished. fingers crossed 馃

    0 5 helpful

Reply to this topic

Sign in or create an account to reply.