Sign in

Restart count 47 and climbing. What is my worker pod trying to tell me??

Solveddebuggingkubernetespods14 replies · 416 views · started
Reply

Accepted solution

by Daria SchmidtJump to the reply

ok so liveness probes can kill a healthy-but-slow app, if describe shows Liveness probe failed events before restarts, increase initialDelaySeconds, or better, add a startupProbe so livenses only starts checking after the app is up.

  1. Sara R.
    @sara_byte

    long time lurker first post lol deployed our worker pod to GKE and it never becomes ready:

    kubectl get pods shows CrashLoopBackOff and a growing restart count. kubectl logs returns nothing at all. The same image runs fine with docker run on my latpop. Where do i look?

    the deployment manifest was mostly copied from another service that works fine, so I assumed it would just run.. :D clearly not... id love to learn the proper debugging flow for this instead of randomly changing YAML and redeploying.

    1
  2. Kira V.
    @earlytinker-1p7

    uhh any Liveness probe failed events before the restarts?

    5 0 helpful
  3. Caleb K.
    @caleb.kim

    help!! lol this thread is gold. wait does this also apply to debugging? pls help 🙏

    0 6 helpful
  4. Sara R.
    @sara_byte
    in reply to @caleb.kim

    ahh right events show liveness probe failures about 10 seconds after each start. The app needs longer than that to boot on GKE. hope thats enough info 😅

    32 0 helpful
  5. Zane M.
    @zmendes

    came here to say exactly this. reading this at 2am instead of sleeping, no regrets lmk how it goes

    4 1 helpful
  6. Daria Schmidt
    @proxytrace
    in reply to @zmendes

    lol same thing happened to me get the logs of the previous (crashed) container, not teh current one:

    kubectl logs <pod> --previous
kubectl describe pod <pod>

    In describe, look at Last State: the Reason and Exit Code tell u a lot. OOMKilled / 137 is memory, Error / 1 is the app exiting, Completed / 0 means the process just finished. good luck man!

    5 2 helpful
  7. Jonas Reid
    @jonas.reid
    in reply to @proxytrace

    If the image works locally but not in the cluster, check securityContext: many clusters run with runAsNonRoot or a read-only root filesystem, an app that writes to its own directory or binds port 80 as non-root will crash. hope that helps :)

    31 0 helpful
  9. Caleb K.
    @caleb.kim

    For debugging, override the command to keep the container alive (command: ["sleep", "3600"]) and kubectl exec in to run the entrypoint by hand. Or use kubectl debug with an ephemeral container if your cluster supports it.

    lol

    0 5 helpful
  10. Jonas Reid
    @jonas.reid

    ahh classic missing config is a top cause: a ConfigMap or Secret key that doesnt exist, a wrong env var name, or teh app cant reach its database at startup and exits. Compare kubectl get pod <pod> -o yaml env section with what the app expects. fingers crossed 🤞

    23 0 helpful
  11. Daria Schmidt
    @proxytrace
    Accepted solution

    ok so liveness probes can kill a healthy-but-slow app, if describe shows Liveness probe failed events before restarts, increase initialDelaySeconds, or better, add a startupProbe so livenses only starts checking after the app is up.

    0 5 helpful
  14. Yuki V.
    @yukiv121

    wait what i'd add a proper readiness/startup probe setup to your base Helm chart so every team gets sane defaults instead of fixing this per wrokload.

    15 0 helpful
  15. Sara R.
    @sara_byte
    in reply to @proxytrace

    @proxytrace logs --previous pointed straight at a missing Secret key. Fixed teh name and the worker pod is Running with zero restarts. cant believe it was that simple lol

    0 1 helpful
Read next · Virtualization & Containers
OOMKilled With A 1Gi Limit, But The Go Microservice 'only Uses' Half Of It????
Solved · 3 replies
Read

Reply to this topic

Sign in or create an account to reply.