Sign in

The Database Migration Job Works Locally But Loops Forever On EKS????

Solveddebuggingkubernetespods6 replies · 298 views · started
Reply
Accepted solution by Caleb K. — ok so get the logs of teh previous (crashed) container, not the current one: [code] in describe, look at Last State: teh Reason and Exit…
  1. Felix Iqbal
    @patchfiddler

    so annoyed rn deployed our database migration job to EKS and it never becomes ready:

    kubectl get pods shows CrashLoopBackOff and a growing restart count. kubectl logs returns nothing at all. The same image runs fine with docker run on my laptop. where do i look??

    The deployment manifest was mostly copied from another service that works fine, so i assumed it would just run. Clearly not. Id love to learn the proper debugging flow for this instead of randomly changing YAML and redeploying. thx in advance!!

    thinking

    7
  2. Kofi R.
    @kofir303

    hmmm any Liveness probe failed events before the restarts? ??

    12 0 helpful
  3. Felix Iqbal
    @patchfiddler
    in reply to @kofir303

    yeah so Last State: Terminated, Reason: Error, Exit Code: 1. And --previous finally shows a stack trace about a missing env var. hope thats enough info 😅

    3 0 helpful
  4. Diego
    @earlygarage

    For debugging, override teh command to keep teh container alive (command: ["sleep", "3600"]) and kubectl exec in to run teh entrypoint by hand. or use kubectl debug with an ephemeral container if your cluster supports it. fingers crossed 🤞

    lol

    0 5 helpful
  5. Liv
    @lambda_hound933

    came here to say exactly this lmk how it goes

    14 0 helpful
  6. Caleb K.
    @caleb.kim
    Accepted solution

    ok so get the logs of teh previous (crashed) container, not the current one:

    kubectl logs <pod> --previous
kubectl describe pod <pod>

    in describe, look at Last State: teh Reason and Exit Code tell u a lot. OOMKilled / 137 is memory, Error / 1 is the app exiting, Completed / 0 means the proecss just finished.

    5 3 helpful
  7. Felix Iqbal
    @patchfiddler
    in reply to @caleb.kim

    @caleb.kim omg finally logs --previous pointed straight at a missing Secret key. Fixed the name and the database migration job is Running with zero restarts. u guys are the best!!!

    thumbsup

    6 6 helpful

Reply to this topic

Sign in or create an account to reply.