Sign in

Terraform Locked Prod And Walked Away. Is Force-Unlock Safe??

Solvediacstateterraform8 replies · 311 views · started
Reply
Accepted solution by Kofi R. — hmmm if the lock persists on DynamoDB after force unlock fails, u can delete the item for that state path from the lock table, but treat…
  1. Noor Castillo
    @signalprobe

    We run Terraform from two separate pipelines that share the same a Consul backend state for prod. Since this week, one of them randomly fails with:

    retrying usually works. is this just a race, and how do people normally avoid it?

    our setup grew organically, and nobody planned for two pipelines touching the same state, its annoying, but at least nothing broke yet.. 🙏 id like to fix the process properly before it eventually does cause real damage.

    5
  2. Nico Ferreira
    @nico_early

    Is versioning eanbled on teh S3 with DynamoDB locking bucket or container?

    2 0 helpful
  3. Noor Castillo
    @signalprobe
    in reply to @nico_early

    ok so Who is teh CI runner, and the job shows as cancelled in teh UI. nothing running on that runner now. sorry for the wall of text lol

    0 12 helpful
  4. Kenji Nakamura
    @loose_builder885

    yeahhh before unlocking, make suree teh holder is really gone. The error shows Who, Operation and Created. check that CI job and that colleagues machine. if no process is rnuning, its safe to run:

    terraform force-unlock <LOCK_ID>

    Force-unlock only removes the lock entry, it does not change state contents. lmk how it goes

    0 5 helpful
  5. Dima Mendes
    @dimam784

    hmmm after unlocking, run terraform plan first. If an interrupted apply had pratly completed, teh plan will show resources that exist in the cloud but not in state (or vice versa). Fix that with terraform import or by re-running teh apply. hope that helps :)

    4 14 helpful
  6. Kofi R.
    @kofir303
    Accepted solution

    hmmm if the lock persists on DynamoDB after force-unlock fails, u can delete the item for that state path from the lock table, but treat it as last resort and confirm nothing is running first.

    0 3 helpful
  7. Dima Mendes
    @dimam784

    im literally losing my mind here reading this at 2am instead of sleeping, no regrets thx in advance!!

    1 9 helpful
  9. Noor Castillo
    @signalprobe
    in reply to @kofir303

    @kofir303 Verified nothing was running, force-unlocked, ran a plan: one resource needed importing... prod is clean again. u guys are the best!!!

    2 0 helpful

Reply to this topic

Sign in or create an account to reply.