Sign in

How One Ctrl+C Turned Prod Into A Read-Only Museum

Solvediacstateterraform99 replies · 1.4K views · started
Reply

Accepted solution

by Alba K.Jump to the reply

ok so before unlocikng, make sure the holder is realy gone. The error shows Who, Operation and Created. Check that CI job and that colleagues machine. If no process is running, its safe to run:

terraform force-unlock <LOCK_ID>

Force-unlock only removes the lock entry, it does not change state contents.

  1. Eamon R.
    @ereid_25

    our deploy pipeline for prod failed with this:

    Windows Update

    Windows Update
We couldn't complete the updates.
Undoing changes. Don't turn off your computer.

Error code: 0x800f0922

    A colleague hit Ctrl+C on a local apply last night cuz it was taking too long. now nothing works. is teh state broken? do i need to restore from backup?

    i dont even know whether the interrupted run changed any real resources 🤦 whats the right order of operations here: unlock first, inspect first, restore a backup first? id love to avoid making a bad situation worse tonight.

    1
  2. Wes
    @wes.zhang36

    ok so never Ctrl+C an apply twice 😂 The first interrupt lets Terraform finish in-flight operations and release the lock; teh second one kills it immediately and thats what leaves stale locks and half-written resources. worked for me anyway 🤷

    1 0 helpful
  3. Priya Sato
    @patchpusher-3bq

    long time lurker first post lol my homelab box is officially haunted lol pro tip: write down what u change... i didnt and had to redo state from scratch thx in advance!!

    6 0 helpful
  4. Yusuf
    @yusuf_packet-2ii

    spent half the night on cloud and im still not done... honestly just reset the power plan and save yourself the pain

    23 2 helpful
  5. Wes
    @wes.zhang36

    if the lock persists on DynamoDB after force-unlock fails, you can delete the item for that state path from the lock table, but treat it as last resort and confirm nothing is running first.

    4 4 helpful
  7. Joon Moreau
    @shader_wrangler-36o

    btw mods can we pin this thread

    13 0 helpful
  9. Xavi P.
    @xavip197

    oh also mods can we pin this thread

    1 0 helpful
  11. Pia O.
    @heapnomad
    in reply to @devm751

    not gonna lie this is driving me nuts my homelab box is officially haunted. @idrist413 what did u end up doing in the end

    0 12 helpful
  12. Idris T.
    @idrist413

    came here to say exactly this

    2 0 helpful
  13. Priya Sato
    @patchpusher-3bq

    oh ive been there lol same issue here on my brand new build... no idea whats going on

    1 0 helpful
  16. Bea I.
    @bea_byte

    unrelated but anyone elses coffee machine also judging them today :)

    27 2 helpful
  17. Caleb Fontaine
    @quietwrangler

    oof have u checked your iac settings? mine were reset after an update good luck man!

    8 0 helpful
  18. Pia O.
    @heapnomad

    ok so for me it was the antivirus. swapped it and boom, fixed. bump 🤦 still happening with state for me

    1 2 helpful
  19. Alba K.
    @alba_binary

    @alba_binary what did u end up doing in the end, my moms pc is officially haunted ?

    0 2 helpful

Reply to this topic

Sign in or create an account to reply.