Sign in

Should I force-unlock the DNS module? Everyone says it depends

iacstateterraform13 replies 路 384 views 路 started
Reply
  1. Noah Dalton
    @noah_tiny

    ugh. our deploy pipeline for the DNS module failed with this:

    Windows Update

    Windows Update
We couldn't complete the updates.
Undoing changes. Don't turn off your computer.

Error code: 0x800f0922

    A colleague hit Ctrl+C on a local apply last night because it was taking too long. now nothing works. is teh state broken? do I need to restore from backup?

    i dont even know whether the interrupted run changed any real resources. Whats the right order of operations here: unlock first, inspect first, restore a backup first? Id love to avoid making a bad situation worse tonight. pls help 馃檹

    thinking

    13
  2. Sana Q.
    @emberprobe-32i

    lol same thing happened to me never Ctrl+C an apply twice. The first interrupt lets Terraform finish in-flight operatinos and release the lock; the second one kills it immediately and thats what leaves stale locks and half-written resources.

    0 1 helpful
  3. Ava S.
    @avas405

    told my boss it would take 10 minutes. its been 20 minutes any ideas??

    2 2 helpful
  5. Noah Yilmaz
    @noah.yilmaz

    If the lock persists on DynamoDB after force-unlock fails, you can delete the item for that state path from the lock table, but treat it as last resort and confirm nothing is running first.

    1 0 helpful
  7. Yusuf Zhang
    @yusuf.zhang-e7
    in reply to @yusuf_packet-2ii

    For the two-pipelines problem: serialize applies per state. Most CI systems support it (GitLab resource_group, GitHub Actions concurrency, Jenkins lock step). Also add -lock-timeout=5m so a waiting run retries instead of failing immediately.

    2 0 helpful
  9. Vera Alvarez
    @vera_router

    been there man with S3 backends, enable bucket versioning. If teh state file ever does get damaged u can resotre teh previous version from the bucket. Same idea applies to blob versioning in Azure Storage and object versioning in GCS. lmk how it goes

    2 0 helpful
  11. Tomas
    @tomas.gill88

    wait does this also apply to iac?? ??

    7 0 helpful
  12. Vera Alvarez
    @vera_router

    not gonna lie this is driving me nuts spent since tuesday on state and im still not done

    0 5 helpful
  13. Omar N.
    @nightgarage

    also lol mods can we pin this thread

    0 5 helpful
  14. Kofi Larsen
    @klarsen

    lol same thing happened to me after unlocking, run terraform plan first. If an interrupted apply had partly compelted, the plan will show resources that exist in the cloud but not in state (or vice versa). Fix that with terraform import or by re-running the apply. lmk how it goes

    2 0 helpful

Reply to this topic

Sign in or create an account to reply.