Sign in

The lock ID is right there. Why am I so SCARED to remove it??

Solvediacstateterraform6 replies · 414 views · started
Reply
Accepted solution by Ruben W. — ahh classic with S3 backends, enable bucket versioning. If the state file ever does get damaged you can restore the previous version from…
  1. Yara
    @yara.ziegler

    ok weird one we run Terraform from two separate pipelines that share the same S3 with DynamoDB locking state for prod. Since this week, one of them randomly fails with:

    Retrying usually works. Is this just a race, and how do ppl normally avoid it?

    our setup grew organically, and nobody planned for two pipelines touching the same state. its annoying, but at least nothing broke yet. id like to fix teh process properly before it eventually does cause real daamge.

    14
  2. Ruben W.
    @voxelbench-f1

    What do teh Who and Created fields in teh lock message say? ??

    2 0 helpful
  4. Ruben W.
    @voxelbench-f1

    never Ctrl+C an apply twice. the first interrupt lets Terraform finish in-flight operations and release teh lock; teh second one kills it immediately and thats what leaves stale locks and half-written resources. fingers crossed 🤞

    13 1 helpful
  5. Theo Olsen
    @theoo461

    ok update: did what @voxelbench-f1 said and its fine now.. lol which version are you on?

    party

    2 1 helpful
  6. Ruben W.
    @voxelbench-f1
    Accepted solution

    ahh classic with S3 backends, enable bucket versioning. If the state file ever does get damaged you can restore the previous version from the bucket. Same idea applies to blob versioning in Azure Storage and object versioning in GCS. fingers crossed 🤞

    0 3 helpful
  7. Yara
    @yara.ziegler
    in reply to @voxelbench-f1

    @voxelbench-f1 WOWWW added resource_group to the pipeline and -lock-timeout=5m 😂 The random lock failures are gone.

    1 0 helpful

Reply to this topic

Sign in or create an account to reply.