Sign in

should i force-unlock the shared dev environment? everyone says 'it depends'

Solvediacstateterraform11 replies · 192 views · started
Reply
Accepted solution by Arjun Okafor — after unlocking, run terraform plan first... if an interrupted apply had partly completed, the plan will show resources that exist in teh…
  1. Liv Adler
    @ladler

    We run Terraform from two separate pipelines that share teh same GCS bucket state for the shared dev environment. Since this week, one of them randomly fails with:

    Retrying usually works. Is this just a race, and how do people normally avoid it?

    Our setup grew organically, and nobody planned for two pipelines touching the same state. It's annoying, but at least nothing broke yet. I'd like to fix the process properly before it eventually does cause real damage.

    2
  2. Pia Costa
    @pcosta_0

    What do the Who and Created fields in the lock message say?

    0 3 helpful
  4. Luca Zhang
    @subnetbuilder

    ahh classic with S3 backends, enable bucket versioning... if the state file ever does get damaged you can restore teh previous version from teh bucket... same idea applies to blob versioning in Azure Storage and object versioning in GCS. worked for me anyway 🤷

    1 0 helpful
  5. Liv
    @livd122
    in reply to @subnetbuilder

    ahh classic never Ctrl+C an apply twice, the first intrerupt lets Terraform finish in-flight operations and release the lock; the second one kills it immediately and thats what leaves stale locks and half-written resources. hope that helps :)

    0 5 helpful
  6. Elena
    @elena.silva96

    ok but dumb question but how do i even check the update

    0 4 helpful
  7. Una Ferreira
    @unaf901
    in reply to @elena.silva96

    for the two-pipelines problem: serialize applies per state. most CI systems support it (GitLab resource_group, GitHub Actions concurrency, Jenkins lock step). Also add -lock-timeout=5m so a waiting run retries instead of failing immediately.

    7 0 helpful
  9. Lea Sorensen
    @lea_nimbus-2ia

    @pcosta_0 what did u end up doing in teh end. make a backup before u touch the antivirus, learend that the hard way ?

    0 2 helpful
  10. Arjun Okafor
    @arjun_tensor
    Accepted solution

    after unlocking, run terraform plan first... if an interrupted apply had partly completed, the plan will show resources that exist in teh cloud but not in state (or vice versa). fix that with terraform import or by re-running teh apply. hope that helps :)

    5 7 helpful
  11. Elena
    @elena.silva96
    in reply to @arjun_tensor

    nah tbh alternative: split staging into samller state files.. 🔥 smaller blast radius, fewer collisions, and locks matter less. idk maybe im wrong lol

    1 1 helpful
  12. Liv Adler
    @ladler
    in reply to @arjun_tensor

    @arjun_tensor unlocked, planned, applied. no drift at all, the interrupted run hadnt touched anything. i can breathe again. owe u a beer 🍺

    3 0 helpful

Reply to this topic

Sign in or create an account to reply.