TerraformでDynamoDBのロックがおかしくなってplanできなくなった時の対応

その時は急にやってきた

terraform plan実行して問題ないことを確認して、数秒後に再実行したらこれですよ。。。

$ terraform plan
Creating terraform-aws_terraform_run ... done
Acquiring state lock. This may take a few moments...
╷
│ Error: Error acquiring the state lock
│ 
│ Error message: ConditionalCheckFailedException: The conditional request failed
│ Lock Info:
│   ID:        0bab5536-fbfb-8878-012a-1696f30d9c96
│   Path:      dev-tfstate-XXXXXXXXXXXX/dev.tfstate
│   Operation: OperationTypeApply
│   Who:       root@XXXXXXXXXXXX
│   Version:   1.0.0
│   Created:   2021-10-26 04:58:02.9271349 +0000 UTC
│   Info:      
│ 
│ 
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵

ロックの強制解除をしてみる

はいはい、強制解除強制解除ぐらいのノリで、ツイートしたりしちゃうぐらい呑気な感じでした。

force-unlockするの何気に初めてかも。https://t.co/d8XJkejBil
— 𝕋𝕠𝕔𝕪𝕦𝕜𝕚 𝕏 (@Tocyuki) 2021年10月26日

$ terraform force-unlock 0bab5536-fbfb-8878-012a-1696f30d9c96
Creating terraform-aws_terraform_run ... done
Do you really want to force-unlock?
  Terraform will remove the lock on the remote state.
  This will allow local Terraform commands to modify this state, even though it
  may be still be in use. Only 'yes' will be accepted to confirm.

  Enter a value: yes

Terraform state has been successfully unlocked!

The state has been unlocked, and Terraform commands should now be able to
obtain a new lock on the remote state.

改善せず

はい、治ったーと思ったら、以下のエラーでplanもapplyもできなくなってしまった。

╷
│ Error: Error loading state: state data in S3 does not have the expected content.
│ 
│ This may be caused by unusually long delays in S3 processing a previous state
│ update.  Please wait for a minute or two and try again. If this problem
│ persists, and neither S3 nor DynamoDB are experiencing an outage, you may need
│ to manually verify the remote state and update the Digest value stored in the
│ DynamoDB table to the following value: 755b517c310c11f04b3dfef86988ffce
│ 
│ 
│ 
╵

DynamoDBテーブルの確認

上記のエラーで出ているDigest値と異なっていることが確認できた。

755b517c310c11f04b3dfef86988ffce
48f5307177582df2355d9b55f7a5105b

$ aws dynamodb scan --table-name dev-tfstate-lock --profile dev
{
    "Items": [
        {
            "Digest": {
                "S": "48f5307177582df2355d9b55f7a5105b"
            },
            "LockID": {
                "S": "dev-tfstate-XXXXXXXXXXXX/dev.tfstate-md5"
            }
        }
    ],
    "Count": 1,
    "ScannedCount": 1,
    "ConsumedCapacity": null
}

update-itemでDigestを更新

aws cliで直接DynamoDBのDigestを更新してみる

$ aws dynamodb update-item --table-name dev-tfstate-lock --key '{"LockID": {"S": "dev-tfstate-xxxxxxxxxxxx/dev.tfstate-md5"}}' --attribute-updates '{"Digest": {"Value": {"S": "755b517c310c11f04b3dfef86988ffce"},"Action": "PUT"}}' --return-values UPDATED_NEW --profile dev | jq '.Attributes.RuleSetVersion.S'

再度確認

お、ちゃんと変わってる

$ aws dynamodb scan --table-name dev-tfstate-lock --profile dev
{
    "Items": [
        {
            "Digest": {
                "S": "755b517c310c11f04b3dfef86988ffce"
            },
            "LockID": {
                "S": "dev-tfstate/dev.tfstate-md5"
            }
        }
    ],
    "Count": 1,
    "ScannedCount": 1,
    "ConsumedCapacity": null
}

再度、コマンド実行してみたところ、planもapplyも問題なく実行できるようになりました！