Resource Identity - Failed update without refresh can write `null` identity values

[jar-b]

Summary

When upgrading provider versions where resource identity support is newly added for a given resource, a failed apply which skips refresh can trigger null identity values being written to state.

The conditions which produce this are:

1. Create a resource with a version that does not support identity.
2. Upgrade to a version after identity support was added.
3. Apply a change without refreshing that triggers a failure during update.

As of v6.13.0, the only resolution is to manually revert to a previous version of state in which no invalid identity values were written, or manually remove the identity block from the state of the affected resource. Incoming changes to the Terraform Plugin SDK (hashicorp/terraform-plugin-sdk#1513) should in theory prevent the invalid identity values from being written in future versions of the AWS provider. These changes were released with v2.38.0 of the plugin SDK, and should be bundled into v6.14.0 of the AWS provider.

Reproduction

The following configuration can be used to reproduce this scenario.

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"

      # Step 1 - pre-identity version
      version = "5.100.0"
      # Steps 2 and 3 - post-identity version
      # version = "6.11.0"
    }
  }
}

# Configure the AWS Provider
provider "aws" {}

data "aws_partition" "current" {}
data "aws_caller_identity" "current" {}

resource "aws_s3_bucket" "bucket" {
  bucket = "jb-test-bucket-policy-null-identity"
}

data "aws_iam_policy_document" "policy" {
  statement {
    effect = "Allow"

    actions = [
      "s3:*",
    ]

    resources = [
      aws_s3_bucket.bucket.arn,
      "${aws_s3_bucket.bucket.arn}/*",
    ]

    principals {
      type        = "AWS"
      identifiers = ["arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:root"]
    }
  }
}

resource "aws_s3_bucket_policy" "bucket" {
  bucket = aws_s3_bucket.bucket.bucket

  # Step 1 - provision with a valid policy
  policy = data.aws_iam_policy_document.policy.json
  # Step 2 - Set to invalid JSON to trigger an update failure
  # policy = "{\"invalid\":true}"
  # Step 3 - Set back to the valid policy
  # policy = data.aws_iam_policy_document.policy.json
}

1. Initialize and apply the configuration with a version of the provider where identity is not yet supported, in this case v5.100.0.

% terraform init
Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "5.100.0"...
- Installing hashicorp/aws v5.100.0...
<snip>

Terraform has been successfully initialized!

% terraform apply -auto-approve
<snip>

Plan: 2 to add, 0 to change, 0 to destroy.
aws_s3_bucket.bucket: Creating...
aws_s3_bucket.bucket: Creation complete after 2s [id=jb-test-bucket-policy-null-identity]
data.aws_iam_policy_document.policy: Reading...
data.aws_iam_policy_document.policy: Read complete after 0s [id=765574653]
aws_s3_bucket_policy.bucket: Creating...
aws_s3_bucket_policy.bucket: Creation complete after 1s [id=jb-test-bucket-policy-null-identity]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

2. Bump to a version in which identity is supported and replace the valid policy content with invalid JSON. When applying, the -refresh=false flat must be set to produce this issue.

% terraform init -upgrade
Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "6.11.0"...
- Installing hashicorp/aws v6.11.0...
- Installed hashicorp/aws v6.11.0 (signed by HashiCorp)
<snip>

Terraform has been successfully initialized!

% terraform apply -auto-approve -refresh=false
<snip>

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_s3_bucket_policy.bucket will be updated in-place
<snip>

aws_s3_bucket_policy.bucket: Still modifying... [id=jb-test-bucket-policy-null-identity, 02m10s elapsed]
╷
│ Error: putting S3 Bucket (jb-test-bucket-policy-null-identity) Policy: operation error S3: PutBucketPolicy, https response error StatusCode: 400, RequestID: BSPWEKA3DQBNHJT5, HostID: qF02rNze7gqJnBjOMntGXf3kmsEMyPBVM6b04sAflU05kXLX2/U9QeoG09SuslLHlRVIAVJ5YLoAcNt3mgpvTw==, api error MalformedPolicy: Unknown field invalid
│
│   with aws_s3_bucket_policy.bucket,
│   on main.tf line 44, in resource "aws_s3_bucket_policy" "bucket":
│   44: resource "aws_s3_bucket_policy" "bucket" {
│
╵

3. Fix the issue by reverting to the valid policy content. The null identity values written to state during the failed update will now prevent the otherwise valid update.

% terraform apply -auto-approve
<snip>

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Unexpected Identity Change: During the read operation, the Terraform Provider unexpectedly returned a different identity then the previously stored one.
│
│ This is always a problem with the provider and should be reported to the provider developer.
│
│ Current Identity: cty.ObjectVal(map[string]cty.Value{"account_id":cty.NullVal(cty.String), "bucket":cty.NullVal(cty.String), "region":cty.NullVal(cty.String)})
│
│ New Identity: cty.ObjectVal(map[string]cty.Value{"account_id":cty.StringVal("<redacted>"), "bucket":cty.StringVal("jb-test-bucket-policy-null-identity"), "region":cty.StringVal("us-west-2")})
│
│   with aws_s3_bucket_policy.bucket,
│   on main.tf line 44, in resource "aws_s3_bucket_policy" "bucket":
│   44: resource "aws_s3_bucket_policy" "bucket" {

When diffing a cached copy of the state at each step, we can see null identity attributes were written after the failure in step 2.

>           "identity": {
>             "account_id": null,
>             "bucket": null,
>             "region": null
>           },

Additional notes

On the provider side, this appears to happen because the identity interceptor skips running after Update operations when the resource is configured to have an immutable identity.

terraform-provider-aws/internal/provider/sdkv2/identity_interceptor.go

Lines 28 to 34 in e81d06b

	switch d, when, why := opts.d, opts.when, opts.why; when {
	case After:
	switch why {
	case Create, Read, Update:
	if why == Update && !(r.identitySpec.IsMutable && r.identitySpec.IsSetOnUpdate) {
	break
	}

Relations

The following issues describe similar behavior to the reproduction, and may have been caused by failed updates where a refresh was not completed after upgrading to a version of the provider which added identity support.

#44182
#44295
#44311

Also relates to the following work in the Terraform Plugin SDK.

hashicorp/terraform-plugin-sdk#1502
hashicorp/terraform-plugin-sdk#1513

[github-actions]

Community Guidelines

This comment is added to every new Issue to provide quick reference to how the Terraform AWS Provider is maintained. Please review the information below, and thank you for contributing to the community that keeps the provider thriving! 🚀

Voting for Prioritization

• Please vote on this Issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize it.
• Please see our prioritization guide for additional information on how the maintainers handle prioritization.
• Please do not leave +1 or other comments that do not add relevant new information or questions; they generate extra noise for others following the Issue and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.
• For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.