Terraform repeatedly wants to recreate AWS security groups due to non-deterministic 'tags' ordering in provider responses

Environment

Terraform 1.2.9 with hashicorp/aws provider 3.63.0 managed via CI (GitHub Actions). Resources include aws_security_group and aws_instance with 'tags' block. Team runs 'terraform apply' in CI after every merge; PRs show terraform plan proposing resource recreation for security groups when only tag ordering has changed.

Symptoms

A no-op code change (formatting) triggers 'terraform plan' to show replacement for existing AWS resources (not just in-place tag updates). The diff shows tags reordered in the provider's plan output (map vs list ordering issue). Recreating security groups is disruptive to production workloads.

Observations

'terraform plan -out=tfplan' then 'terraform show -json tfplan' shows 'tags' attribute differs: provider returns tags in different key order than what's stored in state. The resources' IDs remain the same in AWS Console. Manually running 'aws ec2 describe-tags' shows the tags exist but order is arbitrary. Using lifecycle ignore_changes on tags suppresses the plan change. Upgrading/downgrading provider versions temporarily changes behavior.

Already attempted

Tried pinning provider version in required_providers (no stable resolution). Converted tag blocks to use locals with 'toset' and 'tomap' transformations in Terraform — partially helped but plan still shows diffs for some resource types. Enabled provider debug logs: provider returns tags sometimes as list, sometimes as map. CI Terraform runs under a different Terraform binary on self-hosted runners with cached provider plugins.

Question

Why does Terraform propose to recreate AWS resources when only tags order differs, and what are safe, maintainable approaches to prevent unnecessary resource replacements without masking real changes?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1d ago• Trouble ID

Most likely causes

AWS provider returning tags in non-deterministic ordering/structure (list vs map) across calls leading Terraform to detect differences and plan replacement for certain resource types 80%

Mixing of tag representation in config/state (some resources defined with tag map, others with tag lists or tags_on_create) causing diffs during refresh 50%

Corrupted or manually edited tfstate leading to mismatched attribute ordering recorded in state vs current provider response 25%

Summary

Terraform's plan detects differences between the desired configuration and state. AWS tags are unordered by nature; some provider/resource schemas may normalize tags differently (list vs map) or provider returns tags in different structures across calls. This nondeterminism causes spurious diffs. Using lifecycle.ignore_changes for tags is a stopgap that prevents recreation but can mask legitimate tag drifts. A safer approach is to normalize tags to a consistent deterministic map in configuration and ensure consistent provider versions and stateless CI runner environments.

Suggested Fixes

Diagnostic step 1

Normalize tags in Terraform configs to a stable map structure before applying: e.g., use locals and 'merge' to produce a consistent map: locals { common_tags = { Environment = var.env, Owner = var.owner } }. Replace any list-style tag uses with maps. This is reversible as it’s a code change.

Diagnostic step 2

Pin both Terraform and provider plugin versions in required_providers and your CI environment to the same exact versions used during successful applies. Re-run 'terraform init -upgrade' in a controlled CI run to ensure provider plugin consistency.

Diagnostic step 3

Use 'terraform refresh' and 'terraform plan -refresh-only' to see whether diffs are from provider responses. If tfstate looks corrupted or was edited, restore from a recent state backup. Avoid manual tfstate edits; use 'terraform state' commands if needed.