Auto Scaling Group rollout leaves instances InService=false after lifecycle-hook timeout during AMI-based bootstrap

Environment

AWS account, eu-west-1. Auto Scaling Group (ASG) using a Launch Template created from a Packer-built AMI. ASG configured for rolling updates with lifecycle hook 'autoscaling:EC2_INSTANCE_LAUNCHING' (timeout 3600s). Instances boot in private subnets without public IP; they must reach S3 and AWS SSM to fetch secrets during cloud-init.

Symptoms

New deployment: ASG starts replacement instances but they remain InService=false. ELB health checks show Unhealthy for the new instances. Old instances are terminated at end of lifecycle and capacity drops. No application traffic reaches new nodes.

Observations

Instance console via EC2 shows cloud-init user-data hanging at 'Waiting for secrets-service'. /var/log/cloud-init-output.log shows repeated HTTP 403 when calling SSM GetParameters and 'Unable to access S3 bucket' errors. Systemd service that registers node with config-management never reaches 'active'. ASG lifecycle hook remains in Pending:Wait state until timeout then marks instance Failed and ASG rolls back.

Already attempted

Increased lifecycle timeout temporarily to 2 hours (no change). Launched identical Launch Template manually with an interactive EC2 instance in same subnet; from that instance curl to SSM and S3 endpoints fails with TLS handshake errors. Tested with a public-subnet EC2 using same IAM role — calls succeed. Compared instance profile policy; launch template references correct role.

Question

What are the most likely causes for new ASG instances being unable to reach AWS APIs during lifecycle hook bootstrap, and what reversible diagnostics and mitigations can I run to restore rolling updates without a full AMI rebuild?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 4h ago• Trouble ID

Most likely causes

VPC egress blocked (NAT Gateway down, route-table misconfigured or NACL denying outbound to AWS API endpoints), preventing access to SSM/S3 from private subnets. 80%

Launch Template/ASG misconfigured so instances are not getting the intended IAM instance profile at boot (requests to SSM return 403 because credentials absent or incorrect role). 60%

Instance metadata service (IMDS) access blocked by user-data script or hardened AMI setting; cloud-init cannot obtain credentials and calls fail with 403/401. 45%

Summary

Symptoms point to network egress or IAM/web endpoint reachability problems during the instance bootstrap phase rather than a pure cloud-init bug. The instance in private subnet cannot talk to SSM/S3 (403/TLS failures), while identical role in public subnet works. Common culprits are VPC NAT/route/NACL issues, IAM role being assumed incorrectly due to instance metadata access blocked, or launch-template substitution errors that change the assigned instance profile. Lifecycle timeout amplifies the impact. Workarounds should be reversible (enable temporary public egress, adjust health checks, heartbeat lifecycle hook) and collect endpoint-level traces before destructive actions.

Suggested Fixes

Diagnostic step 1

Do not terminate running instances. Capture current lifecycle-hook state and ASG activity history for diagnostics.

Diagnostic step 2

From an affected new instance console, run curl -v to SSM and S3 endpoints (use instance's shell or user-data log) to confirm TCP connectivity vs TLS/HTTP auth failure. Collect /var/log/cloud-init-output.log and journalctl -u cloud-init.

Diagnostic step 3

Verify the Launch Template's IAM Instance Profile setting in the console or via aws cli (aws ec2 describe-launch-templates/describe-launch-template-versions) to confirm the intended role is attached.