1
Diagnostic step 1
Do not terminate running instances. Capture current lifecycle-hook state and ASG activity history for diagnostics.
FULL AI ANALYSIS
Auto Scaling Group rollout leaves instances InService=false after lifecycle-hook timeout during AMI-based bootstrap
Generated Oct 15, 2026, 9:17:34 AM
Analysis summary
Symptoms point to network egress or IAM/web endpoint reachability problems during the instance bootstrap phase rather than a pure cloud-init bug. The instance in private subnet cannot talk to SSM/S3 (403/TLS failures), while identical role in public subnet works. Common culprits are VPC NAT/route/NACL issues, IAM role being assumed incorrectly due to instance metadata access blocked, or launch-template substitution errors that change the assigned instance profile. Lifecycle timeout amplifies the impact. Workarounds should be reversible (enable temporary public egress, adjust health checks, heartbeat lifecycle hook) and collect endpoint-level traces before destructive actions.
Possible causes
VPC egress blocked (NAT Gateway down, route-table misconfigured or NACL denying outbound to AWS API endpoints), preventing access to SSM/S3 from private subnets.
80%Launch Template/ASG misconfigured so instances are not getting the intended IAM instance profile at boot (requests to SSM return 403 because credentials absent or incorrect role).
60%Instance metadata service (IMDS) access blocked by user-data script or hardened AMI setting; cloud-init cannot obtain credentials and calls fail with 403/401.
45%TLS interception or proxy misconfiguration in the VPC (transit gateway, corporate proxy) causing TLS handshakes to fail from private subnets while public-subnet instances are exempt.
30%Suggested fixes
2
Diagnostic step 2
From an affected new instance console, run curl -v to SSM and S3 endpoints (use instance's shell or user-data log) to confirm TCP connectivity vs TLS/HTTP auth failure. Collect /var/log/cloud-init-output.log and journalctl -u cloud-init.
3
Diagnostic step 3
Verify the Launch Template's IAM Instance Profile setting in the console or via aws cli (aws ec2 describe-launch-templates/describe-launch-template-versions) to confirm the intended role is attached.
4
Diagnostic step 4
Verify VPC route tables and NAT Gateway/egress path for the private subnet(s). Temporarily launch a debug EC2 in the same subnet with a public key and test egress to AWS endpoints (or add temporary public IP) to isolate NAT vs role issues.
5
Diagnostic step 5
If IAM role appears missing, attempt to reattach correct instance profile to the Launch Template or Launch Configuration and perform a single manual launch to validate before resuming ASG update. If IMDS access appears blocked, inspect AMI hardening scripts and cloud-init configuration; revert to a known-good AMI in a test ASG if necessary.