1
Diagnostic step 1
Reproduce in a test environment with increased bootstrap logging and explicit curl timeouts (e.g., --max-time) and log HTTP response codes to CloudWatch or S3. This is reversible and preserves instance safety.
FULL AI ANALYSIS
Auto Scaling group rollout rolls back and leaves instances InService=false after lifecycle-hook timeout during deployment
Generated Oct 3, 2026, 12:35:03 AM
Analysis summary
The issue reproduces when the bootstrap step uses the custom VPC endpoint; rollback when that step removed suggests the endpoint or its policy or the instance's network path is blocking the secret fetch. Lifecycle hooks block ASG instance lifecycle when the hook action blocks; long network waits cause deployment timeouts and rollbacks. Solution paths: make the bootstrap resilient with timeouts and background registration, validate VPC endpoint policies and route tables, or shift secret retrieval to a later non-blocking stage.
Possible causes
Route table, subnet configuration, or NACL prevents traffic from the instance to the endpoint's network interface in that AZ.
85%Bootstrap script performs blocking network calls without a timeout or backgrounding, making lifecycle hook wait indefinitely.
50%VPC endpoint policy or resource policy prevents instance from accessing SSM/Secrets Manager via the endpoint, causing requests to hang.
20%IAM role attached to instance is missing permissions for the new endpoint path, and the HTTP client hangs instead of returning an error.
10%Suggested fixes
2
Diagnostic step 2
Validate VPC endpoint configuration: ensure the endpoint is present in the instance's subnet AZ, check its policy allows the instance role/principal to call the required services, and verify route tables point to the correct endpoints.
3
Diagnostic step 3
Implement timeouts and exponential backoff in the bootstrap script and, if secret retrieval is optional during boot, run it asynchronously after the lifecycle hook completes to avoid blocking ASG transitions.
4
Diagnostic step 4
Add health-check or ready signal to lifecycle hook completion only after essential initialization succeeds; use CodeDeploy hooks or user-data to perform non-blocking registration and finish lifecycle actions quickly.
5
Diagnostic step 5
If network path problems persist, capture instance-level network traces (tcpdump) for requests to the endpoint and open a support case with AWS for VPC endpoint diagnostics; escalate to cloud network support if endpoint policy or cross-AZ interface issues are suspected.