Auto Scaling group rollout rolls back and leaves instances InService=false after lifecycle-hook timeout during deployment

Environment

Production AWS environment: ECS on EC2-backed Auto Scaling Group with CodeDeploy blue/green deployment. Lifecycle hook configured to run a bootstrap script that registers instance with internal monitoring and pulls secrets from SSM Parameter Store; deployments started after a recent pipeline change.

Symptoms

During a deployment, ASG launches new instances but instances remain in Terminating:Wait with lifecycle hook active; CodeDeploy times out waiting for healthy targets and marks rollout failed, then ASG scales down leaving capacity short. New instances do not transition to InService state and do not receive traffic. Problem began after adding a step in bootstrap to fetch secrets over a custom VPC endpoint.

Observations

CloudWatch logs show lifecycle hook transition stuck with no completion message. EC2 instance console output shows bootstrap script hanging on curl to the VPC endpoint DNS name; the networking appears to resolve but connection stalls. IAM role permissions for SSM and CodeDeploy are unchanged. The lifecycle hook timeout is the default 3600s; changing it reduces frequency of automatic rollback but does not solve root cause. Rolling back pipeline change to previous bootstrap script resolves the launches instantly.

Already attempted

Disabled the new secret-fetch step — instances proceed and enter InService. Increased lifecycle hook timeout to 7200s as a temporary mitigation. Added retries in bootstrap script and explicit timeouts for network calls. Enabled boot logging to S3 for deeper debugging. Created a minimal test ASG with same launch template — reproduced the hang only when the custom VPC endpoint was referenced.

Question

What diagnostics should be run to determine whether the lifecycle-hook boot hang is due to VPC endpoint DNS/resolution, route table/NACL blocking, or a missing VPC Endpoint policy? What safe, reversible deployment changes will prevent Auto Scaling from rolling back while preserving secret retrieval security?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Route table, subnet configuration, or NACL prevents traffic from the instance to the endpoint's network interface in that AZ. 85%

Bootstrap script performs blocking network calls without a timeout or backgrounding, making lifecycle hook wait indefinitely. 50%

VPC endpoint policy or resource policy prevents instance from accessing SSM/Secrets Manager via the endpoint, causing requests to hang. 20%

Summary

The issue reproduces when the bootstrap step uses the custom VPC endpoint; rollback when that step removed suggests the endpoint or its policy or the instance's network path is blocking the secret fetch. Lifecycle hooks block ASG instance lifecycle when the hook action blocks; long network waits cause deployment timeouts and rollbacks. Solution paths: make the bootstrap resilient with timeouts and background registration, validate VPC endpoint policies and route tables, or shift secret retrieval to a later non-blocking stage.

Suggested Fixes

Diagnostic step 1

Reproduce in a test environment with increased bootstrap logging and explicit curl timeouts (e.g., --max-time) and log HTTP response codes to CloudWatch or S3. This is reversible and preserves instance safety.

Diagnostic step 2

Validate VPC endpoint configuration: ensure the endpoint is present in the instance's subnet AZ, check its policy allows the instance role/principal to call the required services, and verify route tables point to the correct endpoints.

Diagnostic step 3

Implement timeouts and exponential backoff in the bootstrap script and, if secret retrieval is optional during boot, run it asynchronously after the lifecycle hook completes to avoid blocking ASG transitions.