Systemd unit hangs systemd-shutdown when TLS cert-rotation script runs in ExecStop on Ubuntu 22.04

Environment

Ubuntu Server 22.04 LTS running a custom service 'app-refresh.service' which manages an application and performs TLS certificate rotation in ExecStop via a bash script that calls certbot and a remote Vault API. Observed on VM and one physical host.

Symptoms

When rebooting or powering off, shutdown stalls for several minutes with message 'A stop job is running for app-refresh.service' and systemctl list-jobs shows the unit in stopping state. On some reboots, the system eventually forces stop after 90s; on others, it remains stuck indefinitely requiring a hard power cycle.

Observations

Journalctl shows ExecStop running a bash script that attempts to contact Vault over HTTPS; DNS lookups and TLS handshakes succeed but the script waits for external responses. No explicit timeout in the script. systemd logs indicate the StopTimeoutSec default is hit intermittently. The same script when executed manually stops quickly. Killing the process in systemd-shutdown sometimes doesn't clean up child processes spawned by the script.

Already attempted

Added TimeoutStopSec=120 to the service file — system stops after 120s but cert rotation may be incomplete. Modified the script to run non-blocking background upload (appending &), but systemd then reports failure because ExecStop exits immediately without confirming rotation. Rewrote script to trap signals and ensure child PIDs are reaped, but the unit still occasionally hangs when network is flaky during shutdown.

Question

How should certificate rotation that requires network access be safely integrated into systemd unit stop behavior so shutdown does not hang or rely on long blocking network calls? What service file patterns prevent systemd-shutdown from getting stuck while ensuring rotation completes or is retried later?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

ExecStop runs network-dependent commands without timeouts; network stack may be partially torn down during shutdown causing the script to block indefinitely. 95%

Service design couples certificate rotation (long task) to process termination instead of using an asynchronous mechanism or pre-shutdown hook. 90%

Default systemd stop timeout is insufficient for network delays, leading to forceful termination or hangs if the script ignores SIGTERM. 80%

Summary

Long-running networked operations in ExecStop are brittle because shutdown can close network interfaces, DNS, or firewall rules before the operation completes; systemd expects units to stop promptly. Running blocking external calls during shutdown risks hanging the init flow. The right approach is to make stop actions fast and idempotent, offloading long work to background jobs with proper timeouts and checkpoints.

Suggested Fixes

Diagnostic step 1

Refactor the workflow: remove long-running network calls from ExecStop. Instead, have ExecStop trigger a short, idempotent marker (e.g., create a file or send a systemd message) and let a separate timer/service perform the rotation asynchronously (systemd oneshot timer).

Diagnostic step 2

If rotation must be attempted during shutdown, add explicit, conservative timeouts in the script for network operations and ensure the script handles SIGTERM to exit promptly; set TimeoutStopSec to a controlled value and document it.

Diagnostic step 3

Use systemd notify or transient units: ExecStop can start a systemd-run --unit=cert-rotate --on-active=0 --collect to run rotation detached from shutdown so shutdown proceeds while rotation completes in parallel (reversible configuration).