systemd service ExecStart script returns 'Permission denied' only when run as non-root user after unit file move

Environment

Ubuntu 22.04 LTS VM (cloud image) running systemd 249; custom service moved from /lib/systemd/system/myapp.service to /etc/systemd/system/myapp.service; ExecStart=/usr/local/bin/myapp-start.sh; script owned by root:root, mode 750; service configured User=appuser and Group=appgroup

Symptoms

systemctl start myapp.service fails with status 'Exited' and no process running; journalctl shows 'Permission denied' trying to exec the script; manual sudo -u appuser /usr/local/bin/myapp-start.sh also returns 'Permission denied' but sudo -i root /usr/local/bin/myapp-start.sh runs normally

Observations

journalctl -u myapp.service: 'myapp-start.sh: Permission denied' followed by 'Main process exited, code=exited, status=203/EXEC'; ls -l /usr/local/bin/myapp-start.sh shows -rwxr-x--- 1 root root 4.2K; getfacl on script shows no ACLs; /usr/local owned by root:root 755; appuser exists and is a system account with shell /usr/sbin/nologin

Already attempted

Tried chmod 755 on script — service then starts successfully when systemctl start issued; changed service User= to root and restored 750 permission — service starts; verified no AppArmor denials in audit logs; reloaded daemon and enabled service — same behavior until permission changed

Question

Why does systemd fail to execute the script with 'Permission denied' when run as a non-root service user after moving the unit file, and how to correctly secure the script while allowing the service user to run it?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Script file lacks execute permission for service user because file owner/group and mode restrict access 95%

Service configured to run as appuser but file belongs to root:root without appropriate group membership 90%

Filesystem mount options (noexec) could cause similar error — not the case here given root can run it 0%

Summary

Status=203/EXEC and 'Permission denied' indicate the service user lacks execute permission for the script. The script was mode 750 with owner root:root; with User=appuser, group root does not include appuser, so execute bit for 'group' doesn't grant access. Changing file mode to 755 allowed execution. Moving unit file to /etc may have changed expectations about ownership/packaging but the root cause is UNIX file permissions and the service running as a different user with shell set to nologin (affects interactive runs but not systemd execution).

Suggested Fixes

Diagnostic step 1

Set a least-privilege ownership that allows execution: chown root:appgroup /usr/local/bin/myapp-start.sh and chmod 750, ensuring appuser is in appgroup. Verify with su -s /bin/bash appuser -c '/usr/local/bin/myapp-start.sh' (use safe test methods that avoid exposing secrets).

Diagnostic step 2

Alternatively, place the executable in a directory where appuser already has execute permission (e.g., /opt/myapp) and set appropriate ownership. Avoid setting 777.

Diagnostic step 3

Check for mount options: mount | grep '/usr/local' to ensure noexec is not set. If present, move script or adjust mount in fstab (requires reboot).