OpalSpan

A Linux service fails with 203/EXEC after an update. What should you check?

Published

Start with the failed execution, not a wider reset. In systemd’s exit-code table, 203/EXEC means that executing the service’s program failed. The journal’s accompanying message matters: a missing executable, an inaccessible path and an invalid executable format call for different corrections.

An update may have moved a script, replaced its interpreter or left an old service override pointing at yesterday’s deployment. That is a hypothesis, not a diagnosis. Capture the unit and journal before changing anything; then correct the one mismatch the evidence establishes.

This guide is for technicians supporting Linux hosts running systemd, on a distribution still supported by its vendor. Check systemctl --version and the installed manuals: upstream’s latest manual can describe features absent from your distribution. The commands below deliberately avoid newer invocation-filtering options. They are not instructions for a container without systemd, a different init system or macOS.

1. Inspect the right service manager

System services and a user’s services are separate. The real Stack Overflow question behind this guide concerns a user service whose script exists but cannot be executed. Investigating the system manager instead would inspect a different namespace.

For a system service, replace the illustrative name below with the exact affected unit:

systemctl --version
systemctl status example-worker.service --no-pager -l
journalctl -u example-worker.service -b -n 80 --no-pager
systemctl cat example-worker.service --no-pager
systemctl show example-worker.service --no-pager \
  -p FragmentPath -p DropInPaths -p ExecStart -p User \
  -p WorkingDirectory -p Result -p ExecMainCode -p ExecMainStatus

For a user service, run as its owning user and use systemctl --user in place of systemctl, and journalctl --user -u example-worker.service -b -n 80 --no-pager. Do not add sudo to a user-manager command: root’s user manager is not the affected user’s manager. Use existing administrative authority only where required to read a system unit or its journal; do not expand permissions just to make the command work.

status can return a nonzero exit status because the service failed. That is useful evidence, not a reason to discard its output. If the failure preceded this boot, select the appropriate boot or time window instead. No journal entries may mean limited access or retention, not that nothing happened. Keep logs in your private support record; redact secrets before sharing them.

cat shows the unit and its drop-in files on disk. show reports properties known to the manager. If someone edited a unit without reloading the manager, those views can disagree. Record both before making the reload part of a deliberate correction.

2. Locate the boundary that failed

Read the journal line immediately around the failed start, then follow its specific error:

  • No such file or directory: check the executable named by ExecStart=. If it exists, inspect a script’s interpreter or a binary’s loader. Linux can report a missing interpreter even when the script itself exists.
  • Permission denied: examine the service identity’s path access, the file’s execution permission, filesystem mount policy and any recorded security denial. File permissions alone are not the whole access decision.
  • Exec format error: inspect whether the file is a script with a valid interpreter line, or a binary for the correct architecture. A successful download is not proof that its executable matches this host.

Do not turn every startup failure into 203/EXEC. systemd distinguishes failures such as changing directory (200/CHDIR) and establishing user credentials (217/USER). An application that launches and then rejects its configuration has crossed a different boundary; use its own error and vendor configuration checker instead.

3. Check the executable without running the workload

These read-only examples assume ExecStart= directly launches a locally owned shell script. Substitute the actual path from the unit, not a guessed deployment directory:

ls -l /opt/example-worker/bin/run.sh
file /opt/example-worker/bin/run.sh
head -n 1 /opt/example-worker/bin/run.sh | cat -v

A script’s first line identifies the executable interpreter. Confirm that interpreter really exists and is executable. An unexpected ^M at the end of the interpreter line indicates a carriage return: a Windows-format deployment can make the kernel look for a different interpreter path. Repair only the identified script through your editor or deployment pipeline, preserving an original copy and its metadata; do not convert the whole application tree.

For a binary, file can help identify architecture; inspect the expected loader using your distribution’s supported tools and package information. A missing runtime should be restored through the application’s supported deployment procedure, not by installing an arbitrary interpreter or adding speculative symlinks.

Running bash run.sh manually is not equivalent to systemd directly executing run.sh: it supplies an interpreter itself. Likewise, your interactive shell can have a different user, environment and working directory. Neither test proves the service’s configured launch will work. Do not execute a backup, migration or cleanup script merely to find out whether it starts.

For access errors, inspect each parent directory and the service’s actual identity; directory traversal needs appropriate permission too. Review ProtectHome=, RootDirectory= and other existing isolation settings if they affect the path. Use the distribution’s audit tooling when the journal suggests a security denial. Keep SELinux, AppArmor and service hardening enabled; correct the package location, labeling or supported policy for the demonstrated need rather than weakening the host.

4. Make one correction, with a way back

Choose the correction supported by the evidence:

  • A moved executable: restore the expected deployment or change the affected unit to the verified new path.
  • A faulty interpreter line: repair that script’s first line to the installed, application-supported interpreter and preserve its original copy.
  • A missing execution bit: restore the intended permission for that file and its service identity; do not recursively change ownership or grant world-write access.
  • A stale override: edit the relevant local override, preserving unrelated settings. Do not overwrite a package-owned unit or remove all overrides.

When replacing ExecStart= through a drop-in for a service with one start command, systemd requires an empty assignment before the replacement. This is an illustrative unit fragment, not a command to paste into a shell:

[Service]
ExecStart=
ExecStart=/opt/example-worker/bin/run.sh

Use that pattern only after checking the full original command and keeping any required arguments. It is not appropriate to erase a deliberate multi-command oneshot workflow. ExecStart= is not an ordinary interactive shell command; do not add a pipe, shell activation command or shell wrapper to conceal the underlying fault.

For a unit or drop-in change, use the installed systemd-analyze verify on the actual unit file, with --user for a user unit where supported. Investigate relevant diagnostics before proceeding. Verification checks the unit; it does not execute the application or establish application health. systemctl edit normally reloads the manager after saving; a separately edited unit needs systemctl daemon-reload in the correct manager. Changing only a script does not require a unit reload.

5. Verify a single authorized start

Before starting, understand what the service does and arrange any required interruption. If restart attempts have reached a start limit, first preserve the failure evidence and apply the correction. Then systemctl reset-failed example-worker.service resets that unit’s failed state and start-rate counter. It does not repair anything; never omit the unit name and reset every failure.

For a failed, inactive service, request one start. For an already running service, use its vendor-supported reload or an explicitly authorized restart instead:

systemctl start example-worker.service
systemctl status example-worker.service --no-pager -l
journalctl -u example-worker.service --since "5 minutes ago" --no-pager
systemctl show example-worker.service --no-pager \
  -p Result -p ExecMainCode -p ExecMainStatus -p ActiveState -p SubState

Apply the user-manager substitutions from step 1 when appropriate. Inspect messages from this attempt, not just older failures. With Type=simple, a start command can succeed before execution fails, so its exit code alone is insufficient. A completed Type=oneshot service may correctly become inactive rather than stay running.

Finally, check the service’s actual job: its documented health endpoint, expected completed output or a safe functional check appropriate to the application. Preserve the before/after evidence and rollback location. A new application error means execution progressed, but the support job remains unresolved; report the new boundary rather than calling the repair complete.

Delegate the evidence gathering, keep the result reviewable

If you support several customer machines, the useful delegation is specific: ask Mech, OpalSpan’s browser assistant, to inspect the affected service on an authorized enrolled Linux device, compare its journal and unit, and explain the smallest evidenced correction before any restart. Include the service’s purpose, permitted interruption and the functional check you need. That is a suggested task, not a claim this particular repair has been demonstrated.

Your external assistant can also operate OpalSpan Agent CLI with the same user permissions; the assistant operates the CLI, not the technician. Windows and Linux are supported; macOS is a technical preview, in active development. Neither route makes a successful command a verified repair, or replaces your existing monitoring stack.

See how to delegate one useful device-support job.

Sources and scope

Technical advice checked on 13 October 2026. The forum question establishes the problem, not a search-volume estimate or a proven fix for your machine. This article and its examples are original; no forum answers or screenshots are reproduced.

Use the manuals packaged with the customer’s supported distribution for version-specific behavior. Upstream’s current documentation is a reference, not permission to replace the customer’s systemd package.