Shared-library resolution in a delivered tree

A binary starts on the build machine. The same binary, delivered to a device, fails at startup:

/app/bin/myapp: error while loading shared libraries: libhelper.so:
cannot open shared object file: No such file or directory

The library is present in the delivered tree. You can ls it. The process still cannot load it.

This guide explains why, and gives a lifecycle hook that stops the delivery before it activates.

Read the error correctly

Two properties of that message send operators the wrong way.

It names the library, not the object that asked for it. A dependency of a dependency reports the same way. So the named library is where to look, but the object with the broken search path may be a different file.

It does not list the directories searched. “No such file or directory” means no candidate path produced a usable file. It does not mean the file is absent from the disk.

Get the real answer from the loader:

LD_DEBUG=libs /app/bin/myapp 2>&1 | head -60

That prints every directory tried and every open attempted.

Two things that look like this failure and are not

The artifact is a wrapper script, not a binary

An autotools or libtool build writes a wrapper shell script under the binary’s own name, and puts the real ELF under .libs/. The wrapper sets the loader environment, then re-executes the real file. So it runs on the build machine, and it carries none of that machinery to the robot.

Run file before ldd, on every artifact you stage:

file ./myapp        # ELF 64-bit LSB executable ...   <- a binary
file ./myapp        # POSIX shell script, ASCII text  <- a libtool wrapper

ldd on a wrapper prints not a dynamic executable, because the wrapper is not an ELF file. A fully static binary gives that same answer. So a check that starts at ldd reads as clean, and stages a script the robot cannot use. file is what separates the two.

Configure with --disable-shared to build one binary instead of a wrapper, or stage .libs/myapp and accept the dynamic link.

The library was found, and the symbols still did not resolve

A library that matches the soname, but is not the copy you built against, opens without complaint. The failure moves to symbol-binding time and reads differently:

/app/bin/myapp: symbol lookup error: /app/bin/myapp: undefined symbol: jv_free

There is no “cannot open shared object file” here, because nothing failed to open. Everything in this guide about search order still applies. The loader picked the FIRST match on the search path, and that match was the wrong build.

This is the common ending for a non-static autotools build that a build host also has an older packaged copy of. The build tree’s .libs/ is not on the search path, /usr/lib is, and the system copy wins. Name the sonames the object will accept, then check which file answers each one:

readelf -d ./myapp | grep NEEDED
LD_DEBUG=libs ./myapp 2>&1 | grep -i 'calling init\|trying file'

How the loader searches

The dynamic loader resolves each DT_NEEDED entry in this order:

1. DT_RPATH of the object holding the entry. This applies only when that object has no DT_RUNPATH
2. LD_LIBRARY_PATH
3. DT_RUNPATH of the object holding the entry
4. /etc/ld.so.cache
5. The default directories (/lib, /usr/lib, and the multiarch variants)

Inspect what a file carries:

readelf -d <file> | grep -E 'RPATH|RUNPATH|NEEDED'

$ORIGIN in an RPATH or RUNPATH expands to the directory of the object that carries it. That is what makes a tree relocatable: the path follows the file.

Three traps

1. RUNPATH is not transitive

DT_RUNPATH applies only to the direct dependencies of the object that carries it. It is not consulted when the loader resolves a dependency of a dependency.

DT_RPATH behaves differently. It is inherited down the chain. Modern linkers emit RUNPATH by default (--enable-new-dtags), so the inherited behaviour is the one you no longer get.

The consequence: an executable with RUNPATH=$ORIGIN/lib resolves its own direct dependencies from lib/. It does not resolve a library that one of those dependencies needs. If that second-level library sits in the same lib/ directory, it still fails, and the neighbouring files load correctly. That asymmetry is the signature of this trap.

To get inherited behaviour, link with -Wl,--disable-new-dtags. To keep RUNPATH, give every shared library its own $ORIGIN entry, not only the executable.

2. A copied file keeps the build machine’s path

CMake rewrites RPATH at install time for install(TARGETS) only. install(FILES) and install(DIRECTORY) copy bytes without touching the ELF header.

So a prebuilt vendor library, or a plugin built by a nested project and then copied in, arrives carrying whatever its own build produced. That is often an absolute path on the machine that built it:

RUNPATH: /home/builder/ws/src/app/linux/flutter/ephemeral

That path exists on the build machine, which is why the binary runs there. It does not exist on the device.

Setting CMAKE_INSTALL_RPATH does not reach these files. Nothing in the CMake install step does. Either fix the RPATH in the project that builds the library, or rewrite it while assembling the tree:

patchelf --set-rpath '$ORIGIN' <tree>/lib/*.so

3. A hook gets a cleared environment

atcd clears the environment for every lifecycle hook, then runs the hook through its shebang. A hook receives ATC_HOOK_CONTEXT, a pinned absolute PATH, the hook user’s HOME/USER/LOGNAME/SHELL, XDG_RUNTIME_DIR, and DBUS_SESSION_BUS_ADDRESS. It receives nothing else.

You cannot restore LD_LIBRARY_PATH through --hook-env. The boundary rejects the whole LD_* family, along with GCONV_PATH, LOCPATH, and GLIBC_TUNABLES (crates/atc-lifecycle/src/hook_env.rs). Those variables make the loader run code before the hook’s first line, so the denylist is deliberate and stays.

A hook may still export the variable inside its own body. The denylist governs what an operator passes in, not what the script sets for itself.

A long-running workload is a separate case. A systemd --user service starts after the hook exits, outside atcd’s spawn, so it inherits neither the hook environment nor ATC_HOOK_CONTEXT. Such a unit must establish its own environment, or run a binary that needs none.

If you deliver a ROS install space

catkin and colcon setup.bash export LD_LIBRARY_PATH=<install>/lib. That one directory is all they add.

A package executable installs to lib/<pkg>/. Anything that ships its own private library directory, such as a Qt or Flutter application bundle or a vendored SDK, lands deeper, at lib/<pkg>/<app>/lib. setup.bash never adds that path, so those libraries resolve only through the RPATH of the object that needs them. Combine that with trap 2 and the failure is silent until the first delivery.

Sourcing setup.bash is the normal ROS workflow, so a developer works inside an environment that hides the problem. Nothing in the ordinary build-and-run loop tests the unsourced case.

colcon bakes the build path into the install space

A colcon install space does not relocate on its own. install/setup.sh, install/local_setup.sh, and each package’s share/<pkg>/package.sh default COLCON_CURRENT_PREFIX to the absolute path the install space was built at. Deliver that tree to a different path and the scripts still resolve packages against the build machine’s directory, which does not exist on the robot.

The failure does not name a shared library, so it does not read like anything else in this guide. Packages simply do not resolve, and LD_LIBRARY_PATH stays unset or wrong.

Export the new prefix before you source, in the wrapper script the bundle ships:

#!/usr/bin/env bash
set -e
THIS_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export COLCON_CURRENT_PREFIX="$THIS_DIR/install"
source "$THIS_DIR/install/setup.bash"
set -uo pipefail          # only AFTER the source
exec ros2 run <pkg> <executable> "$@"

set -u goes after the source, never before it. ROS 2’s setup.bash reads AMENT_TRACE_SETUP_FILES without setting it, so under nounset the wrapper dies before it reaches your program. docs/operating/atc-toml.md carries the same rule for a wrapper that sources someone else’s environment.

$THIS_DIR resolves at run time, so the wrapper works at whatever path the delivery lands on. This is the same reasoning as an $ORIGIN rpath, applied to a shell script rather than to an ELF.

Delivered trees are read-only, so the wrapper cannot rewrite the install space in place at first start. The export is the only route.

Stop it at the gate: an install.exit hook

Verify the tree on the device, before the version pointer flips.

Use install.exit, not check.*. check is an observation-only probe that runs before any byte is transferred, so the incoming tree does not exist yet. By install.exit a bundle’s bytes are in versions/<sha>, the flip did not occur yet, and a non-zero exit vetoes the delivery. The device keeps running the previous version.

scripts/verify_libs.sh, inside the bundle:

#!/bin/bash
# install.exit — refuse a tree whose binaries cannot resolve their libraries.
set -euo pipefail

# Every tool this gate depends on, checked before anything is judged. A gate
# that cannot run must veto the delivery, never approve it. Absent `file`, the
# find predicate matches nothing and the script would exit 0 over an unscanned
# tree; absent `ldd`, every ELF would read as resolved.
for tool in python3 find file readelf ldd; do
    command -v "$tool" > /dev/null 2>&1 ||
        { echo "verify_libs.sh: $tool not found on device" >&2; exit 1; }
done

root="$(printf '%s' "${ATC_HOOK_CONTEXT:-}" |
  python3 -c 'import json,sys; print((json.load(sys.stdin).get("self_root") or {}).get("new") or "")')"
[ -n "$root" ] || { echo "no self_root.new in ATC_HOOK_CONTEXT" >&2; exit 1; }

# Sweep EVERY ELF file, not only the executable ones. A shared library is
# conventionally mode 0644, so a `-perm -u+x` filter would skip the exact file
# class this check exists to catch.
#
# The list goes through a file rather than a process substitution. A process
# substitution that fails only sends EOF, so the loop body never runs, `bad`
# stays 0, and a discovery failure reads exactly like a clean tree.
#
# NUL-separated, because a newline in a delivered file name would otherwise
# split one path into two names that resolve to nothing, and the tree would be
# refused for a fault it does not have.
list="$(mktemp)"
trap 'rm -f "$list"' EXIT
find "$root" -type f -print0 > "$list" ||
    { echo "could not scan $root" >&2; exit 1; }

bad=0
while IFS= read -r -d '' f; do
    # Classify HERE, not in a find predicate. As `-exec sh -c 'file -b "$1" |
    # grep -q ELF'` the pipeline reports grep's status, so a file that `file`
    # could not read looks the same as a file that is not an ELF: the predicate
    # is false either way, `-print0` is suppressed, and find still succeeds. The
    # unreadable file would leave the sweep in silence, which is the one outcome
    # a gate must never produce.
    #
    # Readability is tested directly rather than inferred. `file -b` prints
    # "regular file, no read permission" on STDOUT and still exits 0, so a
    # status check alone would let an unreadable file leave the sweep wearing
    # the same answer as a plain text file.
    if [ ! -r "$f" ]; then
        echo "cannot read $f" >&2
        bad=1
        continue
    fi
    if ! kind="$(file -b "$f" 2> /dev/null)"; then
        echo "could not classify $f" >&2
        bad=1
        continue
    fi
    case "$kind" in
        *ELF*) ;;
        *) continue ;;
    esac
    # Check the interpreter BEFORE calling ldd. `ldd` may execute an object that
    # names a non-standard one, which would run that code as the hook user. An
    # unexpected interpreter in a delivered tree is itself worth stopping on, so
    # refuse rather than skip: this is a gate, and it fails closed.
    #
    # A readelf FAILURE is judged apart from an empty result. Discarding its
    # stderr and reading the empty string as "no interpreter" would send an
    # object that could not be parsed straight on to `ldd`, which is the call
    # this ordering exists to protect.
    if ! interp_header="$(readelf -l "$f" 2> /dev/null)"; then
        echo "could not read the program headers of $f" >&2
        bad=1
        continue
    fi
    interp="$(printf '%s\n' "$interp_header" |
              sed -n 's/.*program interpreter: \([^]]*\)\]/\1/p')"
    case "$interp" in
        # An empty value here is a genuine absence, not a failure: a shared
        # library and a static binary both carry no PT_INTERP.
        '' | /lib/ld-linux*| /lib64/ld-linux* | /lib/ld-musl*) ;;
        *)
            echo "$f names a non-standard interpreter: $interp" >&2
            bad=1
            continue
            ;;
    esac
    # `ldd` exits non-zero on a static object, which is a normal result. Every
    # other non-zero exit means the file was never inspected, so it counts
    # against the tree instead of passing as resolved.
    ldd_out="$(ldd "$f" 2>&1)" && ldd_status=0 || ldd_status=$?
    if [ "$ldd_status" -ne 0 ]; then
        case "$ldd_out" in
            *"not a dynamic executable"*) continue ;;
            *)
                echo "could not inspect $f: $ldd_out" >&2
                bad=1
                continue
                ;;
        esac
    fi
    if printf '%s\n' "$ldd_out" | grep -q 'not found'; then
        echo "unresolved libraries in $f" >&2
        printf '%s\n' "$ldd_out" | grep 'not found' | sed 's/^/    /' >&2
        bad=1
    fi
done < "$list"

[ "$bad" -eq 0 ] || { echo "refusing the delivery" >&2; exit 1; }
echo "every ELF file resolves its libraries"

Declare it in atc.toml:

[[bundle]]
name = "app"
source = "install"

[bundle.lifecycle.hook.install.exit]
script = "@self/scripts/verify_libs.sh"
timeout_secs = 120

Three limits to understand before you rely on it.

It measures the device’s environment, not the workload’s. A container changes the mount point, the base image, and /etc/ld.so.cache. A binary that resolves on the host can still fail inside the container. Where the workload runs in a container, run the same ldd sweep in that container instead.

ldd reports what the loader would do for that object alone. It obeys the same non-transitive RUNPATH rule described above. That is what makes it a faithful check, not an optimistic one.

ldd can execute what it inspects. Some versions run the program to collect the dependency list. That runs the code of the ELF interpreter the object names, whenever the object names one other than ld-linux.so. ldd(1) states this and tells you never to run it on an untrusted executable. The hook runs as the hook user on the device, so treat a vendor blob you did not build as untrusted.

The recipe above reads the interpreter first. It refuses any object that names an unexpected one. That guard is what lets it call ldd on a tree it did not build.

For an object you do not trust, parse instead of resolve:

objdump -p <file> | grep NEEDED
readelf -d <file> | grep -E 'NEEDED|RPATH|RUNPATH'

Both read the file and run nothing. They report direct dependencies only, so they answer a narrower question than ldd. Walk the tree yourself, or use a parse-only resolver such as lddtree from pax-utils, when you need the whole dependency graph from untrusted input.

Diagnostic checklist

Run every one of these in the environment the workload runs in. A check on the build machine answers a different question.