atc.toml reference

atc.toml declares what you ship to a robot and how it behaves across the delivery lifecycle. One file feeds both delivery planes: atc ota push sends it over SSH, and atc ota publish --deliver and atc channel deliver embed the same hook tables in the signed manifest a polling device fetches for itself. This page documents the lifecycle-hook schema, because it is the part the parser validates most strictly and the part that has caused the most field confusion.

The schema below is derived from the parser (crates/atc-lifecycle/src/config.rs), which is the source of truth. If this doc and the parser ever disagree, the parser wins — file a docs bug.

There is no [[module]] table. A repo declares modules in atc.kdl, and no TOML syntax reaches one. What a declared module can and cannot do is mapped, with citations, in modules-today.md; a working repo is examples/heartbeat-module/.

Which file atc reads. With no -c, atc reads the nearest atc.toml or atc.kdl in the current directory or a parent. When one directory holds both, atc reads atc.kdl and prints a note that names the atc.toml it did not read. Set ATC_KDL_CONFIG=0 to read the atc.toml instead. The full rule is in Where atc finds the file.

What a bundle’s source ships

[[bundle]]
name = "myapp"
source = "dist"

source names a directory. atc ships every file and every resolvable symlink under it, subdirectories included, at any depth. The two exceptions are the bundle’s exclude patterns and Python bytecode, both described below. Nothing declares that layout, and every push reads the directory again.

Three things the walk does not carry:

• An empty directory has no entry of its own, because a directory is implied by the paths under it.
• A FIFO, socket or device node is skipped, and the push still succeeds. A bundle that needs one must have a hook create it on the robot.
• A dangling symlink is a hard error, not a skip. The push stops with ATC-E0510 and names the link, because a broken link shipped verbatim fails confusingly on the device. An unbuilt node_modules/.bin is the usual source. If the robot does not need that directory, add it to exclude.

Leave paths out with exclude

[[bundle]]
name = "myapp"
source = "dist"
exclude = ["**/.git", "**/*.log", "tmp"]

Each pattern is a glob that matches the path relative to source. * stays inside one directory, so *.log matches only at the top of source. ** crosses directories, so **/*.log matches at any depth. A directory that matches is left out with everything under it, and the walk does not enter it. So a dangling symlink inside an excluded directory does not stop the push.

A pattern that can never match stops the parse with ATC-E0531. That is an empty pattern, a leading or trailing /, a . or .. segment such as ./build, or a glob that does not compile.

Python bytecode is left out by default

The push leaves out every __pycache__ directory and every .pyc and .pyo file. The robot cannot use them. The push does not keep a file’s modification time, and CPython checks a timestamp-based .pyc against the time of its source. So on the robot CPython rejects the .pyc and compiles the source again. On a read-only store it cannot write the result either.

When the default left something out, atc ota push prints one line:

atc: note: bundle web: left out 2 Python bytecode files. Set `ship_python_bytecode = true` to ship them.

Set ship_python_bytecode = true on the bundle only if you build hash-checked bytecode (python -m compileall --invalidation-mode checked-hash). CPython accepts that form whatever the file times are.

In atc.kdl, both keys are children of the bundle node:

bundle myapp {
    source "dist"
    exclude "**/.git" "**/*.log"
    ship-python-bytecode #true
}

On any other artifact kind, loading the file fails with ATC-E0776.

Point source at a clean build output

Apart from exclude and the bytecode default, nothing filters the tree. So a directory you also work in ships what your tools left there. A checkout used as source ships its whole .git history, and a test run leaves .pytest_cache.

Nothing fails on the robot. The push still sends the bytes, and the store holds them for every version it keeps.

atc ota push names what it found before it uploads:

atc: warning: bundle web: 2 staged files are in a tool cache: .pytest_cache/v/cache/nodeids, .pytest_cache/v/cache/lastfailed.
atc:   The robot never reads a tool cache. The store holds these bytes for every version it keeps.
atc:   Point `source` at a clean build output. Delete the cache directory before the push.

It reports two kinds: a tool cache (.mypy_cache, .pytest_cache, .ruff_cache, .tox, node_modules/.cache) and a .git repository. Each warning names up to three paths and counts the rest. The push continues: a bundle that ships one of these on purpose is legitimate, and a refusal would break it. To leave one out for good, add it to exclude.

So reorganizing inside source costs no atc.toml edit. A flat dist/myapp that becomes dist/bin/myapp plus dist/share/VERSION needs no change here, and the next push carries both new directories. atc ota push --dry-run previews the upload size per artifact, though not the paths it would send.

The dist directory that excludes itself

A build that stages into ./dist/ usually copies the project tree first. An unanchored exclude then removes more than the staging directory:

rsync -a --exclude=dist ./ ./dist/     # WRONG: drops every nested dist/
rsync -a --exclude=/dist ./ ./dist/    # anchored: only the top-level one

rsync matches an unanchored pattern against every path component, so --exclude=dist also drops node_modules/<pkg>/dist/. Several npm packages keep their runtime files there. The bundle still validates and still pushes, and the program fails on the device with a missing module.

Anchor the pattern with a leading /, or stage into a directory whose name appears nowhere else in the tree.

A binary that reads files beside itself

A delivery path carries a content hash, so it changes on every version. A bundle that ships bin/ and share/ together must resolve share/ from the directory of its own executable, and must ask the platform for that directory:

Do not assemble the path by string concatenation, such as exe + "/../share". That form breaks as soon as a caller reaches the program through a symlink or a relative path. It breaks on the device, not on the build machine.

The shell row resolves the link first. $0 is the path the caller typed, so a wrapper reached through a convenience symlink would otherwise look for share/ beside the LINK. readlink -f also resolves delivery/current. That lands on the version directory the program runs from, where bin/ and share/ are siblings.

A hook is a different case. A hook runs in a sibling of delivery/, so it must read current_root out of ATC_HOOK_CONTEXT. See “current_root: the path a service unit must name” below.

A third-party binary whose version you do not own

A cloned upstream project bakes its own version into its --version output, and a pin to one commit cannot change it. The bundle version and the program version are two labels, and atc owns only the first.

atc ota push --version v1.0.0 sets the bundle version. That is the label atc ota ls reports and atc ota switch takes, and it is independent of what the shipped program prints.

To report it from the bundle as well, ship a VERSION file beside the binary and a wrapper that reads it:

dist/
  bin/jq             # the upstream binary, untouched
  bin/jq-wrapped     # the wrapper below
  share/VERSION      # v1.0.0, written by your build

#!/bin/sh
# bin/jq-wrapped - report the bundle version, then hand off to upstream.
set -eu
self=$(readlink -f -- "$0")
here=$(cd -P -- "$(dirname -- "$self")" && pwd)
if [ "${1-}" = "--bundle-version" ]; then
  cat "$here/../share/VERSION"
  exit 0
fi
exec "$here/jq" "$@"

Write share/VERSION from the build step that passes --version to atc ota push, so the two labels cannot drift.

Shipping a compiled binary

This section is about compiled binaries. A node or python bundle ships source, so skip it.

Run file on the artifact, then ldd. An autotools or libtool build writes a wrapper shell script under the binary’s own name, and leaves the real ELF under .libs/. ldd tells you nothing useful about that wrapper, which is why file comes first. Configure with --disable-shared to get one self-contained binary instead.

ldd then lists what the binary loads at start. The robot must carry every library it names, or the process dies with error while loading shared libraries.

A second failure reads differently. A library that resolves by soname, but is not the one you built against, loads without complaint. The process then dies at symbol-binding time:

/app/bin/myapp: symbol lookup error: /app/bin/myapp: undefined symbol: jv_free

The file was found, so the “cannot open shared object file” message never appears. A non-static autotools build reaches this often: the real ELF under .libs/ links the soname, and the build host already carries an older copy of it. The binary runs on that host and dies on the robot, or the reverse. readelf -d <file> | grep NEEDED names the sonames it will accept.

The toolchain decides how you link statically, and no flag is a guarantee. For autotools, configure with --disable-shared --enable-static. That governs the project’s own libraries, not libc, so add LDFLAGS=-static when you want libc in the binary too. For Go, build with CGO_ENABLED=0 go build. A plain go build looks static and often is not, because the standard library pulls cgo in for DNS lookups and for os/user. For Rust, build for a musl target.

Check the result rather than trusting the flag. file should say statically linked, and ldd should say not a dynamic executable.

The glibc risk runs one way. glibc keeps older binaries working, so a binary built against an older glibc than the robot’s runs there. Only the reverse needs a static link or vendored libraries: a newer build machine, an older robot. Run ldd --version on both machines to see which case you have, rather than deploying to find out.

atc ota push reads the staged binaries. It refuses a binary that binds a newer glibc than the robot reports, and it warns about a libtool wrapper script. It says nothing about shared libraries, because a dynamic link is the right answer when the robot carries them.

Loader search order, $ORIGIN rpaths, and an install.exit hook that refuses a tree whose binaries cannot resolve: shared-library-resolution.md.

A device with several hosts

A robot can carry several computers, each with its own atcd, for example a nav computer and a compute computer. Each computer is a host, and its role is its job in the robot. Set a host’s role when you install atcd: atc bootstrap atcd install --role nav.

Give each artifact for one role a role. An artifact with no role goes to every host. The [device.hosts] table maps each role to the SSH target of its host:

[[bundle]]
name = "planner"
source = "dist/planner"
role = "nav"

[[oci]]
name = "team/vision"
platform = "linux/amd64"
role = "compute"

[[bundle]]
name = "common"
source = "dist/common"

[device.hosts]
nav = "ops@nav-14"
compute = "ops@compute-14"

A role is 1 to 63 of a-z, 0-9, - and _, and starts with a letter or a digit. atcd refuses an artifact whose role is not its own.

atc ota push --device --version v2 moves every host to v2 as one cutover:

1. It reads each host, and refuses a host whose role is not its key in [device.hosts]. A host that cannot be reached stops the push before any host changes.
2. It pushes v2 to each host with no switch, with only the artifacts for that host’s role and the artifacts with no role.
3. It switches every host to v2 at the same time. Each switch runs the activate and commit hooks.
4. If one host fails, or does not finish its switch within --health-timeout (default 15m), the push rolls back. Every host that runs v2 switches back to the version that every host ran before. A new device, whose hosts share no earlier version, has nothing to roll back to. The error then names each host that runs v2.

--device needs --version or -t, because a second run finds the cutover by its label. For the same reason it refuses --overwrite: push changed content under a new label. The push keeps no state. If it stops in the middle, run the same command again: it reads each host and finishes the cutover. A host that cannot go back to the old version is named in the error, with the atc ota switch command that moves it.

A host that reaches the robot through a gateway needs a ProxyJump entry in your ssh_config. Only a host whose atcd reports the device-cutover-v1 capability can take part. An older atcd is refused with a message to upgrade it. atc.kdl spells an artifact role (role "nav") and this table (device { hosts { nav "ops@nav-14" } }), so atc ota push --device works from either file.

Files signed with each version: attachments

A [[bundle]] or [[oci]] table can name files to keep with each version that atc ota publish uploads. Examples are a wiring diagram or a test report. Devices never download them.

[[bundle]]
name = "field-config"
source = "dist"
attachments = ["docs/wiring.txt", "reports/bench.txt"]

• Each path is relative to the manifest and stays inside its directory: no absolute path and no ... The file is stored under its file name, which must be 1 to 128 of A-Z a-z 0-9 . _ - and start with a letter or a digit. Two files with one name are refused.
• atc ota publish reads each file before it uploads anything. It refuses a file over 256 KiB, and more than 32 files for one artifact.
• The hash of each file is signed into the version’s metadata, so it is part of the version’s identity. A changed file is a changed version, and needs a new --version.
• The publish uploads each file after it creates the version. The server takes only bytes with the signed hash, and nobody can delete the file. A rerun uploads the files again, so it finishes an interrupted publish.
• A delivery does not carry the hashes, so devices see no change.
• atc ota attach ls --version <artifact>@<version> lists the files. A signed name that has no upload yet shows not uploaded.

atc.kdl has no spelling for attachments yet, so a manifest that uses them stays atc.toml.

[ownership]

atc ota publish --deliver reads this table. One manifest owns the delivery config of a server channel: its hooks, workloads, peripherals, state_dirs and member roles. Another manifest that ships artifacts into that channel defers to the owner. The ownership block of atc.kdl is the same table.

[ownership]
id = "robot-base"
owns = ["stable", "beta"]
defers = { canary = "robot-core" }

The channels are server channels, the --channel of atc ota publish. They are not the [channel] table that atc ota push targets. An id and an owner id follow the rules of an artifact name. A file with no [ownership] table makes no claim.

atc ota push, atc ota publish and atc config lint refuse these tables:

atc config lint also warns when owns names a channel more than once (ATC-E0588). That changes no delivery.

The manifest does not list its server channels. So the lint cannot find a channel name that the org does not have. atc channel owner <channel> shows the owner that the server keeps for a channel. Release through atc-server tells what the server does with the claim.

Lifecycle hooks

A lifecycle hook runs a script on the robot at a specific (state, phase) during a delivery. The table key is exactly:

[<scope>.lifecycle.hook.<state>.<phase>]

The .hook. segment is required. Omitting it (e.g. [lifecycle.commit.exit]) is rejected at push time — it does not silently wire nothing.

Body keys

timeout_secs = 0 is push-only. A signed (pull-plane) delivery carrying it is refused on the device — the daemon stays on the prior delivery and reports hook_config_invalid — because an unbounded pre-flip hook wedges the poller with no retry budget to fall back on.

No other keys are allowed in a hook body. A stray run / command / exec (or any other key) is rejected. A [delivery.hook.*] or [lifecycle.hook.gc.*] bookend takes script and timeout_secs only.

A hook script that the artifact installs

If the artifact can carry the script in its own tree, use @self instead. Before the install, a @self hook is the active version’s own hook, and a first delivery runs none (Which version’s script @self runs). This section is for a script that must live at a fixed system path.

Sometimes the artifact itself installs the script its hook runs. For example, a .deb installs /usr/lib/agent/pre-download.sh, and the download.enter hook runs that path. On the first delivery to a device the file does not exist yet. Without a policy, the hook fails with ENOENT and atcd refuses the delivery that would install the script.

Set allow_missing_initial = true for this case:

[[bundle]]
name = "agent"
source = "dist"

[bundle.lifecycle.hook.download.enter]
script = "/usr/lib/agent/pre-download.sh"
allow_missing_initial = true

The first delivery of agent to a device skips the hook. Each later delivery runs it. If the script is missing on a later delivery, the hook fails the delivery as usual, because the package that owns it is already installed.

Set allow_missing = true to skip a missing script on every delivery.

The rules:

• Set one key or the other. A hook that sets both is refused, even when one of them is false.
• Both keys need an absolute script. On @self/… or @<name>/… they are refused, because atc places those trees itself.
• Only a missing file is skipped. A script that exists but has no +x, a path that escapes its allowed root, and every other pre-spawn check still fail the hook.
• “No earlier version” means the device has no prior version of this artifact. The uninstall gate and atc ota run-hook are not deliveries, so allow_missing_initial never skips there. allow_missing does.
• A [[peripheral]] hook refuses both keys. A peripheral hook is the flash, and a skipped flash would still record the peripheral as flashed.
• A symlink at the path whose target does not exist is not a missing script. It fails the hook.
• A stage hook (top-level [lifecycle.hook.*]) takes allow_missing only. It belongs to no one artifact, so it has no first delivery.
• A skipped hook passes its gate. The push output prints a → <state>.<phase> <script> skipped (script missing) line for it. atc ota status lists it in a Skipped Hooks table, and the lifecycle_log rows of atc ota status --json show the outcome skipped_missing. The lifecycle.hook span carries outcome="skipped_missing".
• A device whose atcd predates these keys ignores them, and a missing script fails the hook as before.

States and phases

enter refuses, exit rolls back. On a push, on a pulled delivery and on an atc ota switch, a hook that exits nonzero at enter refuses the step. The state’s work never started, so atc undoes nothing. It runs no rollback.*, no failure.*, and no activate.* for the outgoing version. current does not move, and the push or switch reports refused, the device did not change. A hook that exits nonzero at exit answers after the work, so atc rolls the delivery back.

commit.enter is the one exception. It runs after the pointer flip, so a refusal there rolls back like an exit refusal. delivery.pre refuses with no ladder, as before. A switch crosses two of these gates, activate.enter and deactivate.enter. See Which of these run on an atc ota switch.

A refusal is clean only when every hook that ran at that gate exited nonzero. Some gates run more than one hook, for example a deactivate.enter with a stage hook and one hook per member. If one of them passed, it may have done its work, so atc rolls back as before. A hook that times out, is killed, or cannot start also rolls back, because it may have stopped part way.

A stage hook that refuses at activate.enter or deactivate.enter stops that gate, on a push and on a pulled delivery. No artifact hook runs beside it at that gate. A stage hook that exits nonzero there stays a clean refusal. One that times out, is killed, or cannot start still rolls back. At deactivate.enter the outgoing version’s hooks do not run, so atc records no teardown. Before issue #7215 a push still ran the artifact hooks at both gates.

Write an enter hook so that it changes nothing when it exits nonzero. Its refusal tells atc that the device is as it was.

enter and exit are the only two. There is no error phase to declare — nothing dispatches one, at any state, so <state>.error is rejected at push.

The authoritative list is atc_lifecycle::State::ALL (crates/atc-lifecycle/src/state.rs).

The order they run in

check.enter → check.exit
  → download.enter → download.exit
  → install.enter  → install.exit
  → activate.enter                    (may the new version go live?)
      → deactivate.enter              (the outgoing version stands down)
      → deactivate.exit
      → [the current pointer flips]
  → activate.exit                     (the new version confirms it is live)
  → commit.enter → commit.exit

deactivate runs nested inside activate, which puts every step that can fail ahead of the one destructive step. A veto at activate.enter has torn nothing down.

A name shared across versions

Read this order before you write a hook that claims a name the device holds only once. Examples: a systemd unit name, a TCP port, a lock file, a PID file. Every version claims the same name. The order above decides who holds it at the end.

Start the service at activate.exit. Never at activate.enter. activate.enter runs before the outgoing version’s deactivate.*. A start placed there is stopped again by the version it replaces. The push then reports success with the device on the new version and the service down. activate.exit runs after both deactivate gates, so the order pays the stop back.

Make the start a restart, not an enable --now. deactivate.* reads the OUTGOING version’s declaration. So the hook that stops the service is the one already on the device. Add the pair in v2 and push it over a v1 that declares none, and nothing stops v1. systemctl --user enable --now then exits 0 against a unit that is already active, does not restart it, and leaves v1’s process holding the name. The delivery commits green, atc ota status names v2, and the robot runs v1. A first deploy dispatches no deactivate.* at all, because there is no outgoing version to tear down.

Stop the service at deactivate.enter or deactivate.exit. Both run before the flip, and the incoming activate.exit runs after them. So a stop there needs no test for “am I still the current version”. The order answers that question.

To ask “may this version stand down now?”, put the stop at deactivate.exit. Then deactivate.enter is free for that check, and it runs before anything stops.

Know what a refusal there does. On a push, on a pulled delivery and on a switch, a refused deactivate.enter leaves the device as it was: the outgoing service keeps running, and nothing restarts it. A push does not try again, so push again when the device is ready. A poll tries again on the next poll. To ask the poller to retry sooner or later, exit 21 instead; see How long a pulled delivery defers.

A refused deactivate.exit starts the unwind ladder, because the stop already ran. The ladder runs the outgoing version’s activate.* again to bring the service back.

A worked pair of hooks is in Worked example: a long-running service.

The unwind track is rollback.enter → rollback.exit, and failure.* after that if a rollback hook itself fails.

A rollback restores the delivery that the vetoed one displaced. A first delivery displaced nothing, so a veto after its flip removes delivery/current instead. The push footer then reports “no earlier delivery to restore”, and a service unit that starts from that pointer fails until the next successful push. Give such a unit a ConditionPathExists= line; see install_unit.sh in the worked example.

The pointer flip used to happen between commit.enter and commit.exit. If you have existing commit.* hooks, read the hook cutover runbook: they all still run, but three gates changed which side of the flip they sit on.

uninstall.* is not on this graph at all. The periodic housekeeping sweep and atc ota rm both dispatch uninstall.enter then uninstall.exit for each version they are about to reclaim, and delete only if both return 0. Both run before the deletion, because an uninstall.exit script lives in the tree that is about to go.

Which of these run on a pulled delivery

atc ota publish --deliver and atc channel deliver --lifecycle-config author the same three hook tables atc ota push does (artifact, stage, delivery) from the same atc.toml, using the same parser, and sign them into the manifest. A polling device re-validates those bytes against the manifest signature before it runs anything, but the declaration it runs is yours, unchanged.

All fourteen forward cells now run on a pulled delivery. A pulled delivery walks delivery.pre, then per member check.enter → check.exit → download.enter → download.exit → install.enter → install.exit, then activate.enter → deactivate.enter → deactivate.exit → [flip] → activate.exit → commit.enter → commit.exit → delivery.post. That is push’s order and the whole FORWARD pull walk. The unwind ladder is a separate walk. It is not narrower either.

Read this before you publish to a pull-fed fleet. Until now those six cells were accepted, signed, delivered, and never run. Your channel may already publish a check.*, download.* or install.* hook. A device that upgrades past this change starts running it. Check what those scripts do first.

@self resolves the same way on a pull as on a push. At check.*, download.* and install.enter the hook is the ACTIVE version’s own, with that version’s atc.toml entry and script. From install.exit on, it is the incoming version’s (Which version’s script @self runs).

check.*, download.* and install.* cross one gate per member, on both planes. They have no delivery-wide barrier, so they run at artifact scope only. A top-level [lifecycle.hook.check.enter] table is refused at publish, and a device that somehow receives one warns about it on atcd.pull.undispatched_hook_total. Every other state runs at both scopes.

A member the device already holds crosses check.* and neither of the other two. It fetches nothing and places nothing, so a download.* there would say bytes arrived when none did. Push makes the same skip, member by member, for a version the device holds when that member’s check.exit runs, active or not (#7217). The push transfer itself does not change.

delivery.pre now runs before any byte is fetched. That is where push runs it. On this plane it used to run after the poller staged every member. Two things follow.

A hook that read the incoming version’s files in delivery.pre no longer finds them. Move that work to install.exit or activate.enter.

A poll whose download then FAILS has run delivery.pre already. So a channel stuck on an unfetchable blob re-runs the hook at poll cadence. The same failure on a push costs one operator-driven attempt.

The delivery bookend closes on every ending that ends the walk. delivery.post runs at the end of a pulled delivery whose delivery.pre passed. That holds for a staging failure, for a refusal before the flip, for a refusal after it, and for a spent retry budget. Push pairs the two bookends the same way. A delivery.pre that refuses or defers opens nothing, so the poller runs no delivery.post for it.

One case is neither. A poll superseded between its hooks and its commit leaves the delivery RESUMABLE, so it runs no delivery.post of its own. The poll that finishes the delivery runs it. If that delivery is never advertised again, its delivery.pre stays unpaired.

A deferred poll is not an ending, so it runs no delivery.post either. The poller has a retry scheduled for the same delivery. So a hook that exits 21 at activate.enter, after delivery.pre passed, leaves that delivery.pre unpaired for that poll. The poll that ends the delivery closes the bookend: the commit, a refusal, or an operator-set retry cap that runs out. Do not read the pair as 1:1 across a chain of deferred polls.

A delivery.post that refuses still runs once. The bookend that reports a failure can be the failing hook. The poller records that refusal, keeps the reason the walk already reported, and starts no second unwind.

What a refusing hook costs is not identical on the two planes. A hook that refuses after the flip un-flips the delivery on both. The prior delivery becomes the active one again, and the incoming version’s modules are drained.

The compensation ladder then runs, on all three planes. It dispatches rollback.enter, then the compensating activate.* for the outgoing version, then rollback.exit. If a gate refuses, the compensating activate.* does not bring the outgoing version back, or the take-back cannot restore the prior delivery, it also dispatches failure.enter and failure.exit. The compensating activate.* runs no stage-scope hook. So if a stage deactivate.* stopped the workload and no forward stage activate.* passed after it, the workload stays down, and failure.* follows. The ladder’s own cells honour no --skip-hook and no --hook-env: an operator flag may not suppress a recovery hook.

If the compensating activate.enter refuses, its activate.exit does not run. That is the rule for every refused enter gate (#6389): the state’s work did not start, so its exit has nothing to close. It is a refusal only when every hook that ran exited non-zero, and every member that deactivate.* stopped ran an activate.enter hook. Otherwise activate.exit still runs. A member with no activate.enter did not refuse, and its activate.exit may be what restarts it. After a timeout, a signal or a dispatch fault, the device state is not known.

A ladder that completes ends the delivery rolled_back. A ladder that does not complete ends it failed, and failure.* runs. Each member’s push row (pushes) takes the same status. This applies to a push whose pointer flip landed, and to a pull whose flip landed. It also applies to a push or a pull that a hook refused before the flip, when the ladder brought back members that deactivate.* stopped. A push that fails before any teardown has nothing to roll back, so it ends failed. On a pull, a compensation before the flip that does not bring the outgoing version back runs failure.*. If a member of that version declares no activate.*, nothing can bring it back, so the delivery ends failed and the channel accepts the next pulled delivery (#8055). Otherwise, for example after a refusal by the outgoing version’s own activate.enter, the delivery stays in_progress, and the next poll or the next start tries again.

Before #6750 and #6939, a ladder whose hooks refused could still report rolled_back, and a clean ladder before the flip reported failed. Before #6977, a pull ladder ran no compensating activate.* after the flip, and a dirty pull ladder before the flip still ended rolled_back.

The compensating activate.* names the version it restores as version.new_sha. So a hook that pins or marks new_sha acts on the version that comes back. On the push ladder version.old_sha is the failed version. A pull compensation, a switch unwind and a boot recovery name the restored version in both fields. So do not tear down old_sha unless it differs from new_sha. On rollback.* and failure.*, new_sha is still the failed incoming version.

An unwind hook cannot stop the unwind. A refusal at any of those four gates is recorded and the take-back continues, because the ladder is already the failure path. This differs from a forward gate, where a refusal is a veto.

Before #3298 the poller dispatched none of the ladder, so a rollback.* hook on a pull-fed device parsed, published without error, and never fired.

A hook that refuses before the flip at an exit gate is compensated on both planes, in the poll or push that heard the refusal. If the outgoing version was already torn down, the poller re-runs that version’s activate.* to bring the workload back, under the auto_rollback_veto trigger. A refused enter gate owes no compensation, because nothing was torn down (#6389).

Nothing bounds a permanent pre-flip refusal on the pull plane. A veto writes no retry-budget row, so the next poll dispatches the whole pre-flip half again. A deactivate.exit hook that refuses forever therefore stops and starts the outgoing version once per poll. Each of those polls also runs that version’s two activate gates. A deactivate.enter hook that refuses forever stops nothing: the poll ends at the refusal. A defer (exit 21) is not bounded either, by design (#6431): see How long a pulled delivery defers.

No forward state runs on one plane only. check.*, download.* and install.* used to. The poller fetched and staged bytes without walking them, so such a hook parsed, published without error, and never fired. It fires now (#3412 §1). See Which of these run on a pulled delivery for what changes on a fleet that already publishes one.

Read your peripheral hooks before you upgrade a pulling fleet. An unattended device on a firmware channel writes to hardware on a poll. No operator is present. ATC_HOOK_CONTEXT.trigger reads pull on those runs. It read push on every peripheral hook before this change, on both planes.

Peripheral firmware converges by pull. A bundle a peripheral subscribes to used to be refused at the commit with ATC-E0156. That refusal told the operator to use atc ota push. The poller now runs the flash ladder push runs. It admits the firmware-payload.toml and parks every subscribed peripheral BEFORE the version flip. Then it runs each peripheral’s commit.exit and records last_flashed_sha after it. A flash hook that refuses before the flip leaves the device on the prior version.

A peripheral subscription is device-local and names no channel. The ladder used to read subscriptions in the channel the delivery lands on. Every operator route writes the seeded local/default channel, so the read returned no rows on any other channel. No pulled delivery flashed anything. Every subscription read now keys on the peripheral and the artifact. So the ladder finds the operator’s subscription whichever channel the delivery lands on. atc ota peripheral subscribe and atc ota peripheral apply still take no channel argument.

A peripheral hook must be idempotent. A poll that flashes and then loses its commit re-flashes on the next poll. A re-poll that skipped the flash would flip the pointer over hardware still holding the prior firmware. A crash between the two does the same. Push has the same rule for the same reason: a re-push of a member that is not yet active re-runs the ladder. Write the flash so that running it twice against one payload sha is the same as running it once.

The ladder runs only where the device can run it: pull_plane_hooks_enabled = true and a signature-verified release. Anything else refuses with ATC-E0156 and leaves the prior firmware live. That covers the flag off and an unsigned release. Pull never flips firmware it did not flash.

A release does NOT have to declare a lifecycle hook of its own. It used to. That was a limitation rather than a rule. A release can carry the [[peripheral]] declarations of its source file, hooks included, in the signed manifest. A release that carries none flashes with the device’s fallback: the [[peripheral]] blocks of atcd’s --config file, which atcd imports once into its store at the first start of this release. The old condition refused such a channel.

Such a release now converges. The device resolves the walk the ladder hangs off when a member of the release has a subscribed peripheral. It dispatches no delivery hook, because the release declares none.

That refusal could not fire on a pulled delivery while the subscription read was channel-scoped. It fires now. Such a delivery stops instead of flipping a subscribed member unflashed.

--skip-hook and --hook-env reach no pull cell, and that includes peripheral hooks. This is deliberate, not an omission. Both flags scope to one operator command: atc ota switch --skip-hook commit.exit skips that gate for that switch only. The next command runs the hook again. A poll has no operator on the other end. There is no invocation for an override to scope to.

A pull-plane override has two possible homes, and neither one is safe. In the signed manifest, a skip would let whoever holds the publish key disarm a device-owned safety gate. The device already refuses that shape of privilege for root hooks: the daemon.pull_plane_root_hooks setting in atcd’s store is device-owned for that reason, and the manifest cannot set it.

In atcd’s settings store, a skip would be a standing disarm that no fleet-side surface reports. A device-local --hook-env is worse. The daemon applies extra_env after every framework variable. A standing entry would therefore override ATC_TRIGGER for every later delivery on that device.

To stop hook dispatch on one pull-fed device, run atcd settings set daemon.pull_plane_hooks_enabled=false on the device. Read the comment on that key first. A device that ran with the key off can wedge a channel when you turn the key back on.

One smaller difference: timeout_secs = 0 is refused on the pull plane. See Body keys.

uninstall.* runs on both planes. A version carries its own declaration in an .atc/lifecycle.pb sidecar next to its files, and both planes write one. Push writes it at that artifact’s install.exit. The poller writes it when it commits the delivery that activates the version.

On a pulled version the poller writes a sidecar only when three things hold. The delivery is signature-verified, it carries hook declarations, and the device has pull-plane hooks enabled. An unsigned or legacy delivery declares no hooks at all, so there is nothing to write.

One further case has no sidecar even when all three hold. The server can ask a device to keep a version resident without activating it. Such a version gets its declaration only when a later delivery makes it active, so housekeeping reclaims it silently if it never becomes active.

Read the daemon log if a teardown does not run. The poller warns whenever it commits a delivery whose declared hook config it could not read back.

The sidecar feeds more than the sweep, so expect two other hooks to start running. A version’s own declaration is also what a later ota switch reads for its activate.* hooks, and what a push reads for the outgoing version’s deactivate.* hooks. Before this, a pulled version answered “no declaration” to both, and both were skipped. Now they run, and a deactivate.* hook that refuses can veto the push that displaces a pulled version. That is the intended behaviour on both planes. Check your deactivate.* and activate.* declarations before you publish to a pull-fed fleet.

How long a pulled delivery defers

A hook that exits 21 on a pulled delivery defers the activation. The device stays on its running delivery and reports deferred, which the server reads as in progress. A deferring device is not a failed one, so it does not count toward a rollout’s failure rate.

The poller tries again for as long as both of these stay true:

• the hook keeps exiting 21, and
• the server keeps the delivery as the device’s desired state.

It stops when the hook exits 0 (the delivery activates), or when the server advertises a different desired state. A refusal (a veto) ends the deferral too and reports failed, but a pre-flip veto is not final: the next poll runs the walk again. A new desired state is a new delivery, and the poller applies it at once. A deferral survives an atcd restart, because its state is in the device store.

A deactivate.* hook can stop the running workload before a later gate defers. The poller then starts that workload again in the same poll, so the device runs its prior delivery while it waits. The next attempt stops it again, once per attempt.

A deferring hook can set the wait. To do this, print a line retry-after: <seconds> on stdout before you exit 21, for example retry-after: 120. The key is case-insensitive, and the first such line counts. The poller waits that many seconds, plus up to 10 % jitter, but never longer than hook_retry_max_secs. A value of 0, or one that is not a whole number, is ignored. When several per-artifact hooks defer at the same gate, the largest value wins, so no hook runs again before the time it asked for.

Without a retry-after: line, each retry waits longer than the one before. The first wait is hook_retry_default_secs (default 30 s), and each later wait doubles it, up to hook_retry_max_secs (default 300 s). Every deferral counts as an attempt, with a hint or without one. So after a hinted deferral, the next deferral without a hint takes the next backoff step, not the first. Once the backoff reaches the ceiling, a deferral that lasts all day runs the hook about once every five minutes.

A retry that is due runs at the next check-in, so the wait is never shorter than the check-in interval.

Each retry walks the delivery again from the start. The hooks that run before activate.enter, such as download.exit and install.*, run again on every attempt, for as long as the deferral lasts. Write them so that a second run is safe.

The [daemon] keys hook_retry_max_attempts and hook_retry_max_total_secs are unset by default. Set one to make the device give up: when the cap runs out the poller stays on the running delivery, reports exhausted, and stops retrying that delivery. A cap of 0 gives up on the first defer; to remove a cap, unset the key. Before #6431 these caps defaulted to 10 attempts and 3600 s.

atcd exports the histogram atcd.pull.hook_defer_age (Prometheus: atcd_pull_hook_defer_age_seconds). It records one sample each time a deferral defers again, and the sample is how long that deferral has lasted. Its wait label tells what set the next wait: hint for a retry-after: line, backoff for no line.

Which of these run on an atc ota switch

A switch is a third walk. It re-points the device at a delivery already in the store, so it moves no bytes. atc ota switch <version> walks delivery.pre -> activate.enter -> deactivate.enter -> deactivate.exit -> [flip] -> activate.exit -> commit.enter -> commit.exit -> delivery.post. That walk drops one thing a push does, and it is a gap rather than a carve-out.

No check.*, download.* or install.*. Those three states put bytes on the device. A switch moves none, so it crosses none of them. This is the one walk that still omits them: the pull walk dispatches all six (#3412 §1).

A clean refusal at activate.enter or deactivate.enter leaves the device as it was, as on a push: no rollback.*, no failure.*, and no activate.* for the outgoing version. Any other refusal at activate.enter or after it enters the unwind ladder described above, in the same order push and pull run it. One case still unwinds: a clean deactivate.enter refusal whose teardown record merged into one an earlier switch left standing. That earlier switch still owes the outgoing workload a restart, so this refusal runs the ladder and tries the restart again. If the restart fails, the record stands for the next boot. A stage hook that refuses before the record is written leaves that earlier debt as it was. A switch does not run delivery.post after a refusal.

deactivate.* DOES run, from the outgoing version’s declaration. It did not until #3698. A switch owned no durable record of a teardown. So a daemon that died between the teardown and the flip left the outgoing workload stopped, with nothing able to start it again. The switch now writes that record before the first teardown hook. A boot re-runs the outgoing version’s activate.* from it. All three walks dispatch the same deactivate.* cells. They also dispatch the same rollback.* and failure.* cells: check.*, download.* and install.* stay the switch’s gap, for the reason above.

This changes what a device does. A fleet that switches versions starts running deactivate.* hooks that never ran there before. So does a rollback, because a rollback IS a switch. Read the deactivate.* declarations of the versions already on your fleet before you roll out an atcd carrying this.

A refused teardown VETOES the switch, and the device stays on the version it was already running. The switch un-does nothing, because it had not yet switched. A clean deactivate.enter refusal stops there. A refused deactivate.exit, or a deactivate.enter refusal that is not clean, runs the compensation: the outgoing version’s activate.* is dispatched again, so write both halves to tolerate running twice.

A version whose members declare no activate.* cannot be brought back that way. A teardown that refuses there leaves those workloads stopped until an operator pushes, and the reply says so by name.

A refused switch now runs your rollback.* scripts, and your failure.* scripts when the recovery itself fails. A clean activate.enter or deactivate.enter refusal runs neither. Those hooks ran on a push and on a poll and never on a switch, so a fleet that rolls back starts running a script that has never run there. Read them before you roll out an atcd carrying this. rollback.enter runs before the device is put back, rollback.exit after, and both run twice if you retry a switch that keeps refusing — write them to tolerate that. failure.enter and failure.exit fire only when the outgoing version’s workload is still stopped after the recovery ran, which is the one condition worth paging on.

Stage-scope hooks DO run, at each of the six gates. The scope table says a stage hook runs once per delivery, at that barrier. That holds on a switch as it does on a push and a poll. So a top-level [lifecycle.hook.activate.exit] runs on the switch that makes the delivery live. The switch reaches activate.enter, deactivate.enter, deactivate.exit, activate.exit, commit.enter and commit.exit. A stage hook at rollback or failure runs only if the switch enters that ladder — a refusal at activate.enter or after it, except a clean refusal at activate.enter or deactivate.enter — and the ladder’s cells honour neither --skip-hook nor --hook-env.

The stage arm changes what a device does too. A fleet that switches versions starts running stage hooks that never ran there before. Read your top-level [lifecycle.hook.*] tables as well. A non-zero exit at any of the six gates VETOES the switch. A stage hook that passed at activate.enter or deactivate.enter makes a later member refusal there not clean, so that refusal still unwinds. After the flip a veto un-switches the device back to the version it came from.

Only a delivery committed by an atcd new enough to archive the table has one to read. A delivery committed before that runs no stage hook on a switch until the next push or poll re-commits it.

atc ota switch -a <artifact> is a fourth walk, narrower again. It activates one artifact and not a delivery, so it mints a synthetic single-member delivery. The device must also be in per_artifact assignment mode to accept -a at all.

The -a form runs the per-artifact arm and nothing else. It walks activate.enter -> deactivate.enter -> deactivate.exit -> [flip] -> activate.exit -> commit.enter -> commit.exit. So it runs no delivery.pre, no delivery.post and no stage hook at any gate, at a forward gate or at an unwind cell. It crosses the same six gates the whole-delivery form crosses, and it walks the same unwind ladder when one of them refuses. Read every rule above as the <version> form’s rule. The -a form is the carve-out from the bookends and from the stage arm, and from those two only.

That is a property of the TARGET and not of the walk. The bookends and the stage arm both read the target delivery’s .atc/delivery.pb archive. No push or poll ever committed the synthetic delivery, so it has no archive, and it declares no delivery-scope or stage-scope hook. To read the originating delivery’s archive instead would run one delivery’s hooks over another delivery’s activation. The whole-delivery form runs both arms because its target IS a committed delivery.

deactivate.* is the arm the two forms share, and it did not run on -a until #4211. It reads the OUTGOING version’s own .atc/lifecycle.pb sidecar, which every version carries, so it is a property of the version being replaced and not of the synthetic target. Everything the deactivate.* note above says holds here: a refused teardown vetoes the activation, the device stays on the version it was already running, and a refusal that is not a clean deactivate.enter refusal dispatches the outgoing version’s activate.* again first. A daemon that dies between the teardown and the flip is covered by the same durable record. That record names the synthetic delivery as its target, so a boot that finds the pointer already on it reads the walk as finished and clears the record.

This changes what a device does. A fleet that runs atc ota switch -a on a per_artifact device starts running deactivate.* hooks that never ran on that form before. Read the deactivate.* declarations of the versions already on your fleet before you roll out an atcd carrying this.

The unwind ladder brackets that compensation. #4257 is where it started to run on -a. A refused -a activation dispatches rollback.enter, then the compensation that brings the outgoing version back, then rollback.exit. If a rollback hook itself fails, it also dispatches failure.enter and failure.exit. A take-back that could not restore the device does the same.

These cells read the TARGET version’s own sidecar, which every version carries. So they run at the per-artifact arm, like every other cell of this walk. A hook there cannot veto anything. The unwind track IS the failure path. A non-zero exit is recorded, and the ladder walks on.

One ordering differs from atc ota switch <version>. On that form rollback.enter runs while the pointer still names the target. On -a the pointer is already back, because putting it back is what tells this walk’s background task the activation was refused. Nothing may be awaited before that. So an @name/... cross-ref in a -a rollback.enter hook resolves against the outgoing version.

A refused atc ota switch -a now runs your rollback.* scripts. It runs your failure.* scripts too, when the take-back did not restore the device. These are the same cells a push and a poll already dispatch. So a fleet that declares them for either of those planes starts running them on this form too. Both cells bracket the workload restart, and the pointer is back before either of them runs.

Which hooks run after atcd crashes during a push

If atcd dies during a push, for example on a power loss, the next start of atcd finishes the delivery. What it does depends on where the push stopped. Here n is the version that was active before the push, and v is the pushed version.

Every hook in the table reads daemon_restart_recovery in ATC_HOOK_CONTEXT.trigger. That is how a hook, and an operator reading atc ota status, tells a crash from a hook that refused: a refusal’s rollback reads auto_rollback_veto. Write an activate.* hook so that it can run again on a version that is already active, because these re-runs do exactly that.

A crash after the pointer flip is treated as an activate.exit failure. atcd puts current back on n, then runs these hooks in this order:

1. v’s rollback.enter.
2. n’s activate.enter and activate.exit, if n’s deactivate.* ran during the push. If activate.enter refuses, activate.exit does not run. The refusal rule is the one above: every stopped member ran an activate.enter hook, and each one exited non-zero.
3. v’s rollback.exit.
4. If a hook in steps 1 to 3 exited non-zero, v’s failure.enter and failure.exit.

If every hook passes, the delivery is rolled_back. If one does not, the delivery is failed. atcd runs this rollback at most once. If atcd dies again during the rollback, the next start marks the delivery failed and runs no hook. atcd never pushes v again by itself, because an activate.exit that crashes the device would then crash it on every start. To try v again, push it again.

The delivery-wide delivery.post hook, stage-scope hooks and the workload restart step do not run during this rollback.

A pulled delivery that a hook refuses after the flip gets the same rollback from the next start if atcd dies after the un-flip and before n’s compensating activate.* ran. If atcd dies during that compensation, the next start marks the delivery failed and runs no hook.

A live push whose activate.exit refuses ends with the same hooks and the same status (#6750).

After a crash before the flip, n’s hooks set the delivery status:

• If every member that went down runs an activate.* hook and no hook exits non-zero, the delivery is rolled_back.
• If a hook exits non-zero, or atcd cannot dispatch the hooks, the delivery stays in progress and atcd tries again on its next start. v’s failure.enter and failure.exit run, if v declares them.
• If a member that n’s deactivate.* stopped declares no activate.*, atcd cannot restart it. The delivery is failed, and v’s failure.* run, if v declares them.

While atcd runs these hooks, it refuses a new push or pull into that channel.

Which state’s hooks come from which version

Almost every hook runs the declaration of the delivery arriving. Two do not.

So a deactivate.* hook you add today does not run on today’s delivery. It runs on the next one, when the version you ship now becomes the outgoing version. An atc ota switch that leaves that version reaches it too. A first deploy traverses deactivate and dispatches nothing, because there is no outgoing version to tear down. So does a delivery that flips nothing: a stage-only push, a re-push of the version already active, or a pulled delivery re-cut over unchanged bytes. The rule is the same on both planes.

A push that flips nothing also runs no activate.*, because it activates nothing. That covers a stage-only push and a re-push of the active version. A pulled re-cut still runs activate.*, because it moves current to the new delivery. An atcd older than v0.1.5 did run activate.* there; see Behavior changed.

A pulled re-commit of the delivery already active runs no activate.*, because it moves nothing. A revision that only adds present-only deliveries makes that re-commit. So does the first poll after an atcd restart, when the last walk finished. A restart that resumes an unfinished walk still runs activate.exit.

An atc ota switch to the delivery already active runs no activate.* and no deactivate.*. It moves nothing, so it activates nothing. commit.* and the delivery bookends still run. An atc ota switch -a that repeats the per-artifact activation already current also runs no activate.* and no deactivate.*. An -a of a version that is live under a whole delivery still runs activate.*, because the pointer moves to a new delivery.

Scope

The Stage row holds on all three walks. An atc ota switch <version> runs the stage arm at each of the six gates it crosses, and crosses no other. The atc ota switch -a <artifact> form runs no stage arm at all. See Which of these run on an atc ota switch.

check.*, download.*, install.* and uninstall.* have no stage scope. The first three are artifact-major. The daemon crosses one gate per artifact, not one per delivery, so a stage table there would parse and never run. Push preflight rejects all four rather than accept one silently.

Declare them per artifact instead. That is where they run on a push and on a poll alike, once each, in push-list order. A pull walk reached none of the three until #3644; it dispatches all six cells now (atcd_pull::pull_poller::walk_acquire), and the rest of this page says so at “Which of these run on an atc ota switch”. The atc ota switch walk is the one that still reaches none of them, because it moves no bytes.

The refusal above is about SCOPE and not about a plane. A check, download or install cell declared at STAGE scope is rejected at preflight on every plane, and the device warns about each such cell once per delivery. It still runs it zero times.

uninstall.* is a separate case. It is dispatched per artifact of each version being removed, read from that version’s own sidecar. There is no per-version stage table for a delivery-wide cell to read. Put cleanup for one version in an artifact-scope uninstall.* hook; see Per-version cleanup: uninstall.*. gc.* brackets a whole sweep and names no version.

So the stage-scope states are activate, deactivate, commit, rollback and failure.

The daemon runs the same check over a signed manifest it pulls. Any invalid prong refuses the whole delivery. So a delivery published with one of these tables before the rule tightened stops activating on a pulling device. It used to activate with the hook silently skipped. Move the table under the artifact and publish again. The device cannot be repaired locally.

That move buys a pulling fleet what it buys a pushing one. The delivery activates again, and the hook runs. The pull walk reaches all three states at artifact scope (#3412 §1). Neither plane has a delivery-wide barrier at any of them, which is why the top-level table is refused rather than supported.

A carve-out worth knowing about. A stage-scope deactivate.* is legal and it runs, but it reads the incoming manifest, not the outgoing version’s. Stage hooks are delivery-wide and bound to no artifact, while the outgoing world is per-version, so there is no delivery-wide outgoing table to read from. If you want a teardown veto over the version already running, declare it at artifact scope.

Declare it at artifact scope for a second reason. The durable teardown record above covers the artifact-scope teardown only. A stage-scope deactivate.* can stop a workload, and a crash after it leaves nothing on the device that says so. That is true on all three planes.

Per-version cleanup: uninstall.*

Some work must follow one version off the device. Example: a hook loaded an OCI image into podman storage, and the tag must go when atc deletes that version. Declare that cleanup as an artifact-scope uninstall.* hook, not as gc.*:

[[bundle]]
name = "myapp"
source = "dist"           # ships scripts/release_image.sh

[[oci]]
name = "myapp-image"
source = "registry.example.com/myapp:1.4"

[oci.lifecycle.hook.uninstall.exit]
script = "@myapp/scripts/release_image.sh"
timeout_secs = 60

Both removal paths run uninstall.*: the housekeeping sweep and atc ota rm. gc.* runs on the sweep only, and it names no version.

What the hook receives. atc runs the gate once for each version it removes, before it deletes the version directory. On atc ota rm the delivery is already gone at that point, so the gate holds back the version and nothing else. ATC_HOOK_CONTEXT holds:

The hook runs in the directory of the version being removed. That is true for a @self script and for a @<peer> cross-ref.

A non-zero exit keeps the version. A non-zero exit, a timeout, or a script that cannot start withholds the removal. exit 21 is a refusal here too, not a defer. The housekeeping sweep asks again while the artifact has an active version. After that, the next atc ota rm or purge in the channel asks. So a cleanup hook that fails on every run keeps its version on disk for ever.

If the cleanup is best effort, report the failure and end the script with exit 0:

#!/bin/sh
# uninstall.exit on myapp-image. Release the podman tag of the version that
# atc removes. ctx() is the helper from "How hooks run".
sha=$(ctx version.new_sha) || exit 0
podman rmi "localhost/myapp-image:$sha" > /dev/null 2>&1 ||
  printf '%s\n' "release_image.sh: no tag for $sha" >&2
exit 0

The tag must be the one your load hook wrote. This example assumes the image’s install.exit ran podman tag <image-id> "localhost/myapp-image:$sha" with the same version.new_sha. A different tag makes podman rmi miss, and the script still exits 0.

Ship both artifacts in one push. An OCI version holds an image layout and none of your scripts, so its hook names a script that a bundle ships. A @myapp/... ref to a bundle in the same push is accepted. An uninstall.* hook runs at GC, after the push, so it puts no order on the push. See “Same-push dependencies” under “Script references”.

At uninstall.* and gc.*, @<peer> finds the ACTIVE version, not a copy kept with the image. No push is in flight when GC runs. So @myapp resolves to the myapp version that is active in the channel when the hook runs. The delivery pointer delivery/current/myapp names the same version. For an uninstall.* hook, two cases leave no script to run, and each one keeps the image version:

• The active myapp version does not ship the script. An older version that you switched back to is the usual cause.
• myapp is not in the active delivery, for example because you removed it first.

So ship the script at one fixed path in every bundle version, and remove the image versions before the bundle. To remove that dependency too, install the script at a fixed path outside the store, such as /usr/local/libexec/myapp/release_image.sh. atc does not manage or roll back that file.

gc.enter / gc.exit

The housekeeping sweep’s bracket. Declared at the top level under [lifecycle.hook.gc.*], because that is where you expect to write a hook, and carried in .atc/delivery.pb alongside the [delivery.hook.*] bookends, because a sweep bracket has a bookend’s shape and lifetime rather than a state’s. An artifact-scoped [bundle.lifecycle.hook.gc.*] is rejected at push.

[lifecycle.hook.gc.enter]
script = "@ops/scripts/pause_reclaim.sh"   # @self is forbidden here
timeout_secs = 30

gc.enter is your opt-out. A non-zero exit skips that sweep’s delivery GC and per-artifact version GC, and nothing else. The partial-transfer, orphan-staging, stream-cache and pull-partial reapers always run, whatever gc.enter says. Those reclaim disk to stop the device filling up, and a wedged operator hook that could block them would be a worse failure than the one the opt-out prevents.

gc.exit is reporting only. The sweep is over by the time it runs, so its exit code gates nothing.

Each channel runs its own gc.* bookend. The sweep is device-wide, but the declaration belongs to a delivery. So the sweep runs the bookend once for each channel whose active delivery declares it. On a device with one channel, that is once per sweep. On a device with agency/atcd beside your own channel, a hook declared in both deliveries runs twice in one sweep. At gc.enter, a non-zero exit from any one of them skips the delivery GC and version GC for the whole sweep.

A @<peer>/... script in gc.* resolves to the peer’s ACTIVE version when the sweep runs, the same rule as uninstall.*.

atc ota rm runs no gc.* hook. The pair brackets the housekeeping sweep only. rm is an operator command, not a sweep. It runs uninstall.enter and uninstall.exit for each version it removes, and it skips both gc.* gates. So a gc.enter opt-out does not stop an rm.

gc.exit does not run after the rm. It runs after the uninstall pass of a later sweep, and only on a sweep whose gc.enter let that pass run. For cleanup that must run when one version goes, use uninstall.*. That gate runs on both removal paths.

uninstall.* and gc.* both write a lifecycle_log row. A dispatched uninstall.enter or uninstall.exit is journalled like any other hook. A dispatched gc.* bookend is journalled under the id of the active delivery that declares it, with scope and trigger both gc and the uninstall cell it brackets. A refusal at any of the four also goes to the daemon log and the atc.housekeeping.uninstall_veto span. An uninstall.* refusal reads uninstall.<phase> refused for …; withholding its removal this sweep.

What a gc.* hook reads. A sweep is not a delivery, so the bookend context names no artifact and no version. ATC_HOOK_CONTEXT holds:

• push_id: the id of the active delivery whose [lifecycle.hook.gc.*] declaration runs. A sweep has no session of its own, so this can name a delivery pushed long before the sweep.
• state uninstall, phase enter or exit, and trigger gc.
• null for artifact, version, self_root.new, self_root.old and current_root.

An uninstall.* hook also reads state uninstall and trigger gc. Its artifact is not null, which is how a shared script tells the two apart. ATC_DELIVERY_CONTEXT carries hook_scope gc as well.

[lifecycle.hook.gc.error] and uninstall.error are both rejected at parse, as is every other .error cell: enter and exit are the only gates anything dispatches, so a .error cell would parse, push green, and never run. You get an error naming the two phases that do dispatch, rather than a hook that silently never fires.

Progress hooks: reporting DURING a state

Every hook above fires at a barrier. [lifecycle.progress.<state>] fires between two, on a cadence you set, while the state’s work runs:

[lifecycle.progress.download]
script = "@ops/scripts/report-progress.sh"
interval_secs = 5

The script gets two extra environment variables on top of the usual hook environment: ATC_PROGRESS_PCT, the percentage at that tick, and ATC_HOOK_POINT, which reads download.progress.

Read ATC_HOOK_POINT, not ATC_HOOK_CONTEXT.phase. progress is not a phase, so the context document carries the placeholder "phase":"enter", the same placeholder a delivery bookend carries. No gate waits on a progress hook.

Rules, all enforced at atc ota push:

It runs on push and on pull. atc ota push and a polled delivery both tick at the declared interval while a member transfers. atc ota push refuses to send the table to a device whose atcd predates the feature.

The two planes differ in two places:

• The exit code. On pull, a non-zero exit cancels the delivery. The member in flight stops, later members are not staged, and nothing activates. A timeout, a signal or a spawn failure does not cancel. On push, the exit code is logged and nothing acts on it.
• The first tick. Pull waits one full interval after a member’s transfer starts. Push arms its timer when the push starts, so the first tick in a transfer lands somewhere inside the first interval. A push tick that comes before the daemon knows the member’s total size runs no script, so the first run can come one interval later.

What it does NOT do, today:

• It runs once per member. The task brackets one artifact’s transfer. A three-bundle delivery runs the hook three times over, not once across the delivery.
• A slow script does not queue. A tick that arrives during a live run is skipped. Nothing is killed for being slow.
• It does not run a script in a member that is not installed yet. A tick resolves @<name>/<path> against the channel’s installed <name>. On the channel’s first delivery of <name>, that tree does not exist until the delivery commits, which is after every download. So atcd skips the hook and logs the skip, once per push or once per pulled member. For a first delivery, use an absolute path or a member that the channel already holds. atc ota push accepts the script, because later deliveries run it.

Script references

@self is forbidden in top-level (stage-scope) [lifecycle.hook.*] tables, because a stage hook runs once across all artifacts and isn’t bound to one tree. Use an artifact-scoped table ([bundle.lifecycle.*] / [oci.lifecycle.*]) for @self.

A stage hook’s @<name> must name an artifact in the same push or an artifact that is active in the channel. If it names neither, atc ota push fails at preflight, before any blob moves:

stage hook `commit.exit`: script `@sidecar/…` names no artifact in this push or active in channel `local/default`

Which version’s script @self runs

A version’s own scripts run only after that version is installed. Before that, an artifact hook’s @self/<path> belongs to the version that is ACTIVE on the device. This is the version that the delivery replaces. A rollback and a failure report belong to the version that the unwind restores, which is that same version.

At those gates the hook is the active version’s own hook. Its table entry comes from the active version’s atc.toml: whether the hook exists, its timeout_secs and its <path>. Its script comes from the active version’s tree.

An absolute-path hook and a @<name>/<path> hook name no version tree of the artifact. At check.*, download.*, install.enter, rollback.* and failure.*, they keep the INCOMING atc.toml. At deactivate.* and uninstall.*, they use the outgoing version’s declaration, as a @self hook does.

What follows from the rule:

• A guard that a release adds. A @self guard that release N+1 adds at check.*, download.* or install.enter runs first at the upgrade AFTER N+1, from the copy that N+1 installed. It does not run on N+1’s own delivery.
• A release that needs its own prerequisite. Declare it as an absolute-path or @<name> hook. It runs on that release’s delivery, the first delivery included.
• A guard that a release removes or changes. It still runs on that release’s delivery, with the active version’s script, path and timeout.
• Two hooks at one gate. The active version can declare a @self hook at a gate where the release declares an absolute or @<name> hook for the same artifact. Then both run: the active @self hook first, then the release’s hook. If either one refuses, the delivery is refused. A release cannot switch off the guard of the version that it replaces, and it keeps its own prerequisite.
• A first delivery. The device has no active version, so no @self hook runs at these gates and none writes a journal row. An absolute or @<name> hook still runs.
• An unwind with no version to restore. A first delivery that is refused after the switch runs no @self hook at rollback.* or failure.*.
• A version installed before atcd kept declarations. If the active version’s declaration cannot be read, no @self hook runs at these gates. The push journals deactivate_config_unavailable for that member.
• A missing script. The active version’s own @self script must be in its own tree. If the tree does not hold it, the hook fails, as at every other gate. It vetoes at check.*, download.* and install.enter. It marks the unwind dirty at rollback.* and failure.*.

[[bundle]]
name = "agent"
source = "dist"

# Runs the ACTIVE version's guard before any byte of the new version moves.
# A release that adds this table runs it first at the next upgrade.
[bundle.lifecycle.hook.download.enter]
script = "@self/hooks/pre-download.sh"

# A prerequisite of THIS release. It runs on this release's own delivery.
[bundle.lifecycle.hook.check.exit]
script = "/usr/local/libexec/agent/battery-ok.sh"

A pulled delivery that declares no hooks still runs the active delivery’s hooks: its @self guards at these gates and its deactivate.* hooks.

atc ota run-hook is not a delivery, and this table does not apply to it. It runs the hook that the ACTIVE version declares, from that version’s own tree, at every gate. It cannot run the guard of a version that is not active yet. Without -o, if no artifact declares the cell, it runs the active delivery’s stage hook. With -o <name>, it never runs a stage hook.

The hook runs in the active version’s tree, and ATC_HOOK_CONTEXT says so. At these gates self_root.old is the active (or restored) version’s tree, which holds the script and is the working directory. self_root.new names the incoming tree only if that tree is already on the device, and the hook must not run scripts from it. version.old_sha is the active version, and version.new_sha is the incoming one. On a first delivery the hook does not run.

A device whose atcd does not advertise active-hook-table-v1 reads the INCOMING atc.toml for a @self hook at these gates. It skips a script that the active tree does not hold.

Requires atc and atcd from nightly-2026-10-08 or later. No stable release has this rule yet: 0.1.5 does not. A stable atc refuses @self at download.enter itself, before it connects to the device. It prints self-dependency at download_enter is not allowed (circular), then ATC-E0401: atc.toml is malformed.

On a stable release, move that hook to install.exit, activate.* or commit.*, where @self is the incoming version’s tree in every release. Or give it an absolute script path that is already on the device before the first push. allow_missing_initial is also newer than 0.1.5, so a stable release cannot skip a missing script.

On the daemon side, the rule needs the self-active-tree-v1 capability. An older atcd refuses @self at download.enter and fails a @self hook at check.*. At download.exit, install.enter and rollback.*, it runs the incoming script. A @self hook at failure.* also requires the self-failure-active-tree-v1 capability. An atcd that has only self-active-tree-v1 runs the incoming script at failure.*.

So atc ota push refuses to send such a declaration to a daemon that does not advertise the capability, and names the capability and the gates. A pulled delivery has no handshake. Upgrade every device on a channel before you publish such a declaration to it.

Same-push dependencies

One push can carry an artifact and the artifacts it depends on. A dependency is a requires entry or an @<name>/... hook script. atc orders the push so that each dependency that an order can satisfy resolves. atcd refuses the two that no order satisfies: a cycle, and a check.* hook that names another artifact of the push.

Which version @<name> names. When <name> is in the same push, an artifact hook’s @<name> resolves to the version of <name> that this push installs, at check.* through commit.*. That is true before and after the flip, on a first deploy and on an upgrade. Other hooks resolve it to another version, the same release @self names there:

• deactivate.*, and the activate.* that a rollback runs again, run the OUTGOING version’s declaration. @<name> names the version active before the push.
• rollback.* and failure.* restore the earlier version. @<name> names the version of <name> they restore, or none on a first deploy. The incoming <name> may never have been placed: a download.exit veto unwinds before the install.
• uninstall.* and gc.* run at GC, after the push. @<name> names the version active when the hook runs.
• Delivery bookends, stage hooks and progress hooks resolve through the delivery pointer, as before.

An @<name> or a requires entry that names the artifact itself is not a same-push dependency. @<name> keeps resolving to its active version at check.* through commit.*, as before, so the artifact must be active, or atcd refuses with missing dependency. Use @self for the tree this push installs.

When <name> is not in the push, @<name> resolves to its active version, and <name> must be active, or atcd refuses with missing dependency: <name>.

The order. A push downloads its artifacts one at a time, in the order it sends them. Every state after install.* runs once all of them are installed. So atc ota push sends each artifact after the same-push artifacts that it requires, and after the artifacts that its download.* hooks name. Other artifacts keep the order of atc.toml.

An install.* hook runs after every download, so its @<name> resolves in any order. So does an @<name> at a later cell, for example activate.enter or uninstall.exit.

A push cannot order a check.* hook. A push runs check.* for every artifact before it downloads any. So a check.* hook that names another artifact of the same push finds nothing, in any order, and atcd refuses it. Move the hook to download.* or install.*, or push the other artifact first.

A pulled delivery stages in the same order. A device that pulls a delivery takes each artifact through check.*, download.* and install.* before the next, and stages an artifact after the artifacts it depends on. So on a pull, a check.* hook can name another artifact of the same delivery.

Both planes refuse a cycle, because no order can install a after b and b after a. A push and a pull report the same message. A device refuses a pulled cycle before it downloads anything, stays on its current release, and reports the rollout failed. An atcd from before this check can have installed such a delivery, or started to and parked it on a hook retry. The device keeps the first and can finish the second. The message:

same-push dependency cycle: a -> b -> a. No order of the push installs each artifact after the ones it depends on. Remove one dependency.

Older versions. An atc that predates the ordering sends artifacts in atc.toml order. atcd cannot reorder a push, so it refuses one that lists an artifact before its dependency: artifact "a" is listed before "b", which it depends on. Upgrade atc. An atcd that predates the ordering refuses every same-push dependency with depends on b which is in the same push — split into separate pushes. Upgrade atcd, or split the push as it says.

How hooks run

Write hooks so a second run is harmless — including a second run that OVERLAPS the first. atcd will not re-run a hook for a delivery it knows it already started, but it can lose that knowledge: if the daemon restarts partway through an activation, a retry starts the delivery again from the top. The same applies if atc ota switch reconnects after a drop early enough that it has not yet seen any progress from the daemon.

The first instance may still be running. Hooks are spawned into their own process group (process_group(0)), so they do not receive signals sent to atcd. atcd kills that group when a hook exceeds its own timeout_secs, but a daemon that is SIGKILLed, OOM- killed, or panics runs no cleanup at all — the hook is reparented and keeps going. A retry can therefore put two instances of the same script on the device at once.

Requires a daemon whose unit grants CAP_KILL. With hook_run_as_user set (the default), the hook runs as a different uid than atcd, and POSIX allows kill(2) only from root or a matching uid. A unit rendered before that capability was granted therefore takes EPERM on both SIGTERM and SIGKILL: timeout_secs still fires on time, and the hook still runs to completion while being reported as a timeout. Re-bootstrap (atc bootstrap atcd install) to re-render the unit; systemctl show atcd -p AmbientCapabilities should list cap_kill. See #3242.

Sequential safety is not enough for that. Do both:

• Idempotent steps, so a re-run is a no-op: mkdir -p not mkdir; write to a temp path and rename(2) over the target rather than appending; check whether a unit is already enabled before enabling it; make “already done” an exit-0 path, not an error.

• Mutual exclusion, so two instances cannot interleave: take an flock on a well-known path for the duration of the hook — blocking with your own bound, e.g. flock -w 60 /run/atc/myhook.lock -c '…'. Wait for the lock, then do the work (your idempotent steps make the second pass cheap) and report your own real outcome; if the wait expires, exit non-zero, because at that point you genuinely do not know whether the other instance is progressing or wedged.

  Bound the lock wait yourself — do not rely on timeout_secs. A hook declaring timeout_secs = 0 runs unbounded (that is the documented rule), so a plain blocking flock behind an orphan that never exits stalls the deployment forever instead of failing it. -w is what makes the failure mode legible on every configuration.

  Do not flock -n and exit 0 on losing the race. Exit 0 tells atcd the hook succeeded and the lifecycle advances — but the instance holding the lock is a leftover from a dead daemon that may still fail or leave its work half-finished. You would be reporting completion for work you neither did nor observed. Exiting non-zero is no better: it fails a deployment that may well be fine. Blocking is the only option whose exit code describes something the hook actually knows.

• Prefer single atomic operations over read-modify-write sequences; two concurrent read-modify-writes lose an update no matter how idempotent each looks alone.

A hook that only works on a clean, solitary first run will eventually fail a deployment on a device that rebooted at the wrong moment.

Background and the design decision behind this: docs/internal-site/src/design/2026-07-27-switch-wait-and-resume-design.md (“Accepted limit: one window can still run hooks twice”).

atcd exec’s the script directly — there is no sh -c wrapper. The file needs the executable bit and a shebang; a script without +x is rejected on the device just before it runs (not executable: mode=…). Neither is pre-checked when you author the manifest or at push preflight — a bad script only surfaces once its hook fires during a delivery — so chmod +x and check head -1 for the shebang in source before pushing. An absolute script that does not exist fails the hook with ENOENT, unless the hook sets allow_missing or allow_missing_initial (A hook script that the artifact installs). Those keys never excuse a script that exists without +x.

For an ARTIFACT-scope hook, the working directory is the tree that the @self script came from, if that directory exists. For most cells this is the new version. At check.*, download.* and install.enter it is the ACTIVE version, and at rollback.* it is the restored version, which is the same one (Which version’s script @self runs). For deactivate.*, for uninstall.*, and for the ladder’s compensating activate.*, it is the OUTGOING version. Those cells read their declaration from that version too. So a hook that ships data beside itself always gets the copy of its own version, even through a relative path.

Two shapes have no declaring version, and they keep the old rule. They run in the artifact’s new version directory, or in the script’s own directory when there is none. A @<peer>/... cross-ref names a script in another artifact’s tree. An absolute path names a script the operator owns.

A STAGE-scope or DELIVERY-scope hook is bound to no artifact. @self is refused there, so such a hook has no version tree at all. Its working directory is the script’s own directory.

The hook does not inherit atcd’s environment. The env is cleared, PATH is pinned to /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, and atcd exports the whole delivery context as a single JSON variable, ATC_HOOK_CONTEXT (see ATC_HOOK_CONTEXT below), plus the hook user’s login environment (HOME/USER/LOGNAME from the passwd entry; SHELL is the resolved bash on PATH, not the passwd login shell, which stays nologin; plus XDG_RUNTIME_DIR, DBUS_SESSION_BUS_ADDRESS). Parse the context with awk:

set -eu

# Read one dotted field out of ATC_HOOK_CONTEXT. A second argument gives the
# value to use when the field, or an object above it, is null. With no second
# argument, a field under a null object prints "". An unknown key or a null
# value exits nonzero, so a typo stops the hook instead of giving "".
ctx() {
  awk -v path="$1" -v nargs="$#" -v dflt="${2-}" '
    function fail(why) { print "ATC_HOOK_CONTEXT." path ": " why > "/dev/stderr"; exit 1 }
    function find(t, k,   j, c, d, q) {
      k = "\"" k "\":"
      for (j = 1; j <= length(t); j++) {
        c = substr(t, j, 1)
        if (q) { if (c == "\\") j++; else if (c == "\"") q = 0 }
        else if (c == "{") d++
        else if (c == "}") { if (--d == 0) return 0 }
        else if (c == "\"") { if (d == 1 && substr(t, j, length(k)) == k) return j + length(k); q = 1 }
      }
      return 0
    }
    BEGIN {
      s = ENVIRON["ATC_HOOK_CONTEXT"]; n = split(path, key, ".")
      for (i = 1; i <= n; i++) {
        if (substr(s, 1, 4) == "null") { print dflt; exit 0 }
        if (substr(s, 1, 1) != "{") fail("not an object")
        p = find(s, key[i])
        if (p == 0) fail("no such field")
        s = substr(s, p)
      }
      if (substr(s, 1, 4) == "null") { if (nargs < 2) fail("null"); print dflt; exit 0 }
      if (substr(s, 1, 1) != "\"") fail("not a string")
      for (j = 2; j <= length(s); j++) {
        c = substr(s, j, 1)
        if (c == "\"") { print v; exit 0 }
        if (c == "\\") { j++; c = substr(s, j, 1); if (c != "\"" && c != "\\" && c != "/") fail("holds a control escape") }
        v = v c
      }
      fail("not a string")
    }'
}

sha=$(ctx version.new_sha)
state=$(ctx state)
name=$(ctx artifact.name)
old=$(ctx version.old_sha "")   # null on a first install

artifact.* and version.* are empty in a hook bound to no artifact. A stage hook, a delivery.* bookend and a gc.* bookend get null for artifact and version. The helper stops at that null, prints "" or its second argument, and exits 0. A per-artifact hook always has a non-empty artifact.name, so [ -z "$(ctx artifact.name)" ] is true only in a hook bound to no artifact.

A ctx copied from an older version of this page exits 1 with not an object on those reads. Under set -eu, a delivery.pre that does so vetoes the delivery, and a gc.enter skips the sweep. Paste the helper above in its place.

A stage hook and a gc.* bookend can read push_id, state, phase and trigger. A delivery.* bookend gets placeholders in state and phase (commit, enter), so read its real phase from ATC_DELIVERY_CONTEXT. Its trigger is real, and it matches ATC_DELIVERY_CONTEXT: pull on a pulled delivery, user_activate on a switch. Before #7223 it read push on every plane.

Use awk, not jq or python3. The meta-atc device image carries neither of them. atcd pins the hook PATH to the standard system directories, so the hook sees only what the image already installed. awk is on every base image, busybox included.

The helper handles the fields atcd writes: strings, nulls and nested objects. It decodes \", \\ and \/. It refuses any other escape, such as a control character, and names the field. A hook that reaches for a missing reader gets an EMPTY value, not an error. It writes a broken file while the push reports success. set -eu plus the null check above turn that silent case into a failed state.

There are no discrete ATC_PUSH_ID / ATC_NEW_VERSION_SHA / ATC_STATE / ATC_PHASE / … variables — those were removed. Every field lives inside ATC_HOOK_CONTEXT. (There is likewise no ATC_VERSION; read .version.new_sha.)

Hook user: hooks inherit atcd’s own uid unless hook isolation is configured. On the bootstrap/systemd path it always is: bootstrap creates the atc-hooks user and bakes --hook-run-as-user into the unit (from [daemon] hook_run_as_user, default atc-hooks) — a separate user in the daemon’s group that can read the 0750 store but cannot corrupt it, and drives its own systemctl --user without sudo. A manually started atcd (the no-systemd foreground recipe) has no hook user configured — no atc-hooks user exists on such a device — so hooks run as whatever user launched the daemon (your SSH login user). Neither --hook-run-as-user <user> nor [daemon] hook_run_as_user can isolate them on that recipe: switching the hook uid requires a root (or CAP_SETUID/SETGID) daemon, which the un-sudo’d foreground recipe deliberately avoids — set neither there (see the no-systemd section of deploy-your-first-update.md). A hook user that resolves to the daemon’s own uid is rejected at startup, never silently accepted.

A hook that must read a root-owned file, a K3S kubeconfig for example, reaches it through a supplementary group on atc-hooks. Do not reach for hook_run_as_user = "root". atcd resolves that group set at every spawn, so the grant needs no ATC change and no daemon restart. The recipe and its cost are in the atcd non-root runtime model.

That is why writing hook_run_as_user into the file is what turns isolation on for a hand-launched daemon, and leaving it out is not (#3366). The key’s DEFAULT reaches only atc bootstrap atcd install, which creates the account and grants the unit the capabilities before it bakes the flag. A hand-launched atcd run --config <file> applies the key when the file states it, and keeps hooks on its own uid when the file does not — so a config written for a bootstrap install does not turn a foreground daemon into one whose every hook dies in its pre-exec. --hook-run-as-user still wins over the file, and atcd logs a warning when it overrides a different stated value.

[daemon] socket_mode reads the same way: --socket-mode wins, then the file, then 0660. So does [daemon] socket_owner: --socket-owner wins, then the file, then no grant at all.

Socket owner: the daemon’s own user owns the control socket. The controller opens it by SSH-exec’ing atcd attach as your SSH LOGIN user, so those two accounts must line up. Under the default install they do. The installer adds the login user to the daemon’s group, and the group bits of the 0660 socket then reach it.

Under run_as_user = "root" there is no such group. An operator may not join the root group, so the login user takes ATC-E0420 on every command while the daemon stays healthy.

[daemon] socket_owner closes that gap. atcd chowns the socket to the named account right after bind, and grants exactly that one uid. atc bootstrap atcd install --root <user>@<host> fills the key in from the target’s own user@. The same install also puts the store root (/var/atc, mode 0710) in that account’s primary group, so the account can traverse to the atcd binary the remote_path symlink resolves into; without that grant every atc ota verb fails with ATC-E0470. State it yourself when your target names no user, or when the account that runs atc is not the account the daemon expects.

Do NOT reach for socket_mode = "0666" instead. That mode hands an admin control socket to every local process on the device.

atcd REFUSES to start when the named account does not resolve, or when the chown fails. A grant that quietly did not happen leaves a socket nobody can open, behind a systemctl status atcd that still reads green.

On timeout_secs expiry the hook’s whole process group gets SIGTERM, then SIGKILL after a 2-second grace.

A hook that calls setsid() leaves that process group, so no signal reaches it. atcd therefore also puts each hook in its own cgroup (v2) and kills that cgroup on the same ladder. A process cannot leave its cgroup, and every descendant inherits it. The cgroup is the backstop. The process group stays the first move.

That backstop needs a cgroup subtree atcd may write to. The daemon probes for one at startup and says which mechanism is in force:

lifecycle hook bound: cgroup v2 at /sys/fs/cgroup/system.slice/atcd.service/atc-hooks …
lifecycle hook bound: process group only (…)

The second line is not an error. atcd runs in containers and under atcd run with no systemd, and a hook still runs and is still bounded by its process group there. It says only that a hook which daemonizes can outlive its timeout_secs. To get the backstop on a device installed before #3562, re-bootstrap (atc bootstrap atcd install). The regenerated unit carries Delegate=. Do not add the directive by hand: bootstrap rewrites the unit whole, so a hand edit is erased by the same command that would apply it. Where there is no systemd, point ATC_HOOK_CGROUP_ROOT at a writable cgroup-v2 directory instead. The same answer reads across a fleet as the atc.lifecycle.hook_cgroup_delegation metric. See #3321.

With timeout_secs = 0 there is no expiry: the hook runs until it exits or the delivery preempts it, so a wedged hook wedges the delivery. Use it only for work whose duration you genuinely cannot bound (a long flash, a large migration).

File modes: the packer records each source file’s mode, but the device materializes every bundle file read-only (0o440), adding u+x,g+x only when the source file had any execute bit. So only the exec bit survives the trip. Run chmod +x on your hook script in source before pushing. Write, “other”, and setuid/setgid bits are always stripped.

Directory modes are not part of that guarantee. The directories that hold delivered files keep the daemon user’s own mode, so a manual write as that user can still succeed. Do not predict which way a write goes. hook_run_as_user picks the account (see Hook user above), and the directory’s mode comes from atcd’s umask when the store was built. A permissive umask leaves the directory group-writable, and atc-hooks is in the store group, so even a non-owning hook can write. Neither input is a guarantee, so only the file modes are.

Keep writable state in a directory that the bundle declares with state_dirs. See Writable state: state_dirs below.

A write beside the binary also does not follow the delivery pointer. Each version lands in a directory of its own, so the file stays with the version that was current when it was written. Retention prunes it later, and a rollback to that version brings it back.

Phase names in the journal: each state’s journal labels are the six internal sub-phases (pre_hook → hook_done → pre_side_effect → side_effect_done → pre_hook_exit → hook_exit_done). An enter hook runs at <state>/pre_hook; an exit hook at <state>/pre_hook_exit. The push output names the hook as you wrote it, so a commit.exit hook reads commit.exit there. The pre_ means “about to run this hook”, not “before the state”.

Only enter and exit gate hook dispatch, so those are the only two that surface as a pre_hook* phase in the journal. Match on the hook line, ✓ commit.exit, not on the journal label.

<state>.error is not a declaration point at any state. Nothing dispatches an error gate — not for commit, not for any other state — so a .error cell would parse, push green, and never run. atc ota push rejects it and names the two phases that do dispatch, enter and exit.

It does not name a replacement cell, because which states you may use depends on the scope you declared at. deactivate.exit is valid under [bundle.lifecycle.*] and refused under [[peripheral]]. Pick enter or exit on a state your scope supports.

At artifact and stage scope that is usually <state>.exit, or the unwind hooks rollback.* and failure.*, which run after a delivery fails. Under [[peripheral]] those are not all available — failure.* and rollback.exit are refused there too, and rollback.enter is the peripheral’s one unwind hook. See the plane table below.

Which states a [[peripheral]] may declare. A peripheral’s hooks run on a shorter ladder than an artifact’s, and atc ota push refuses a cell the ladder never reaches. The whole plane:

Use check.enter, check.exit or install.enter to refuse a flash before it starts. All three run before any byte reaches the peripheral, and a non-zero exit from any of them stops the delivery with nothing written. check.enter is the earliest of the three. Declare the rejected states at artifact or stage scope instead.

Writable state: state_dirs

A bundle declares the writable directories its program needs. atcd creates them before every install.* and activate.* hook, so a check.* or download.* hook of a first delivery runs before they exist. They persist across versions.

[[bundle]]
name = "myapp"
source = "dist"
state_dirs = ["data", "cache"]

In KDL, the key is state-dirs "data" "cache" on the bundle node.

Rules. Each entry is one path segment: not empty, not . or .., and with no /. A list must not name one directory twice. Only a bundle may declare state_dirs. A bad entry fails the load with ATC-E0533, at atc ota push and at atc channel deliver. A device checks a signed list again and refuses a bad one.

Location. The root is <hook-home>/.local/state/atc/<bundle>/, where <hook-home> is the home directory of the hook user (see Hook user above). Each entry is a subdirectory of that root. atcd creates the root and each entry as the hook user, with mode 0700. The step does not depend on a hook: a bundle with no hooks gets its directories too.

Not in the delivery tree. atcd does not link, mount or copy the root into any version’s directory. A data/ that a program creates beside its own binary is a plain directory of that version, and the next version starts with an empty one. Give the program the root path: a hook reads ATC_STATE_DIR and passes it on, for example in the systemd unit that an install hook writes. Run the program as the hook user. No other account can open the 0700 root, so a program that you start over ssh as a different user cannot use it.

A bundle with no hooks. A one-shot CLI that you run by hand has no hook to pass the path on. Run it through atc shell instead:

atc shell user@robot -a myapp -- sh -c './myapp --data "$ATC_STATE_DIR/data"'

atc shell -a <bundle> runs the command as the hook user, in the bundle’s current version. When atc.toml declares state_dirs on that bundle, it also sets ATC_STATE_DIR to the same root that a hook gets. The program can read the variable itself, or take the path as an argument. The single quotes stop your local shell from expanding the variable. Without state_dirs, the variable is unset.

The shell sets the variable only when the root exists in the home of the user it runs as. With hook isolation off, hooks run as atcd’s own user, and the shell becomes atc-hooks. The root is then in another home, so the shell leaves ATC_STATE_DIR unset and prints the path it looked for on stderr.

What a hook sees. A hook of that bundle gets ATC_STATE_DIR, the root, and a state_dirs map in ATC_HOOK_CONTEXT from each name to its full path. A hook that atc ota run-hook, an uninstall or a recovery starts gets the same values. These read the root on disk, because the request carries no list.

Shared across channels. The root is keyed by bundle name alone. Two channels that ship a bundle of one name share its state. This is decided behaviour, not an accident.

Deletion. atc ota rm deletes the root when it removes the last installed version of that bundle name on every channel. A version switch, keep_n pruning and the housekeeping sweep never delete it. A failed delete does not fail the removal. atcd logs it and counts it in atcd.state_dirs.remove_failures.

Version skew. On a push, an atcd without the bundle-state-dirs-v1 capability cannot create the directories. atc ota push prints a warning for each bundle with state_dirs. On the pull plane, the list rides on the signed delivery manifest and there is no handshake. An atcd that predates the field ignores the list and creates nothing. So atc channel deliver prints a note for each bundle with state_dirs.

ATC_HOOK_CONTEXT

Every hook receives one atc variable, ATC_HOOK_CONTEXT, holding a compact JSON object. Field presence is stable: artifact, version, self_root.new, self_root.old, current_root, and peripheral are always present. JSON null means “absent”, so a parser never has to special-case a missing key. A hook bound to no artifact gets null for the whole artifact and version values: a stage hook, a delivery.* bookend and a gc.* bookend. When version is an object, version.old_sha is always present.

{
  "push_id": "01J…",                       // session id: a ULID on push,
                                           // "pull-activate-<delivery-id>" on pull,
                                           // "manual-run-hook-<conn-id>" for `atc ota run-hook`
  "artifact": { "kind": "oci",             // "oci" | "bundle"; the whole object is
                "name": "tempo" },         // null for a hook bound to no artifact
  "version":  { "new_sha": "abc…",         // version being installed/activated; the whole
                "old_sha": null },         // object is null like `artifact`. old_sha: the
                                           // previously-active version, or null on first install
  "state":  "install",                     // any name from the states table above
  "phase":  "enter",                       // enter | exit
  "trigger": "push",                       // push | pull | user_activate | gc | manual_run_hook | …
  "artifacts_dir": "/var/atc/…/artifacts", // artifacts umbrella dir on the device
  "self_root": { "new": "/var/atc/…/versions/abc…",  // new version's root dir, or null
                 "old": null },            // previous version's root dir, or null.
                                           // At check.*, download.*, install.enter,
                                           // rollback.* and failure.*, @self and the
                                           // working dir are "old"
  "current_root": "/var/atc/…/delivery/current/tempo",  // stable pointer, or null
  "peripheral": null                       // peripheral-flash context, or null
}

Read fields with awk, through the ctx helper in How hooks run. The document arrives in the ENVIRONMENT, not on stdin, so ENVIRON["ATC_HOOK_CONTEXT"] is the read; a per-artifact hook gets /dev/null on stdin. The same call works from #!/bin/sh and from #!/usr/bin/env bash. The schema is defined in crates/atc-lifecycle/src/hook_context.rs, which is the source of truth.

current_root: the path a service unit must name

current_root is the artifact’s delivery/current/<name> pointer. atc flips it atomically on every push, switch and rollback, so a systemd ExecStart that names it follows every later version with no edit. Take the field; never derive the path yourself:

set -eu
# Print one dotted field of ATC_HOOK_CONTEXT, or the second argument when
# it or an object above it is null. With no second argument, a null field
# exits 1 and a field under a null object prints "". A missing key exits 1.
ctx() {
  awk -v path="$1" -v nargs="$#" -v dflt="${2-}" '
    function fail(why) { print "ATC_HOOK_CONTEXT." path ": " why > "/dev/stderr"; exit 1 }
    function find(t, k,   j, c, d, q) {
      k = "\"" k "\":"
      for (j = 1; j <= length(t); j++) {
        c = substr(t, j, 1)
        if (q) { if (c == "\\") j++; else if (c == "\"") q = 0 }
        else if (c == "{") d++
        else if (c == "}") { if (--d == 0) return 0 }
        else if (c == "\"") { if (d == 1 && substr(t, j, length(k)) == k) return j + length(k); q = 1 }
      }
      return 0
    }
    BEGIN {
      s = ENVIRON["ATC_HOOK_CONTEXT"]; n = split(path, key, ".")
      for (i = 1; i <= n; i++) {
        if (substr(s, 1, 4) == "null") { print dflt; exit 0 }
        if (substr(s, 1, 1) != "{") fail("not an object")
        p = find(s, key[i])
        if (p == 0) fail("no such field")
        s = substr(s, p)
      }
      if (substr(s, 1, 4) == "null") { if (nargs < 2) fail("null"); print dflt; exit 0 }
      if (substr(s, 1, 1) != "\"") fail("not a string")
      for (j = 2; j <= length(s); j++) {
        c = substr(s, j, 1)
        if (c == "\"") { print v; exit 0 }
        if (c == "\\") { j++; c = substr(s, j, 1); if (c != "\"" && c != "\\" && c != "/") fail("holds a control escape") }
        v = v c
      }
      fail("not a string")
    }'
}
root=$(ctx current_root)
[ -n "$root" ] || { echo "install_unit.sh: current_root is null" >&2; exit 1; }

The set -eu line and the emptiness test are part of the recipe. A reader that is absent from the device gives an empty $root, and the hook then writes a broken ExecStart and exits 0. The push reports the hook as successful.

It is a path, not a directory the field checked. At install.exit on a first install, nothing under delivery/ exists yet. That is exactly the hook which writes the unit file. So write the value out, and do not test it first. self_root.new is the field filtered by existence, and current_root is not.

Do not read the incoming version through it before the flip. On an upgrade, current_root names the OUTGOING version at every cell up to the flip, install.exit included. A test such as [ -x "$root/bin/myapp" ] there inspects the old tree. To inspect the version being installed, read self_root.new. An @self hook at install.exit also runs in the incoming tree, so a path relative to its working directory reads that tree too.

Do not compute it from the hook’s working directory. A hook runs in artifacts/<name>/versions/<sha>, which is a sibling of delivery/, not a path under it. Counting levels up from $PWD and stripping from /delivery/ both produce a plausible string that resolves to nothing.

current_root is null for a peripheral-flash hook. That plane’s artifacts_dir is the artifact’s own version dir, not the channel umbrella. No channel pointer can be named from it, and a firmware flash has no service pointer to follow.

What each field names at each cell

The table gives, for each delivery cell of an ARTIFACT-scope hook, whose declaration runs, which tree a @self script runs in, and which version each field names. It uses three words for two versions:

• incoming: the version that the delivery or the switch brings.
• outgoing: the version that was active before it.
• active and restored: the outgoing version again. It is “active” before the install, and “restored” on the unwind ladder, which brings it back.

How to read the columns:

• A declaration is the atc.toml that decides whether the hook exists, its timeout_secs and its script path. “active” and “restored” name the version that was active before this delivery, so they are the outgoing version’s declaration. A @self hook at those cells is that version’s hook: a release cannot add, remove or change it there.
• A @self/<path> script joins the tree of the version whose declaration it is. That tree is also the hook’s working directory. A @<name>/... or an absolute script names no version tree. That hook runs in self_root.new if that directory exists, else in the directory of the script.
• The trigger column gives the value the hook reads. It does not say which walk dispatches the cell. A switch runs no check, download or install. A push that flips nothing runs no activate and no deactivate; see Behavior changed.
• On a first deploy there is no outgoing version. old_sha and self_root.old are null, no @self hook runs at an “active” or “restored” cell, and no deactivate.* runs.

A stage hook (top-level [lifecycle.hook.*]) is bound to no artifact. It gets null for artifact, version and current_root, and it reads the INCOMING manifest at every cell, deactivate.* included. A peripheral-flash hook gets null for current_root.

The table pins the version pair that a PUSH dispatches. Three other callers pass a different pair to the compensating activate.*:

• A pulled delivery, an atc ota switch and a boot recovery pass the restored version as both new_sha and old_sha. Only a push passes the failed version as old_sha.
• A boot recovery runs the same cells as the compensation, under the trigger daemon_restart_recovery, with the outgoing declaration in the outgoing tree.

uninstall.* and atc ota run-hook are not deliveries. An uninstall.* hook runs the declaration of the version being deleted, in that version’s tree. atc ota run-hook runs the declaration of the version that is active now, in that version’s tree, at every cell. It gives new_sha as that version, old_sha as the version before it, and trigger as manual_run_hook. Without -o, if no artifact declares the cell, it runs the active delivery’s stage hook, which gets null version fields.

version.new_sha has three meanings

So new_sha names the failed version at rollback.* and the restored version one step later, at the compensating activate.*. To tell a rollback from a forward activation, read trigger. Do not compare new_sha with the version that current_root names:

case "$(ctx trigger)" in
  auto_rollback_veto | daemon_restart_recovery) action=restore ;;
  *) action=forward ;;
esac

The compensating activate.* runs the outgoing version’s own script, so a version’s activate.* must also bring that same version back after a failed upgrade. Write it to run twice without harm.

What current_root resolves to

current_root is <channel>/delivery/current/<artifact>. delivery/current is a symlink to the active delivery’s directory. In that directory, <artifact> is a symlink with the relative target ../../artifacts/<artifact>/versions/<sha>. So sha below is the sha of the version that current names at that moment:

root=$(ctx current_root)
sha=$(basename "$(readlink -f "$root")")

The last column of the table says which version that is at each forward cell. Before the flip it is the outgoing version. After the flip it is the incoming version. On a first deploy the link does not exist before the flip, so readlink -f prints nothing. Test [ -e "$root" ] before you read the sha.

On the unwind ladder:

• At rollback.enter, current_root still names the version that current held when the hook refused. After the flip, that is the incoming version. atc ota switch -a is the exception: it puts the pointer back first.
• At the compensating activate.*, at rollback.exit and at failure.*, it names the restored version if atc put the pointer back. If that revert failed, or a newer delivery took the channel, it names whatever current names then. Read the link. Do not assume.
• A first deploy refused after the flip has no version to restore, so atc removes delivery/current.

A progress hook ([lifecycle.progress.download]) gets the pointer of the artifact being DOWNLOADED, not of the artifact that ships the script. Its new_sha is the version being downloaded. Its old_sha, self_root.new and self_root.old are null.

An outgoing version with no deactivate.* hooks

Warning: no artifact hook stops the outgoing version when it declares no deactivate.* hook. deactivate.* reads the OUTGOING version’s declaration, not the incoming one. A version shipped before you added the hook declares none. So does a version installed by an atcd too old to keep the declaration. On a push, a pull or a switch away from such a version, a workload that only your hooks start and stop keeps running while the incoming activate.exit runs.

Two things can still stop it: an incoming stage-scope deactivate.*, and the stop that atc runs for a workload that atc itself manages.

So write the incoming activate.exit to take over a workload that still runs. Use systemctl --user restart, not enable --now. See A name shared across versions.

A refusal later in that delivery has no teardown to undo either. If no member’s deactivate.* ran, the unwind runs no compensating activate.*.

Behavior changed: a push without --switch runs no activate.*

Changed in atcd v0.1.5 and nightly-2026-10-02 (#6361, 2026-10-01). A push that flips nothing dispatches no activate.* and no deactivate.*, at any scope. Two pushes flip nothing: a push without --switch (stage-only), and a re-push of the version already active. An artifact-scope commit.* does not run either. A stage-scope commit.* still runs.

Before that change, such a push ran activate.enter and activate.exit with current still on the old version. So an activate.exit hook written for that behavior often carries a guard: exit early when new_sha is not the version that current_root names. On atcd v0.1.5 and later, a forward activate.exit runs only after the flip, so that guard never fires there. Keep it only while a device runs an older atcd.

Do not move the same guard to activate.enter. There it fires on every upgrade, because current_root still names the outgoing version.

What a hook does not receive

ATC_HOOK_CONTEXT is the only atc variable. atc shell -a <name> also sets ATC_ARTIFACTS_DIR, ATC_ARTIFACT_KIND, ATC_ARTIFACT_NAME, ATC_SELF_ROOT and ATC_NEW_SELF_ROOT for interactive work, and it starts you in delivery/current/<name>. A hook receives none of those. It starts in the versioned tree instead. So a script written in that shell fails on its first real push with unbound variable. Debug against the document, or against atc ota run-hook, which dispatches through the real hook path.

A hook’s exit status is its last command’s

There is no separate success signal. The shell’s usual rule applies, and a non-zero status vetoes the delivery. A diagnostic hook that ends in a find over one unreadable directory therefore fails a push whose real work already succeeded, and the reported failure names the hook rather than the find. End any script that reports rather than gates with an explicit exit 0.

Worked example: a commit.exit hook

Drop a marker file once a bundle has been committed (made active) on the robot:

[config]
autoswitch = true        # commit/switch on push without an explicit --switch

[[bundle]]
name = "myapp"
source = "dist"          # directory ATC chunks + sends

# Runs AFTER myapp is committed (made active) on the device.
[bundle.lifecycle.hook.commit.exit]
script = "@self/scripts/mark_committed.sh"   # path inside myapp's own tree
timeout_secs = 60

scripts/mark_committed.sh lives inside the bundle source (dist/), so it is shipped with the artifact and resolved via @self. It must be chmod +x with a shebang in dist/ — see How hooks run.

A marker is a report, so commit.exit is the right gate for it. If the script were instead the thing that brings myapp into service, activate.exit is the better home: it is the first gate after the pointer flips, so the window where the device points at the new version but is not yet running it stays as short as possible. See the hook cutover runbook.

With multiple [[bundle]] entries, a [bundle.lifecycle.hook.*] table binds to the [[bundle]] defined immediately above it (TOML array-of-tables semantics) — keep each bundle’s hooks directly under that bundle.

Worked example: a long-running service

This is the pattern behind “ship a bundle and keep it running across reboots”. atc has no [service] key and starts no process of its own. A systemd unit does that, and three lifecycle hooks install it and drive it.

Bootstrap already prepares the device for the rootless half of this. It creates the atc-hooks user, gives it the home directory /var/lib/atc-hooks, and runs loginctl enable-linger for it. Linger is what starts that user’s systemd manager at boot with no login, so a --user unit the hooks enable comes back after a reboot. Every hook already receives XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS for that manager, so the scripts below need no sudo and no root daemon.

The declaration

[[bundle]]
name = "myapp"
source = "dist"

# Write the unit and reload systemd BEFORE the pointer flips. A unit file that
# systemd refuses fails the delivery here, instead of after the robot is live.
[bundle.lifecycle.hook.install.exit]
script = "@self/scripts/install_unit.sh"
timeout_secs = 60

# Start the new version. activate.exit is the first gate after the flip.
[bundle.lifecycle.hook.activate.exit]
script = "@self/scripts/start_service.sh"
timeout_secs = 60

# Stop the outgoing version. This declaration runs on the NEXT delivery, when
# the version you ship now becomes the outgoing one.
[bundle.lifecycle.hook.deactivate.enter]
script = "@self/scripts/stop_service.sh"
timeout_secs = 60

dist/scripts/install_unit.sh

The unit points at delivery/current/myapp, not at this version’s own directory. delivery/current is the pointer atc flips atomically on every push, switch and rollback. A unit pinned to one version’s SHA cannot follow it.

The hook reads that path out of ATC_HOOK_CONTEXT.current_root rather than hard-coding it. The same script then works on any channel and any store root. It also never walks up from its own working directory, which is a sibling of delivery/. See current_root.

ConditionPathExists= guards the one case where the pointer is absent. A rollback restores the earlier delivery, and the pointer follows it. A first delivery has no earlier one, so a veto on it removes delivery/current. With Restart=always and no condition, the unit then restarts every few seconds into “No such file or directory”. The launcher’s one useful log line is buried under the repeats. The condition makes systemd skip the start and report the unit as inactive instead. The next successful push creates the path, and the activate.exit hook starts the unit as usual.

#!/bin/sh
# install.exit — pre-flip. The new tree is on the device and readable, and
# `current_root` still points at the OLD version (or at nothing, on a first
# install). Either way the path is the one the unit must name.
set -eu

# awk, not jq or python3: the device image carries neither, and a missing
# reader gives an EMPTY root rather than an error. `set -eu` and the test below
# make that case stop the hook.
# Print one dotted field of ATC_HOOK_CONTEXT, or the second argument when
# it or an object above it is null. With no second argument, a null field
# exits 1 and a field under a null object prints "". A missing key exits 1.
ctx() {
  awk -v path="$1" -v nargs="$#" -v dflt="${2-}" '
    function fail(why) { print "ATC_HOOK_CONTEXT." path ": " why > "/dev/stderr"; exit 1 }
    function find(t, k,   j, c, d, q) {
      k = "\"" k "\":"
      for (j = 1; j <= length(t); j++) {
        c = substr(t, j, 1)
        if (q) { if (c == "\\") j++; else if (c == "\"") q = 0 }
        else if (c == "{") d++
        else if (c == "}") { if (--d == 0) return 0 }
        else if (c == "\"") { if (d == 1 && substr(t, j, length(k)) == k) return j + length(k); q = 1 }
      }
      return 0
    }
    BEGIN {
      s = ENVIRON["ATC_HOOK_CONTEXT"]; n = split(path, key, ".")
      for (i = 1; i <= n; i++) {
        if (substr(s, 1, 4) == "null") { print dflt; exit 0 }
        if (substr(s, 1, 1) != "{") fail("not an object")
        p = find(s, key[i])
        if (p == 0) fail("no such field")
        s = substr(s, p)
      }
      if (substr(s, 1, 4) == "null") { if (nargs < 2) fail("null"); print dflt; exit 0 }
      if (substr(s, 1, 1) != "\"") fail("not a string")
      for (j = 2; j <= length(s); j++) {
        c = substr(s, j, 1)
        if (c == "\"") { print v; exit 0 }
        if (c == "\\") { j++; c = substr(s, j, 1); if (c != "\"" && c != "\\" && c != "/") fail("holds a control escape") }
        v = v c
      }
      fail("not a string")
    }'
}
root=$(ctx current_root)
[ -n "$root" ] || { printf '%s\n' "myapp: current_root is null" >&2; exit 1; }

# An ALLOWLIST, not a list of the characters that happen to break today. The
# artifact-name validator permits every one of them, so this is the only gate.
case $root in
  *[!A-Za-z0-9/._+:@%\ -]*)
    printf '%s\n' "myapp: unsafe character in the delivery path" >&2; exit 1 ;;
esac

# `%` is inside the allowlist, because a real path can hold one. Double it, or
# systemd reads a specifier: a name holding `%n` resolved to the UNIT name.
esc=$(printf '%s' "$root" | sed 's/%/%%/g')
unit_dir="$HOME/.config/systemd/user"
mkdir -p "$unit_dir"
cat > "$unit_dir/myapp.service" <<UNIT
[Unit]
Description=myapp
# A vetoed first delivery removes delivery/current. Without this line the
# unit restart-loops on "No such file or directory" until the next push.
ConditionPathExists=$esc/bin/myapp

[Service]
ExecStart="$esc/bin/myapp"
Restart=always

[Install]
WantedBy=default.target
UNIT

systemctl --user daemon-reload

One allowlist, and one escape. An enumeration of the characters that break is the wrong shape here. It is only ever as complete as the last person to look. Two rounds of review found %, then "/\/newline, then asked about $. The allowlist above ends that: a character must be added to it to reach the unit file.

% is the one allowed character that still needs work. A real path can hold one, and systemd expands it. Measured with systemd-analyze verify: a path holding %n resolved to …/pct<unit-name>/…, which names no directory. Doubling it to %% resolved the literal path.

$ needs no escaping, which is the counter-intuitive part. systemd expands no variable in the FIRST argument of a command line. The executable is that argument. Measured the same way, with Environment=HOME=/pwned set. A path holding $HOME resolved to the literal $HOME path. $$HOME resolved to a literal two-dollar path, so doubling would have broken a path that worked.

In an argument AFTER the executable a $ does expand, and $$ is the escape there. The ExecStart above has no such argument, and $ is outside the allowlist, so the question does not arise for a delivered path.

The allowlist is deliberately stricter than systemd. Measured over 26 characters, systemd fatally rejects only a single quote and a tab in an executable path. The message is “Executable path contains special characters”. It treats `, ;, &, | and the rest as literal, because it runs no shell, so those would work. They are refused here anyway. A bundle named a;b is a typo far more often than an intention, and a narrow allowlist is the version of this rule that cannot rot.

validate_artifact_name permits every character named above. It rejects an empty name, /, :, NUL, ./.., the reserved oci and .atc, and an over-long name. It rejects nothing else, so all of these are pushable today. Tightening it is tracked in #4293. Until then, this guard stands between a pushed name and a generated unit file.

install.exit is the right place to refuse. The hook has written no unit yet, so a non-zero exit vetoes the delivery with nothing installed, and it leaves the running service untouched.

The heredoc is unquoted (<<UNIT, not <<'UNIT') so $esc expands. Every other $ in the TEMPLATE expands too, and the shell gets there before systemd does. So a variable you want SYSTEMD to expand needs a backslash: write \$MAINPID, which reaches the file as $MAINPID. $$MAINPID does not work here, because the shell reads $$ as its own PID and writes that instead.

The interpolated path needs none of that. $esc is a variable VALUE, and the shell does not rescan what it substitutes. So the path is written once, exactly as the allowlist and the sed left it.

dist/scripts/start_service.sh

Use restart, not enable --now. enable --now exits 0 against a unit that is already active, and starts nothing. So a delivery whose outgoing version declared no deactivate.* keeps the old process alive under a green push. See A name shared across versions.

#!/bin/sh
# activate.exit — post-flip. delivery/current already names this version.
set -eu

systemctl --user enable myapp.service      # survive the next reboot
systemctl --user restart myapp.service     # always replace the running process
systemctl --user is-active --quiet myapp.service

The is-active line is the hook’s own check. restart reports success for a start job that is only queued, so without it a service that starts and dies still commits a green delivery.

dist/scripts/stop_service.sh

#!/bin/sh
# deactivate.enter — pre-flip, and the incoming activate.exit runs after it.
set -eu

systemctl --user stop myapp.service

The stop needs no test for “is my version still the current one”. The order guarantees the answer: deactivate.* runs inside activate, and the incoming version’s activate.exit runs after both deactivate gates.

The three scripts bind to the versions that ship them

A hook is resolved out of the version tree its declaration came from. @self/scripts/stop_service.sh means that version’s copy of that file. So stop_service.sh here is not the script that stops the version in this push. It is the script that stops the version already on the robot, and that copy shipped with it.

Editing atc.toml or a script therefore reaches the deliveries after this one, never the way out of a version already installed. Two consequences bite in practice:

• The pair does nothing on the push that first adds it. The outgoing version declared no deactivate.*, so nothing stops the old process, which is why start_service.sh must restart rather than enable --now.
• A relabelled push of unchanged content still runs the old stop_service.sh. That copy can be wrong: it can stop a unit the incoming version just started. Fixing the file does not fix that push. Push a version carrying the corrected script, then push again.

The push output does not tell you whose copy ran. A hook line reports the declared ref, @self/scripts/stop_service.sh, which reads the same for every version. So a stale deactivate.enter looks exactly like a current one. Reason it out from the order above, not from the output.

A wrapper that sources someone else’s environment

A ROS node, a Python venv, and most other environments need a small wrapper. The wrapper sources a setup script, then execs the program. Point ExecStart at the wrapper, not at the program.

Give that wrapper a #!/bin/bash shebang, not #!/bin/sh. On a device image /bin/sh is dash. Catkin’s install/setup.bash and ROS 2’s /opt/ros/<distro>/setup.bash both need bash. They read $BASH_SOURCE to find their own directory, and they use bash-only syntax elsewhere. Under dash they report Bad substitution or builtin: not found, and neither names the shell.

Source setup.bash, not setup.sh. This is a second fault with its own symptom, and the right shebang does not fix it. A POSIX shell cannot tell a sourced script its own path, so colcon’s install/setup.sh carries the prefix of the machine that built it. The robot has no such directory, and the script says so: The build time path '<path>' doesn't exist.

atc ota push catches neither mistake. It ships your script and never reads it, so the push reports success and the unit is broken.

Ship a ROS 2 install space built without --symlink-install. With that flag, colcon gives an ament_python package an editable install. Its install/<pkg>/lib/python3.X/site-packages/ then holds a <pkg>.egg-link and no .py files. The .egg-link holds an absolute path to build/<pkg> on the build machine, and that directory links into src/. The bundle holds neither directory.

The console script under install/<pkg>/lib/<pkg>/ looks the package up by its metadata, and the .egg-info with that metadata also stays in build/. So the push succeeds, and the node fails on the robot with importlib.metadata.PackageNotFoundError or ModuleNotFoundError.

Run a plain colcon build for the install space you ship. A ~/.colcon/defaults.yaml with symlink-install: true under build: sets the flag on every build, so check that file too. atc ota push warns when a bundle carries a .egg-link, or a __editable__ pointer from pip install -e. It does not refuse the push.

Do not set -u in that wrapper. ROS 2’s /opt/ros/<distro>/setup.bash reads AMENT_TRACE_SETUP_FILES without setting it, and a venv activate does the same with variables of its own. Under nounset the wrapper dies before your program starts, and the failure names a variable you never wrote. set -e alone is fine.

#!/bin/bash
# Sourced environment first, strict mode after it. Never around it.
set -e
. /opt/ros/humble/setup.bash
exec "$(dirname "$0")/../bin/my_node"

Checking a --user unit’s state

The unit belongs to the hook user’s manager, not to yours. So systemctl --user status myapp.service, run over ssh as your login user, answers Unit myapp.service could not be found. The push can be correct, and the service can run. Ask with the hook identity:

atc shell robot@device -- systemctl --user status myapp.service

atc shell reconstructs the hook environment, XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS included. A plain sudo -u atc-hooks systemctl --user ... sets neither variable, and answers Failed to connect to bus. Set XDG_RUNTIME_DIR yourself where atc shell is not available:

sudo -u atc-hooks env XDG_RUNTIME_DIR="/run/user/$(id -u atc-hooks)" \
  systemctl --user status myapp.service

Reading a --user unit’s logs

journalctl --user -u myapp.service, run as atc-hooks over atc shell robot@device, can answer No journal files were found while the unit is actively logging. The user session’s journal is not always reachable from a shell that never logged in. The records still reach the system journal, where each one carries the user unit name. Read them from outside:

sudo journalctl _SYSTEMD_USER_UNIT=myapp.service _UID="$(id -u atc-hooks)" -f

Name the two fields. journalctl also has a --user-unit= flag, which matches on the caller’s own session, so a root shell needs the explicit form above. For every unit of that user at once, ask for the manager instead:

sudo journalctl -u "user@$(id -u atc-hooks).service" -f

An empty journal is not a dead service. systemd gives the unit a pipe, not a terminal. A Python or C runtime buffers stdout in blocks on a pipe. A line then sits in the buffer until the buffer fills or the process exits. A rospy node that prints one line per message can show one startup line for hours. It handles every message correctly.

Which answer applies depends on how the node writes the line.

• print() and a bare write reach stdout, and nothing else. Set Environment=PYTHONUNBUFFERED=1 in the unit. The journal then gets each line as the node writes it. For a C program that the unit starts, wrap the command in stdbuf -oL. stdbuf cannot change a Python program, because Python sets its own buffering.

• rospy.loginfo and the other framework loggers reach the framework’s own files instead. ROS writes them under ~/.ros/log/<run_id>/. Read them as the hook user:

  atc shell <target> -- sh -c 'ls -t ~/.ros/log/'
  

  Quote the command. The tilde must expand on the device, and your own shell expands an unquoted one first. Sort by time rather than reading latest. The latest symlink is a roslaunch convenience, so a node the unit starts directly can leave none.

A node can use both, and each line reaches one of the two places. So read both before you call the service broken.

A ROS 1 roslaunch service logs almost nothing there for a second reason. roslaunch redirects each node’s output into its own files under ~/.ros/log/<run_id>/. So output="screen" never reaches systemd’s capture. Read ~/.ros/log/latest/ for the node’s own logs. The journal carries only roslaunch itself. An unbuffered stdout does not change that.

A service that needs time to stop

At shutdown, systemd stops the hook user’s systemd --user manager (user@<uid>.service). The manager then stops its own units. A unit cannot outlive its manager. So the manager’s TimeoutStopSec caps the stop time of every hook-launched user service, whatever the unit’s own TimeoutStopSec says.

Ubuntu and Jetson L4T set that cap to 5 s. They ship /usr/lib/systemd/system/user@.service.d/timeout.conf with TimeoutStopSec=5. A service that saves state on SIGTERM and needs longer is killed before the save ends.

By default, atc bootstrap atcd install raises the cap to 180 s for the hook user only. It writes /etc/systemd/system/user@<uid>.service.d/zz-atc-hook-user-stop-timeout.conf, where <uid> is the numeric uid of the hook user. The file name sorts after timeout.conf, and systemd applies the drop-in that sorts last. The install then reads the value back with systemctl show. It prints a warning when another drop-in still sets a different value.

Other users’ managers, and the user@.service template, keep their values. A daemon-reload applies the change to a running manager, so the hook user’s manager does not restart.

Bootstrap writes no drop-in when hooks run as the daemon user, as with --root. atc bootstrap atcd upgrade does not write it. Run atc bootstrap atcd install again on a device installed before this change. atc bootstrap atcd uninstall removes the file when it removes the hook user: on a purge, when no other atcd store uses that user. It removes only a file whose first line is the bootstrap header. --keep-store keeps the hook user, and the file with it.

The install records the uid that it wrote the drop-in for, in /etc/atc/<service>.hook-user-stop-timeout-uid. The hook user can change in three ways:

• --set daemon.hook_run_as_user=<user> names another hook user.
• Hooks run as the daemon user again.
• The account is deleted and created again with a new uid.

Then the next install, and a purge, remove the drop-in of the recorded uid. They remove only that one path, and only a regular file with the bootstrap header. They keep it when the record of another atcd unit names that uid. They never remove a drop-in of any other uid.

--keep-store keeps the drop-in and the record. A purge keeps the record while the drop-in it names stays. Three limits remain:

• A failed first install rolls back the unit, but it keeps the drop-in and the record, as it keeps the hook user.
• A device installed before the record has none. Its next install writes one. If that install also changes the hook user, the drop-in of the former uid stays. Remove it by hand.
• Two atcd units can share one hook user. If the other unit was installed before the record, it has no record that names the uid. A change of the hook user of this unit then removes the shared drop-in. The next install of the other unit writes it again.

Two flags of atc bootstrap atcd install change the drop-in:

• --hook-user-stop-timeout <SECS> writes the drop-in with that value. The value must be from 10 to 3600. Above 30 min, poweroff.target stops waiting (JobTimeoutSec=30min) and powers the host off.
• --no-hook-user-stop-timeout removes the drop-in, then runs systemctl daemon-reload, so the vendor’s value applies again. A drop-in that sorts after it still applies. It removes only a file whose first line is the bootstrap header, and it keeps a symlink at the path. With no file, it changes nothing.

The two flags conflict, and each one refuses --root. When hooks run as the daemon user, there is no hook user’s manager. Then a flag changes nothing, and the install prints a note that says so.

The flags are not saved. Each install sets what its own flags say. An install with no flag writes the 180 s drop-in again, also after --no-hook-user-stop-timeout. So give the flag to every install of the device.

atc bootstrap atcd upgrade does not read the flags, and it does not change the drop-in. A repair by atc ota push of an existing unit does not change the drop-in either. When atc ota push installs atcd on a device for the first time ([bootstrap].auto), the install writes the 180 s drop-in.

Check your units against the value you set:

• Set the unit’s TimeoutStopSec to the longest stop you expect. The cap must be at least that value plus a margin of about 30 s.
• A unit with ExecStop= can take about twice its TimeoutStopSec. systemd applies the timeout to each ExecStop= command and again to the wait after SIGTERM.
• Units that are not ordered against each other stop in parallel. The slowest ordered chain counts, not the sum.

If a unit needs more than 180 s, use --hook-user-stop-timeout. You can also add a drop-in for the hook user that sorts after zz-atc-hook-user-stop-timeout.conf, for example zzz-local-stop-timeout.conf. The install keeps your value. It prints the warning only when it runs its script, and an install that finds nothing to repair runs none. atcd install reads the file, not the value systemd holds.

With or without a flag, the install keeps a regular file at the drop-in path that bootstrap did not write. --no-hook-user-stop-timeout also keeps a symlink there. An install that writes a value, with or without the flag, refuses a symlink and stops. The install names a kept file only when it runs its script.

These points are yours to check. atc does not and cannot verify them:

• Late or network mounts. user@<uid>.service stops before local fstab mounts unmount. A unit that writes to a network mount, an automount or a late mount at stop needs ordering that atc does not add.
• Other users’ managers. The drop-in covers only the hook user. A workload that runs in another user’s manager keeps that manager’s timeout.
• Container stop timeouts. podman stop sends SIGKILL after 10 s by default, whatever systemd allows. A unit that runs a container must raise it: podman run --stop-timeout, podman stop -t, or StopTimeout= in a Quadlet file.

systemctl --user needs more than linger

Bootstrap enables linger, which is necessary and not sufficient. A rootless systemd --user manager also needs dbus installed, and cgroup delegation for user slices. A nested container usually has neither. Docker-in-Docker is the common case, and a device image built that way behaves the same. The manager then fails with status=219/CGROUP, however correct the unit is. A missing dbus on its own gives Failed to connect to bus.

A hook cannot fix either. The next section gives the two supported paths for such a device.

Root work on a device with no user session

This section is for a device with no dbus or no systemd --user manager. On such a device a hook cannot drive systemctl --user. Use one of two supported paths, in this order.

1. Give the hook user a supplementary group. Use this path when the hook must read or write a root-owned file, socket, or device node. Add the hook user (atc-hooks by default) to a group that owns the resource. Then give that group the access. When the hook user differs from the daemon user, atcd reads the group set at every spawn, so you need no daemon restart. When hooks run as the daemon’s own user, as on the hand-launched no-systemd recipe, they inherit the daemon’s groups. Then restart atcd after the group change. The recipe, with the K3S kubeconfig as the example, is in the atcd non-root runtime model.
2. Ship a root-owned system unit in a .deb. Use this path when the work must run as root, for example a service that writes to /usr or /etc. This path needs systemd as PID 1, for bootstrap and for the unit itself. Bootstrap the device with atc bootstrap atcd install --root. atcd then runs as root, and its hooks inherit root. The hooks install the .deb and start its unit with systemctl. The complete producer tree is examples/deb-systemd-service/.

A root hook user under a non-root daemon is refused by design. atc refuses hook_run_as_user = "root" while run_as_user is not root. The error is ATC-E0506. This is a decision, not a gap (#4785). There is no opt-in that lifts it. To run hooks as root, run atcd as root with path 2.

A sudoers rule that lets the hook user call sudo is not a supported path for this device shape. atc does not install, test, or check such a rule.

Ubuntu 24.04 and user namespaces

Ubuntu 24.04 sets kernel.apparmor_restrict_unprivileged_userns=1. With that setting, a binary needs an AppArmor profile that grants userns before it can use an unprivileged user namespace. Without one, the kernel confines the process to the unprivileged_userns profile and strips its capabilities.

Ubuntu ships such profiles for podman, buildah and crun. It ships none for skopeo. So podman works for the hook user, and skopeo copy fails with unshare: Operation not permitted. atc oci import from a registry runs skopeo, so a first push can fail this way while every rootless check passes.

Install a profile for skopeo. It is Ubuntu’s podman profile with the name and the path changed:

sudo tee /etc/apparmor.d/skopeo > /dev/null << 'EOF'
abi <abi/4.0>,

include <tunables/global>

profile skopeo /usr/bin/skopeo flags=(unconfined) {
  userns,

  include if exists <local/skopeo>
}
EOF
sudo apparmor_parser -r /etc/apparmor.d/skopeo

A file under /etc/apparmor.d loads again at boot. atcd ctl setup-rootless --user <name> reports check R6 as WARN while the sysctl is 1 and the kernel lists no skopeo profile in /sys/kernel/security/apparmor/profiles. It also warns when the file changed after the kernel loaded it, and when the file does not attach to /usr/bin/skopeo. The file alone is not a pass. A profile the parser refused, or that nobody loaded, leaves skopeo failing. The delivered agency/podman bundle needs no extra step: setup-rootless --provision installs /etc/apparmor.d/atc_podman for it.

Do not set the sysctl to 0. That lifts the restriction for every unprivileged binary on the host, and it hides the missing profile instead of adding it. Ubuntu 20.04 and 22.04 have no such sysctl and need no profile.

What the push checks after activate.exit

A hook can exit 0 and still leave a unit that does not run. So after a push that activates a bundle, atc ota push asks the hook user’s systemctl --user about the unit of that bundle. It polls every half second for up to 10 seconds. A unit counts as running when it stays active for 2 seconds with no restart. A slow starter therefore does not fail the first poll.

A unit that is not running gives a WARNING in the push summary. The warning names the unit and its last state, for example failed, activating (auto-restart) or restarting. The push still succeeds, the exit code does not change, and atc reverts nothing. --json carries the same list as inactive_units.

The push finds the unit in one of two places:

• An atc.kdl systemd <name> { from <bundle>; unit "<unit>" } workload. The push checks that unit when it activates <bundle>. If systemd does not know the unit, the warning reports not-found.
• Otherwise, the unit named <bundle>.service, which is the name this example uses. The push checks it only when the hook user’s manager has it loaded. A bundle with no such unit gets no check and no warning.

The check says nothing when it cannot run: a push over --socket, or a device with no user session. No poll starts with less than 2 seconds of the budget left. When a later poll fails, the push still reports each unit an earlier poll saw failed, inactive, not-found or restarting. It drops a unit last seen activating, because that unit may be a slow starter.

What this example does not cover

• A root-owned system unit, and a .deb. Use examples/deb-systemd-service/, a complete producer tree with its own e2e test. That path needs atc bootstrap atcd install --root, because a .deb writes to /usr and /etc, and the unprivileged hook user cannot.
• A device image with no user-session support. systemctl --user needs libpam-systemd and dbus-user-session. Bootstrap probes for both and reports what it found. On a device that has neither, systemctl --user answers Failed to connect to bus. For the supported paths on such a device, see Root work on a device with no user session.
• The systemd <name> { unit ... } workload syntax. It parses and it persists, but nothing starts it. See modules-today.md. Hooks are the mechanism that works today.

Debugging hooks

Hook stdout/stderr is recorded on the controller once the hook is dispatched: one <state>.<phase>__<hook-id>.{stdout,stderr} pair per hook run, under $XDG_CACHE_HOME/atc/hooks/<timestamp>-<target> (~/.cache/atc/hooks/… by default — the exact path is printed at push start). Pass --capture-hooks <DIR> to pin a specific directory; recording is unavailable only under --json.

spawn_error means the exec itself failed — the hook never ran, so there is no exit status. The OS error is still reported: the CLI prints it as a [stderr] line on the failure replay, and it is written to the hook’s .stderr capture file. Look for spawn failed: Exec format error or spawn failed: Permission denied. The same text is also available after the fact, base64-encoded in stderr_tail_b64, from atc ota status --json. Triage the usual causes on the device:

• missing or bad shebang (ENOEXEC / Exec format error) — check head -1 of the shipped script under the artifact’s version dir;
• missing exec bit on the shipped file;
• hook-user setup failure (see the hook-user paragraph above).

Warning: running the script by hand over SSH masks a missing shebang — bash falls back to interpreting ENOEXEC files as sh scripts, so a script that “works” when run manually can still spawn_error under atcd, which exec’s it directly.

Run a hook out-of-band without a push:

atc ota run-hook user@robot commit.exit

-o NAME narrows to one declared artifact, spelled as ota push’s selector (--artifact stays as a hidden, deprecated alias). Without it, run-hook picks the single artifact declaring the cell and refuses when several do.

Which scope run-hook reaches. A hook declared under an artifact ([bundle.lifecycle.hook.activate.exit]) is read from that version’s .atc/lifecycle.pb. A stage hook — top-level [lifecycle.hook.<state>.<phase>] — is bound to no artifact, so it is read from the active delivery’s .atc/delivery.pb archive instead, the same file the gc.* bookends live in. run-hook consults that archive only when NO artifact declares the cell. So when the same cell is declared at both scopes, run-hook runs the artifact hook, and the stage hook is not directly reachable — a push runs both, run-hook runs one.

A stage hook is archived by the delivery that declared it. A delivery committed by an atcd older than this behaviour wrote no stage table, so run-hook reports no hook configured for its stage cells until the next push or poll re-commits the delivery.

A stage hook run this way sees no delivery members. On a push or a poll, a stage hook’s context lists the artifacts of the delivery being applied, each with the version it is moving from and the version it is moving to. run-hook applies no delivery, so it reports an empty list rather than a made-up one. Write a stage hook that must also be reachable by hand so that it reads the device state it acts on, not the member list.

run-hook requires a bootstrap-provisioned device (the atc-hooks user plus its NOPASSWD sudoers drop-in); on a manual no-systemd install it fails with “device not provisioned for a hook shell” — trigger the hook via a push instead, or exec the script from the version dir with ATC_HOOK_CONTEXT set by hand.

On a no-systemd device (where run-hook is unavailable), confirm a hook ran by triggering a push and checking the hook’s own side effect on the device (e.g. its marker file) alongside the captured stdout/stderr under ~/.cache/atc/hooks/ on the controller.

Testing hooks without a device

atc ota push --local runs the whole push against a throwaway atcd on your own machine. There is no target, no SSH, no bootstrap and no root:

atc ota push --local --switch --hook-output=always

It makes a test root, starts a real atcd on it, pushes, and stops the daemon again. Because it is the real daemon, a hook that passes here ran against the real executor — same ATC_HOOK_CONTEXT, same working directory, same timeout, same exit-code rules. A hook that exits non-zero vetoes the delivery, and the command exits non-zero with it.

--switch or --no-switch is required, and --local will not guess. --switch activates the pushed version, so the activate and commit hooks run too. --no-switch stages, which runs the download and install hooks and stops. Guessing would either report half a hook run as a whole one, or override a project whose own policy is to stage.

Keep the root to look inside it. Give --local-root an absolute path. A relative path makes every hook fail to spawn (spawn failed: No such file or directory (os error 2)), and the failure vetoes the whole push:

atc ota push --local --local-root /tmp/hook-test --switch --hook-output=always
ls /tmp/hook-test/store    # the store atcd wrote
cat /tmp/hook-test/atcd.log  # what the daemon said
ls /tmp/hook-test/hooks    # each hook's recorded stdout and stderr

Keep the path short as well. The root holds atcd’s Unix socket, and a socket path may not exceed 103 bytes. A longer one is refused up front with ATC-E0463.

A root you name is created and kept, and holds everything about the run. A root you do not name is a temp directory that goes away with the command. A root that already holds something else is refused, so --local-root . cannot scatter a store through your project.

To activate a version that a --no-switch push staged in a kept root, run atc ota switch --local-root <DIR> with the version or --delivery-id. The push prints that command. The switch starts atcd on the root for this command only, so the activate and commit hooks run as they do for --switch. A temp root holds nothing after the push, so that push prints no command.

Four things this does not do.

• It is not a sandbox. The test root relocates atcd’s store and nothing else. Hooks run under your uid, in your home, against your user services. A hook that writes an absolute path writes it here, on this machine. The command prints this every time it runs.
• A hook that is still running when the command ends is not killed. Every hook gets its own process group and only atcd knows them, so killing the daemon leaves a slow hook running. It keeps making changes on this machine after the command has returned.
• It is not the device’s hook user. On a robot, hooks drop to atc-hooks. Locally there is no privileged daemon to drop from, so they run as you. A hook that works only because of something your account has can still fail on a device — and one that needs the service account’s home can fail here and pass there.
• Pushing the same content twice runs no hooks at all. The daemon short-circuits a push whose content is already active. Change something between runs into a kept root, or the second run does nothing and reads as a broken feature. A changed second push is also the only way to reach the deactivate hooks on a push, which need a previous version to deactivate. atc ota switch back to the earlier version reaches them too.
• It needs an atcd for this machine. Released atcd builds are Linux only. atc looks for atcd beside itself, then on PATH; ATC_LOCAL_ATCD names one directly.

--local takes no target, no --socket, no --ssh-port and no --dry-run: each of those names a device, and local mode is the absence of one. Two local pushes cannot share one root — the second is refused rather than becoming a second writer on the same store.

Common mistakes (all now rejected at atc ota push)

These shapes used to parse with no error and wire no hook — push still reported success and the hook never ran. They are now rejected with a message that shows the correct shape:

lifecycle = "x"                         # lifecycle must be a table, not a scalar
lifecycle = []                          # ...not an array
lifecycle.commit.exit = 12345           # missing the required `.hook.` segment
[lifecycle.commit.exit]                 # missing `.hook.` — `commit` is unknown
[lifecycle.hook.commit.exit]
run = "do-thing.sh"                     # wrong body key — it's `script`, not `run`

The correct form for the last one:

[lifecycle.hook.commit.exit]            # stage scope: @self forbidden, use @<name>/...
script = "@myapp/scripts/x.sh"

…or, more commonly, artifact-scoped under the bundle:

[bundle.lifecycle.hook.commit.exit]
script = "@self/scripts/x.sh"