ATC documentation

atc ships software to robots over SSH. You push a versioned bundle or an OCI image, the device checks every byte against its digest, and you switch to it when you’re ready. If the new version misbehaves, you switch back. The device keeps recent versions on disk, as many as its retention settings allow, so a rollback to one of them downloads nothing.

New here? Start with Onboarding. You install a released atc binary, with no repo to clone and nothing to build, and ship your first verified update in six steps.

Also useful:

• The one-page command cheatsheet, also as a PDF.
• The source, the issue tracker and the contributor docs are in the agency-code repository.
• Agency Tool Company builds atc. Our site says what we do and how to reach us.

Onboarding: install atc and ship your first update

Who this is for: you have a device (a robot, a Linux box, a container) and you want to deliver software to it from your own machine using the atc CLI. You do not need this repository, and you do not build anything from source — you install a released, signed atc binary and go. This is the fast path from zero to one real, verified update. It assumes you can use a terminal and SSH, and nothing else about atc.

Working in this codebase (contributing)? Start at building-from-source.md instead. That path builds the binaries from source.

Mental model

• atc — the controller. Runs on your machine (laptop / CI). It delivers artifacts to a device over SSH. This is what you install below.
• atcd — the daemon. Runs on the device. It owns a content-addressed store at /var/atc and applies updates atomically. On a systemd device, atc installs it for you, or the device installs it with atcd install (Path A below). On a device without systemd as PID 1, atcd install --no-service seeds it with no unit, or you stage it by hand (Path B).
• Every push is versioned and atomic: bytes are staged first, then the active version is flipped only when you ask with --switch. Rollback is just a flip back to a prior version — nothing is rebuilt or re-downloaded.
• You declare what to ship in an atc.toml manifest and run atc from that directory.

The steps below are sequential — run them in order.

Two pages cover this journey. This one is the complete path: both device types, lifecycle hooks, container images, rollback, and the traps each step hides. deploy-your-first-update.md is the short version of the same journey, with a troubleshooting table. Read one of them, not both.

1. Install atc (on your machine)

curl -fsSL https://atc.dev/install.sh | sh

This drops the atc binary at ~/.local/bin/atc. If that directory isn’t on your PATH, add it before anything else — otherwise the next command fails with atc: command not found:

export PATH="$HOME/.local/bin:$PATH"   # this shell only

If each step runs in a fresh shell (CI, a script, a new terminal), this export won’t carry over. In an interactive shell, add the line to ~/.bashrc/~/.profile now so every later atc invocation finds the binary. But a non-interactive runner (CI, a headless script — bash invoked as sh -c …) sources neither file, so there the reliable fix is to set PATH explicitly in each step, or just call the binary by its full path ($HOME/.local/bin/atc). Inside one script a single export at the top holds for the rest of that script. It is per-step runners, such as CI jobs or separate sh -c invocations, that need it repeated.

Then sanity-check:

atc --version

The one-liner does two things: it fetches the atc binary over TLS (verifying its sha256 digest against the index — transport + integrity), then runs atc upgrade as its final step. That self-update re-checks the signed index — now verifying an ed25519 signature over it against a key pinned inside the binary (cryptographic authenticity) — and provisions a bundled skopeo into the atc-store, the tool atc shells out to for OCI artifacts (atc oci import/push; used only if you ship a container image, see the aside in step 3). So the provisioned skopeo … line you see scroll past during install is expected, not a stray side effect, and the ed25519 check has already run by the time the one-liner exits — it is not a separate step you trigger later. (It’s best-effort: a network or cross-arch hiccup, or a custom ATC_BASE_URL, skips it with a note — re-run atc upgrade yourself to provision skopeo.) Full trust story: verifying-the-binary.md.

The install script honors a few env vars — ATC_BIN_DIR (where to install), ATC_TRACK (which release track to fetch), ATC_BASE_URL (which index origin to fetch from). These configure this install only; they don’t change where a later atc upgrade self-update pulls from.

Put each variable after the pipe, on sh. The script reads it there. To install from the nightly track:

curl -fsSL https://atc.dev/install.sh | ATC_TRACK=nightly sh

ATC_TRACK=nightly curl … | sh sets the variable for curl only. The script does not see it and installs stable, with no error.

Which release track should you use? stable is the default and the right answer for a first install. The nightly and internal dev tracks, version pinning, and the token the dev track needs are all in Appendix: release channels, at the end of this page. Skip it for now.

2. Bootstrap the daemon onto the device

Every command from here on uses deploy@device as the SSH target. Replace it with your own login user and the device’s address, for example pi@192.168.1.5, or with a Host alias from ~/.ssh/config.

Preflight — which path applies to you. atc bootstrap installs atcd as a systemd service, so systemd must be PID 1 on the device. Check that first — it decides which of the two paths below you take:

# accept-new: this is the doc's FIRST contact with the device. Without it the
# host-key prompt hangs a non-interactive shell (details in the note below).
ssh -o StrictHostKeyChecking=accept-new deploy@device 'ps -p 1 -o comm='   # systemd → Path A ; anything else → Path B

This asks what runs as PID 1 (the init process), not whether systemd is installed. A box can carry the systemd tools yet boot a different init — e.g. sshd as PID 1 in a container — and that’s Path B.

• Prints systemd → Path A (the atc bootstrap command just below).
• Prints anything else — a bare container, a minimal image → Path B: don’t run the bootstrap command (it aborts with ATC-E0419, installing only the systemd unit); use the foreground recipe in Path B instead.

Don’t infer the path from the OS or distro — a perfectly ordinary-looking Debian/Ubuntu host can still be Path B if it runs without systemd as PID 1 (e.g. inside a container-based sandbox). Always run the ps check; never assume from how the box looks.

First SSH to a fresh device? The initial connection triggers a host-key trust prompt that hangs a non-interactive run — it fires before any sudo or password prompt. The robust fix is to set StrictHostKeyChecking=accept-new for the host in ~/.ssh/config — it seeds the key on first connect no matter how the device is reached. You can instead pre-seed with ssh-keyscan device >> ~/.ssh/known_hosts (bare host, no deploy@), but ssh-keyscan ignores ~/.ssh/config aliases and talks straight to DNS on port 22 — so for a device reached through a config alias, a ProxyJump, or a forwarded port it fails with getaddrinfo … name resolution (or silently returns nothing). Ask ssh what an alias resolves to — ssh -G <alias> | grep -E '^(hostname|port) ' — then ssh-keyscan -p <port> <real-host>. That covers a renamed host or a forwarded port, but not ProxyJump: ssh-keyscan takes no proxy option and reads no config, so it dials the target directly and either hangs or grabs the wrong key. For a device behind a jump host, use accept-new — the one approach that works whatever the config does.

Concretely: with Host myrobot / HostName 192.168.1.1 / ProxyJump jumphost in your config, ssh myrobot reaches the device through the jump host but ssh-keyscan myrobot tries to resolve myrobot in DNS and fails, because it reads neither the alias nor the ProxyJump. ssh -o StrictHostKeyChecking=accept-new deploy@myrobot … takes the same route ssh already takes, so it works.

Path A — systemd device

atc installs and starts atcd on the device over SSH:

atc bootstrap atcd install --yes --non-interactive deploy@device

Because you installed atc with the one-liner (no repo, no locally-built daemon), atc auto-downloads the arch-matched atcd from the same signed release origin and verifies it (ed25519 signature + sha256) before it is ever copied to the device. This works automatically for linux-x86_64-musl and linux-aarch64-musl devices; for any other device arch (32-bit ARM, RISC-V, …) supply a binary yourself with --from <path/to/atcd>.

• --enroll: enroll the device with a control plane in the same command. The install mints a single-use enrollment token and stages it on the device. No file on your machine holds the token. The server and org come from --server and --org, else the repo’s atc.toml [server] table, else your atc auth login. To use a token minted ahead with atc org enroll-token (one token for many devices, or a token for another person), pass --enrollment-token-file <path> instead.
• --yes (-y) skips the confirmation prompt; --non-interactive is key-only SSH with no prompts. Use both in automation — a sudo or SSH password prompt hangs in a non-TTY shell. Passwordless sudo + key auth on the device is strongly preferred.
• --setup-rootless: delivered podman in one command. Without it, the install leaves delivered podman not set up. atcd ctl podman env then refuses with no_recorded_identity, and atc shell has no podman on PATH. With the flag, bootstrap runs three steps after the install, in this order: atcd upgrade as the store owner, atcd ctl setup-rootless --provision, and atcd ctl setup-rootless --apply. The owner comes from User= in the unit, so a --root install updates as root. The setup-rootless steps install setuid newuidmap and newgidmap helpers, so the flag is opt-in and nothing else runs them for you. It sets up the account that the unit runs hooks as (atc-hooks on a new install). Name another account with --rootless-user <name>. A lifecycle hook gets the delivered engine only when it runs as that account. With --root there is no hook account, so you must pass --rootless-user. A failed step stops the run and names the step. The install stays in place, and you run the same command again. Each step is safe to repeat.
• A slow or flaky link. When the SSH connection fails, bootstrap tries the step again, up to 5 attempts per step and 120 s of waiting per run. It prints one bootstrap: ssh to … retrying line for each attempt. A step that changes the device runs again only when ssh never reached the device. If the connection drops while such a step runs, bootstrap stops with ATC-E0303 and says that the step possibly ran. Run the same command again. A dropped link never removes an install that already started.
• The boot-confirm guard. The install adds a guard that systemd runs before each atcd start. It counts the starts. When a new atcd version fails to reach a healthy ready state on more starts than the guard’s limit (ATCD_BOOT_CONFIRM_LIMIT), the guard rolls back to the previous version. Later sections use this name for it. The guard writes to the atcd journal, so read it with ssh deploy@device 'sudo journalctl -u atcd'. A rollback shows as atcd-boot-confirm: <N> attempts > <limit> — rolling back to prior atcd, then atcd-boot-confirm: reverted to prior atcd.

You do not name the device. A hostname, an IP address and localhost are all valid targets. atcd names itself, so the name does not depend on how you reached the box. It uses a word pair and a 6-character hex suffix (fluffy-kitty-3fa9c1), made from a hash of the hardware identity that bootstrap reads during install. The same hardware gets the same name back after a reflash. A box with no usable hardware identity gets a word pair made from its device id instead. A box that already has a name keeps it.

ATC-E0419 with no systemd means the device has no systemd as PID 1. The message prints a foreground recipe, which is Path B below.

Path A on the device: atcd install

Use this when the device installs atcd itself, for example from a provisioning script, a first-boot job or Ansible. It needs no atc anywhere. The device needs curl, OpenSSL 3, awk, and sha256sum or shasum. Without systemd as PID 1, the script installs with --no-service (Path B). Run on the device:

curl -fsSL https://atc.dev/install-atcd.sh | sudo sh -s -- --socket-owner deploy

The script downloads the atcd for the device’s CPU. It checks the ed25519 signature on the release index and the sha256 of the binary. Then it runs atcd install with your arguments. If a check fails, it stops before it runs atcd.

atcd install does the same install as atc bootstrap atcd install, and it takes these flags:

More than one store on a device. An install that changes the device points /usr/local/bin/atcd at its own store, so the install that ran last owns the link. A link to the atcd of another store is not a difference. --check reports the symlink action only for no link, a file in place of the link, or a link that names no file. --link also reports it when the link names another store.

To add a second store and keep the link where it is, pass --no-link. That install does not make the link, change it, check it, or remove it on a rollback. Its rollback also keeps the boot-confirm guard, because the first install uses it too. It removes the atc shell sudoers file only when that run made the file.

A --no-link install with hook isolation writes the atc shell sudoers file only when the file is absent or holds the grant of its own accounts alone. Any other file stays as it is. When that file does not grant the accounts of the new store, atc shell of it exits 4. atcd install, --check included, prints a kept /etc/sudoers.d/atc-shell line for it. --check reports no action for it, because no run changes it. A path there that is not a regular file stops the install.

The store saves --no-link as an empty no-link directory in the store root. It stays across a reboot and an atcd update. A later atcd install of that store keeps the choice with no flag. atc bootstrap atcd upgrade of that store does not point the link at it, and atc bootstrap atcd uninstall of it keeps the link and the guard.

The uninstall removes the sudoers file only when two things are true. The file holds the grant of that store, and no other atcd store on the device uses the same accounts. A store that saved --no-link keeps the file when the uninstall keeps the accounts, as --keep-store does. When the uninstall deletes the accounts, it removes their grant too. A store with no hook isolation never removes it.

The prompt lists the file as removed only when the run removes it. When the prompt cannot read a fact, its line names every condition.

--check reports the no_link action when the saved choice differs from the run’s: --no-link on a store with no saved choice, or --link on a store with one. --link removes the saved choice and takes the link. atc bootstrap atcd install --no-link and --link pass the flag to atcd install on the device:

sudo atcd install --no-link --store-root /var/atc2 \
  --set daemon.service_name=atcd2 --socket-path /run/atcd2/atcd.sock

For atc to reach that store, give it a link of its own, and set [daemon] remote_path in the repo’s atc.toml to that link. Do not set remote_path to the store’s binary itself: atc bootstrap atcd upgrade points remote_path at that binary, and would replace the binary with a link.

sudo ln -s /var/atc2/channel/agency/atcd/delivery/current/atcd/atcd /usr/local/bin/atcd2

atcd install is safe to run again. On a device that already has the correct install, it changes nothing. On a damaged install, it repairs only the difference, so the repair is the same command. The exit code tells you the result:

atcd install refuses, with exit 1, on a device without systemd unless you pass --no-service (see Path B). It also refuses when a systemd drop-in sets ExecStart for the unit, because the install cannot own a command that a drop-in replaces. The message names the drop-in file. Remove it, then run the install again.

install-atcd.sh exits 1 if a download or a check fails. Otherwise it exits with the atcd install code. These variables change what it fetches. Put them after sudo, as sudo env ATC_TRACK=nightly sh -s -- …, because sudo drops your environment.

Each track also has its own copy of the script, which defaults to that track: https://atc.dev/agency/nightly/install-atcd.sh. The copy at https://atc.dev/install-atcd.sh changes only on an official release.

With Ansible. One task covers a normal run and --check mode. In check mode it passes --check, so atcd install changes nothing and exits 3 on a difference:

- name: Install atcd
  become: true
  ansible.builtin.shell: >-
    set -o pipefail &&
    curl -fsSL https://atc.dev/install-atcd.sh |
    sh -s -- --json --socket-owner deploy {{ '--check' if ansible_check_mode else '' }}
  args:
    executable: /bin/bash
  check_mode: false
  register: atcd_install
  changed_when: >-
    atcd_install.rc == 3 or
    (atcd_install.rc == 0 and (atcd_install.stdout | from_json).changed)
  failed_when: atcd_install.rc not in [0, 3]

check_mode: false lets the task run in check mode, which is safe because it then passes --check. Keep pipefail. Without it, a failed download feeds sh an empty script, which exits 0. To install a binary that is already on the device, run atcd install --json … in place of the curl pipe.

Name the device

atcd keeps the device name in <store_root>/device_name. On a default (root) install that is /var/atc/device_name. A rootless install uses its own store root. The file holds one name as plain text. atcd trims surrounding whitespace. atcd ignores an empty value, an IP address and a loopback name such as localhost.

atcd resolves the name at every start and again when it enrolls. The enrol request carries the resolved name. atcd picks the first match in this list:

1. The name the server returned after a --replaces reinstall or a rename.
2. An atc.device seed from a unit an older bootstrap wrote.
3. The file <store_root>/device_name.
4. A name made from the hardware digest, such as adjective-animal-xxxxxx.
5. A word pair made from the device id.

atcd writes the winning name back to the file when the file is missing or differs. On a first start with no file, atcd therefore writes the hardware-derived name. To choose the name, write the file before the atcd unit first starts. Do this in the factory image or in a first-boot step. The store directory must exist.

sudo install -d /var/atc
printf '%s\n' robot-07 | sudo tee /var/atc/device_name

This works for the default store root. The install refuses a store root other than /var/atc that already holds files and is not an atcd store.

The hardware digest comes from the DMI UUID or the device-tree serial. Containers on one host share the host UUID. Cloned VM images keep the UUID of the source. Devices like these derive the same name, so set the file on each one.

The server keeps the name a device enrolled with. If you edit the file later, only the name in telemetry changes. After a --replaces reinstall or a rename, the name from the server comes first, so an edit changes nothing.

To rename a device after it enrolled, use the console or the CLI. In the console, open the device and select Rename. On the command line, run:

atc fleet device rename <device-id> robot-07

The server refuses a name that another device in the org has. It also refuses an empty name, an IP address and a loopback name such as localhost. The device picks up the new name at its next check-in. Its diagnostic reports use the new name at once. Its telemetry uses it from the next start of atcd.

If another device in the org already holds the requested name, the server enrolls the new device as <name>-<last 6 hex digits of its device id>. It returns a warning in the enrol reply. atcd logs that it was renamed and saves the new name. Telemetry reports the new name from the next atcd start. The run that enrolled keeps the name it started with. A retry of the same enrolment is not renamed, and a --replaces reinstall is not renamed. To keep the identity of a reflashed box, use atc bootstrap atcd install --replaces <device>.

Path B — not systemd (containers, minimal images)

On the device, with atcd install --no-service. It seeds the store and installs the boot-confirm guard, writes no unit, and calls no systemctl. It has the same --check, --json and exit codes as Path A. install-atcd.sh adds the flag itself when PID 1 is not systemd:

curl -fsSL https://atc.dev/install-atcd.sh | sudo sh -s -- --socket-owner deploy

This line runs on the device, so the device needs curl, OpenSSL 3, awk, and sha256sum or shasum. Without curl the installer does not start, and only the device shell reports that curl is not found. When the installer starts, it stops and names the first other tool that is missing. A device without these tools takes the by-hand flow below.

Nothing supervises atcd here. On success, the install prints the one command that starts it in the foreground, and --json carries it as start_command. Run that command under your own supervisor (the container entrypoint, for example).

From your machine, by hand. atc bootstrap won’t stage atcd here (it aborts at the systemd preflight), so you stage it by hand. Two steps: fetch the arch-matched atcd onto your machine, then start it on the device in the foreground.

1. Fetch atcd from the release origin (the same signed index the installer used — this is exactly what bootstrap does for you on a systemd device). Run on your machine; needs jq. Run it with bash, not sh. The block uses a bash array (auth=()), which dash rejects as a syntax error.

# Pick target to match the DEVICE's CPU (not your machine's) — a wrong-arch
# push looks identical in `ota status` later, so get it right here:
case "$(ssh deploy@device 'uname -m')" in
  x86_64)  target=linux-x86_64-musl ;;
  aarch64|arm64) target=linux-aarch64-musl ;;
  *) echo "unsupported device arch — copy your own atcd to deploy@device:/tmp/atcd" >&2; exit 1 ;;
esac
# Read the track your atc came from. Every release prints it in `atc --version`.
# `atc version --json` first shipped in 0.1.5. On 0.1.4 and older it fails with
# "unrecognized subcommand 'version'", so this line parses `--version` instead.
track=$(atc --version | sed -n 's/.*track \([a-z][a-z]*\)).*/\1/p')
: "${track:?could not read track from 'atc --version'. Set it by hand, e.g. track=nightly}"
# The internal `dev` track is token-gated; without the header a dev-stamped atc
# gets a bare 403 here. Send it ONLY for that track and only to atc.dev, mirroring
# what `atc` itself does — a credential should never ride a request that does not
# need it.
auth=()
if [ "$track" = dev ] && [ -n "${ATC_CHANNEL_TOKEN:-}" ]; then
  auth=(-H "X-Atc-Channel-Token: $ATC_CHANNEL_TOKEN")
fi
idx=$(curl -fsSL "${auth[@]}" "https://atc.dev/agency/$track/index.json")
url=$(printf '%s' "$idx" | jq -r --arg t "$target" '.artifacts."agency/atcd"[$t].url')
dig=$(printf '%s' "$idx" | jq -r --arg t "$target" '.artifacts."agency/atcd"[$t].digest' | sed 's/^sha256://')
# `jq -r` prints the string "null" for a missing key rather than failing, and the
# next line would then try to fetch a host named `null` ("curl: (6) Could not
# resolve host: null") — a confusing error for a track that simply has no atcd
# for this arch. Check before fetching:
case "$url" in
  ''|null) echo "no atcd published for $target on track $track" >&2; exit 1 ;;
esac
# Keep these three chained. `sha256sum -c` reports a mismatch only by EXIT
# STATUS, so as separate lines a tampered download still reaches the scp.
curl -fsSL "$url" -o atcd && \
echo "$dig  atcd" | sha256sum -c && \
scp atcd deploy@device:/tmp/atcd

The device has curl and OpenSSL 3? Fetch on the device instead. install-atcd.sh also checks the signature on the index, which the block above does not. It picks the device’s CPU itself, and ATC_DOWNLOAD_ONLY writes the verified binary and installs nothing, so do not pass it arguments here. To install as well, run the script with no ATC_DOWNLOAD_ONLY: it adds --no-service itself (see the start of Path B).

track=$(atc --version | sed -n 's/.*track \([a-z][a-z]*\)).*/\1/p')
ssh deploy@device "curl -fsSL https://atc.dev/install-atcd.sh | ATC_TRACK=$track ATC_DOWNLOAD_ONLY=/tmp/atcd sh"

For the dev track, also set ATC_CHANNEL_TOKEN in the remote command.

Need a specific atcd version, not the track’s current one? The block above always resolves whatever the track publishes today, which is what you want for a normal bootstrap. To pin an older build — testing an upgrade or reproducing a customer’s version — take the binary from that release instead of the track index. Releases are tagged by date (nightly-2026-08-12), and each one attaches index.json alongside the binaries:

tag=nightly-2026-08-12          # the release you want
# $target is the DEVICE's target from the case statement above, e.g.
# linux-aarch64-musl — the same wrong-arch trap applies here.
gh release download "$tag" -R Agency-Tool-Company/agency-code \
  -p "atcd-$target" -p 'index.json' -p 'index.json.sig'
mv "atcd-$target" atcd

Verify the index signature before you trust its digest. The digest below only proves that atcd matches what index.json claims, so an index an attacker also replaced would still say OK. Check the ed25519 signature first with the pinned pubkey, exactly as verifying-the-binary.md describes (it carries the current key and the OpenSSL alternative):

atc sign verify index.json index.json.sig <pinned-pubkey-hex> && \
dig=$(jq -r --arg t "$target" '.artifacts."agency/atcd"[$t].digest' index.json \
  | sed 's/^sha256://') && \
echo "$dig  atcd" | sha256sum -c && \
scp atcd deploy@device:/tmp/atcd

The && chain is load-bearing, not style. atc sign verify and sha256sum -c both report failure through an exit status (VERIFY FAILED: …, exit 1), not by stopping your shell. set -e does not cover this: it has no effect on commands pasted into an interactive shell, which is how an operator runs this. Run them unchained and a rejected signature falls through to the digest check against the attacker’s index, then copies the substituted binary to the device.

Then continue with step 2 below — nothing else about Path B changes. Note that Path A’s --from <path> flag is a bootstrap option, and Path B runs no bootstrap command at all, so a pinned version here is purely a matter of which bytes you scp.

scp reads ~/.ssh/config exactly as ssh does, so aliases, Port, and ProxyJump all resolve — but a -o flag you passed on an earlier ssh command line does not carry over. A successful PID-1 check with inline -o StrictHostKeyChecking=accept-new has already written the device key to known_hosts, so this later scp finds it and does not prompt. If that check never completed — you skipped it, or it failed before connecting — this scp is the first contact and stops on the host-key prompt. Pass the option again (scp -o StrictHostKeyChecking=accept-new …), or set it in ~/.ssh/config once and stop repeating it.

The index URL derives the track from atc --version (which prints … track <name>)), so atcd always comes from the exact track your atc was installed from — controller and daemon can’t drift apart, whatever the installer’s current default track is.

2. Install and start it on the device. Two rules the controller depends on: the binary must live at /usr/local/bin/atcd (the fixed path atc probes), and atcd must run as the same user you SSH in as (start it without sudo so it owns the socket — a root-owned socket makes every later atc command fail Permission denied).

For the simplest copy-and-paste flow, run the whole block as one SSH command. The final setsid … & line backgrounds and detaches atcd; it also works if you already ran the setup lines separately, so don’t start a second copy; two daemons contend for the same socket.

On a container device, plan for restarts before you run it: /run is tmpfs — it does not survive a container restart, so the mkdir + atcd run lines must run on every container start. Bake them into your entrypoint, not a one-time setup script.

ssh deploy@device '
  sudo install -m 0755 /tmp/atcd /usr/local/bin/atcd
  sudo mkdir -p /var/atc /run/atcd
  sudo chown -R "$(id -un):$(id -gn)" /var/atc /run/atcd
  # no sudo on the next line — atcd must own the socket (the second rule above)
  setsid /usr/local/bin/atcd run \
    --store-root /var/atc --socket /run/atcd/atcd.sock \
    </dev/null >/tmp/atcd.log 2>&1 &
'

Release builds log nothing, so an empty /tmp/atcd.log is normal. It is not a sign that the daemon failed to start. Confirm it came up before you continue:

ssh deploy@device 'pgrep -x atcd >/dev/null || { echo "atcd is NOT running" >&2; exit 1; }'

From here the device behaves like any bootstrapped one — skip the bootstrap command and continue below. Nothing here needs jq or python3 on the device. The later steps that do say so where they need it. The hook script in step 3 needs only awk, which every base image carries. The OCI arch check in step 5 needs jq, which a minimal image omits, so expect to install it there.

Started it with sudo by mistake? Then root owns the socket and the very next atc command fails with Permission denied — the error names the socket, not the cause, so it’s easy to misread as a config problem. Stopping it takes sudo too: a plain pkill atcd as the SSH user cannot signal a root-owned process and exits non-zero having done nothing, so the stray daemon keeps the socket and the retry fails identically.

ssh deploy@device 'sudo pkill -x atcd; sleep 1; pgrep -x atcd || echo "stopped"'

Confirm it printed stopped before restarting — then re-check the setsid line for a stray sudo and start it again without one. Never start a second copy while the first is alive: two daemons on one /var/atc contend for the same socket and store.

Restarting atcd (Path B). There is no unit to systemctl restart, so a crashed or wedged daemon is a manual stop-then-start. pgrep -x atcd is the liveness check; if it is already dead, skip straight to the setsid line in step 2 above. To bounce a running one:

ssh deploy@device '
  sudo pkill -x atcd || true      # sudo: it may be root-owned; non-zero = none running
  sleep 1
  pgrep -x atcd && { echo "still running — not starting a second" >&2; exit 1; }
  sudo mkdir -p /run/atcd && sudo chown -R "$(id -un):$(id -gn)" /run/atcd
  setsid /usr/local/bin/atcd run \
    --store-root /var/atc --socket /run/atcd/atcd.sock \
    </dev/null >/tmp/atcd.log 2>&1 &
'
atc ota status deploy@device

The /run/atcd re-create matters after a container restart, not a plain daemon crash. /run is tmpfs, so the socket dir is gone on a fresh container even though /var/atc survives. Never start a second copy while the first is alive. Two daemons contend for the same socket and store.

Upgrade atc first — both paths. atc and atcd exchange a protocol generation at handshake, and accept only an exact match. A stale atc therefore cannot push to a daemon you just upgraded, and the reverse is refused too. So update your own binary before you touch the device:

atc upgrade
atc --version          # confirm the new version and track

Only a push checks that generation. atc ota status and atc ota ls use a separate handshake, so they still answer across a push-wire skew. A clean status therefore proves the daemon is up, not that a push will pass. A push wire mismatch warning means the pair cannot push until you upgrade the named binary. Rollback carries the full skew rule, the warning, and the error a mismatch prints.

Upgrading atcd later (Path A). On a systemd device the controller does the whole swap for you:

atc bootstrap atcd upgrade --yes --non-interactive deploy@device
atc ota status deploy@device   # confirm the new daemon answers

It ships a verified atcd, restarts the unit so the daemon boots the new version (boot-confirm-guarded), and waits for it to come back. --track <name> aligns the device to a track other than your atc binary’s own. Downgrades are always permitted, so this also puts a device back on an older revision.

The new binary comes from ATCD_LOCAL_BIN or a local build when one is present, otherwise the signed release origin. The command therefore needs either a local binary or network access. It is deliberately not taken from the device’s own store. The daemon can write there, and the privileged step must not execute bytes the daemon controls. The store at /var/atc is untouched.

Upgrading atcd later (Path B). atc bootstrap atcd upgrade is Path A only — it restarts a systemd unit and chowns the store to the daemon’s own user, so on a Path B device it fails (no unit, and no such user to chown to). Replace the binary by hand instead. Re-run 1. Fetch atcd from the release origin above to fetch the new binary, then:

ssh deploy@device '
  set -e
  # sudo: the running daemon may be root-owned (see the sudo note above), and an
  # unprivileged pkill would fail silently and leave it holding the socket.
  sudo pkill -x atcd || true          # non-zero simply means none was running
  sleep 1
  # Explicit exit, not `! pgrep`: under `set -e` a negated command is a tested
  # condition, so a surviving daemon would NOT abort the script.
  if pgrep -x atcd; then echo "atcd still running — not starting a second" >&2; exit 1; fi
  sudo install -m 0755 /tmp/atcd /usr/local/bin/atcd
  # same rules as first start: /run is tmpfs, and no sudo on atcd itself
  sudo mkdir -p /run/atcd && sudo chown -R "$(id -un):$(id -gn)" /run/atcd
  setsid /usr/local/bin/atcd run \
    --store-root /var/atc --socket /run/atcd/atcd.sock \
    </dev/null >/tmp/atcd.log 2>&1 &
'
atc ota status deploy@device   # confirm the new daemon answers

The store at /var/atc is left alone: the upgrade is a binary swap, not a store migration. One exception — if the old daemon had been running under sudo, its store writes are root-owned and the unprivileged daemon can’t read them. Hand the store back before starting: ssh deploy@device 'sudo chown -R "$(id -un):$(id -gn)" /var/atc'.

Verify the daemon is reachable — both paths. After Path A’s bootstrap or Path B’s foreground start, confirm the controller↔daemon handshake (the authoritative proof it’s up):

atc ota status deploy@device

An (empty) store listing means the controller can reach the daemon. A just-started daemon can miss the first 5 s ping and return ATC-E0308: … ping … timed out — that’s a cold-start race, not a failure. Re-run the same command once and it answers instantly.

3. Declare what to ship — atc.toml

Work in a directory you own (not this repo). Scaffold a minimal manifest:

mkdir -p ~/first-deploy && cd ~/first-deploy
atc ota init        # writes a minimal atc.toml (one [[bundle]])

atc ota init writes a scaffold: one [[bundle]] named after the current directory with source = "dist", and the rule about where writable state goes. It also carries the lifecycle-hook and long-running-service blocks as comments. Pass --full to also write the compiled-binary block, or read it in docs/operating/atc-toml.md. Pass --format kdl to write an atc.kdl instead. It won’t write beside an existing atc.toml or atc.kdl. This walkthrough doesn’t use the scaffold’s values — its bundle name (app), source directory (app_bundle), and the live commit hook that step 5 verifies all differ. Don’t reconcile the two: replace the generated atc.toml wholesale with this complete example, which ships a tiny bundle (a directory of files) and drops a marker file when the update commits:

[[bundle]]
name = "app"
source = "app_bundle"             # a directory you stage: payload + scripts/

[bundle.lifecycle.hook.commit.exit]   # fires after "app" commits active
script = "@self/scripts/mark.sh"      # @self = this artifact's root on the device
timeout_secs = 60

This marker hook reports that the delivery is durable, which is what commit.exit is for. A hook that brings the new version into service, such as starting a unit or swapping containers, belongs at activate.exit. That is the first gate after the flip, so the gap where the device points at the new version without running it stays short.

If you stage without activating (--no-switch, or no --switch while autoswitch is off), no activate.* hook runs. Nothing is activated, so a restart at activate.exit does not restart the running version. See lifecycle-hook-cutover.md.

source resolves against the directory holding atc.toml, not against your current directory. That still holds under -c/--config, where the manifest can live anywhere. An absolute path is not rejected: it is taken as written, so the push works on your machine and breaks on anyone else’s. Keep source relative. A path that resolves to nothing fails with ATC-E0504: bundle source not found: <path>.

Check the shared libraries of each binary you stage. A binary that links libraries from the build machine fails on the device. It reports error while loading shared libraries. Run file <binary> first, then ldd <binary>, on a machine that does NOT hold the build tree.

file comes first because the artifact may not be a binary. An autotools or libtool build writes a wrapper shell script under the binary’s own name, and leaves the real ELF under .libs/. ldd reports nothing useful about that wrapper, so a check that starts at ldd reads as clean and stages the wrapper.

The glibc risk runs one way. glibc keeps older binaries working, so a binary built against an older glibc than the robot’s runs there. Only the reverse needs a static link: a newer build machine, an older robot. Run ldd --version on both machines to see which case you have.

atc ota push answers that question for you when it can. It reads the robot’s glibc over the ssh connection it already holds, and it compares that against the release each staged binary binds. A robot that meets the release says nothing at all. A robot below it REFUSES the push (ATC-E0469), and names both numbers. Pass --allow-glibc-skew when the workload runs under a glibc the probe cannot see, inside a container or a chroot on the robot.

A push that cannot read the robot’s glibc says nothing, because it measured nothing. Run ldd --version on the robot yourself then.

A static link is the reliable answer. The flag depends on the toolchain, and none of them is a guarantee on its own:

• autotools: configure with --disable-shared --enable-static. That governs the project’s OWN libraries. It does not make libc static, so the binary can still link glibc dynamically. Add LDFLAGS=-static (or libtool’s -all-static) when you want libc in the binary too.
• Go: build with CGO_ENABLED=0 go build. A plain go build looks static and often is not. The standard library pulls cgo in for DNS lookups and for os/user, which links the build machine’s glibc. go build prints no warning about it. Two consequences to know. A program that genuinely needs cgo will not build this way. And the pure-Go resolver reads /etc/hosts and /etc/resolv.conf directly, so an exotic nsswitch.conf behaves differently.
• Rust: build for a musl target, such as x86_64-unknown-linux-musl.

Then check the result rather than trusting the flag. file should report statically linked, and ldd should report not a dynamic executable. If ldd still lists libraries, the flag did not do what you wanted.

Copying the libraries ldd names into source is NOT sufficient on its own. The loader must also be told to look beside the binary, and a RUNPATH covers only the direct dependencies of the object that carries it. Read shared-library-resolution.md before you take that route.

Delivered files are read-only at runtime. The device materializes every bundle file 0o440, or 0o550 when the source file is executable. No delivered file keeps a write bit. A program that writes beside its own binary works in local development and fails after the deployment. Keep writable state under $HOME, or under a path you pass in an environment variable.

The directories that hold those files are not read-only. They keep the daemon user’s own mode, so a manual write over ssh as that user can still succeed.

Do not predict which way a write goes. Two inputs decide it and neither is a guarantee. [daemon] hook_run_as_user picks the account a hook runs as, defaulting to atcd’s own uid. The directory’s mode comes from atcd’s umask when the store was built, so a permissive umask leaves it group-writable, and atc-hooks is in the store group. The same program can take EACCES on one device and succeed on the next. Only the file modes are a guarantee.

A write beside the binary does not follow the pointer. Each version lands in a directory of its own, so a file written there stays with the version that was current when it was written. Retention prunes it later, and a rollback to that version brings it back. Do not read “the next push moves on” as “the data is gone”.

There is no version field. The version is assigned at push time (step 4). A manifest still carrying version = is a hard error.

The device decides what stays on disk. Two device settings control it:

• storage.delivery_keep_n (default 5) keeps the newest labeled deliveries on each channel. A delivery is one committed push. It keeps the versions it installed, so each of these deliveries is a rollback target.
• storage.version_keep_n (default 0) sets the total number of versions to keep for each artifact. Versions that a kept delivery installed count toward this total, and they stay even when they exceed it. The device removes the least recently used of the other versions first.

The active version is always kept. The device prunes in a housekeeping sweep, so you never prune by hand. To change a setting, run atc ota settings set deploy@device storage.delivery_keep_n=10. The change takes effect when atcd restarts.

The hook script below parses its context with awk. Every base image carries it, busybox and debian:stable-slim included. Do not use python3 in a hook: a minimal image such as debian:stable-slim omits it, and so does the meta-atc image. A hook that calls a missing python3 exits nonzero. atcd reads that as a veto: it rolls the whole delivery back and reports the push as failed.

Do not use jq in a hook. Many device images omit it. atcd also pins the hook PATH to the standard system directories, so a hook sees only what the image already installed. A hook that calls a missing jq gets an EMPTY value rather than an error. It writes a broken file and still exits 0, and the push then reports the hook as successful.

Now create the bundle and its hook script:

mkdir -p app_bundle/scripts
echo "hello from atc" > app_bundle/payload.txt
cat > app_bundle/scripts/mark.sh <<'SH'
#!/usr/bin/env bash
set -euo pipefail
# Lifecycle hooks run with a CLEARED environment. The delivery context arrives
# as ONE JSON variable, ATC_HOOK_CONTEXT. There is no ATC_VERSION or any other
# scalar to read a delivery field from. Parse the JSON with awk.
#
# `ctx` reads one dotted field. A second argument gives the value to use when
# the field, or an object above it, is null. With no second argument, a field
# under a null object prints "". An unknown key or a null value exits nonzero,
# and `set -e` then stops the hook, so no step acts on an empty value.
ctx() {
  awk -v path="$1" -v nargs="$#" -v dflt="${2-}" '
    function fail(why) { print "ATC_HOOK_CONTEXT." path ": " why > "/dev/stderr"; exit 1 }
    function find(t, k,   j, c, d, q) {
      k = "\"" k "\":"
      for (j = 1; j <= length(t); j++) {
        c = substr(t, j, 1)
        if (q) { if (c == "\\") j++; else if (c == "\"") q = 0 }
        else if (c == "{") d++
        else if (c == "}") { if (--d == 0) return 0 }
        else if (c == "\"") { if (d == 1 && substr(t, j, length(k)) == k) return j + length(k); q = 1 }
      }
      return 0
    }
    BEGIN {
      s = ENVIRON["ATC_HOOK_CONTEXT"]; n = split(path, key, ".")
      for (i = 1; i <= n; i++) {
        if (substr(s, 1, 4) == "null") { print dflt; exit 0 }
        if (substr(s, 1, 1) != "{") fail("not an object")
        p = find(s, key[i])
        if (p == 0) fail("no such field")
        s = substr(s, p)
      }
      if (substr(s, 1, 4) == "null") { if (nargs < 2) fail("null"); print dflt; exit 0 }
      if (substr(s, 1, 1) != "\"") fail("not a string")
      for (j = 2; j <= length(s); j++) {
        c = substr(s, j, 1)
        if (c == "\"") { print v; exit 0 }
        if (c == "\\") { j++; c = substr(s, j, 1); if (c != "\"" && c != "\\" && c != "/") fail("holds a control escape") }
        v = v c
      }
      fail("not a string")
    }'
}
name=$(ctx artifact.name)
sha=$(ctx version.new_sha)
echo "atc update $name @ $sha ran" > /tmp/atc-app-ran
SH
chmod +x app_bundle/scripts/mark.sh

If a commit hook exits nonzero (a missing reader, a script error), atcd treats it as a veto and automatically rolls the whole delivery back to the previously-active version, reporting the push as failed — a bad hook can’t strand the device half-updated. That is why the hook above fails loudly instead of writing an empty marker.

What the hook receives. $ATC_HOOK_CONTEXT is one JSON document with stable, always-present keys (an absent value is JSON null, never a missing key):

Read a field with the ctx helper above, e.g. ctx version.old_sha. The full ATC_HOOK_CONTEXT schema and hook reference is in atc-toml.md — ATC_HOOK_CONTEXT.

PATH is pinned, not inherited. “Cleared environment” includes PATH: your shell’s PATH never reaches the hook. atcd exports a fixed, absolute one —

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

— so a hook shelling out by bare name resolves only standard system locations. Anything your hook calls (curl, jq, a tool in /opt) must already be installed on the device and reachable from that PATH; otherwise call it by absolute path. So the hook above reads its context with awk, which every base image carries. It does not use jq or python3, which an image may omit. This is deliberate. An inherited PATH would let a writable directory on the daemon’s environment decide what a hook executes. HOME, USER, LOGNAME, and SHELL are set from the hook user’s passwd entry, as in a login session.

The full set a hook gets. ATC_HOOK_CONTEXT carries the delivery context, and it is the only place to read delivery data. The rest of the environment is fixed:

Nothing from your shell reaches the hook. atc ota push --hook-env KEY=VALUE injects one extra variable for one push, and atc ota switch --hook-env does the same for one switch.

Which user is that? It depends on which path you took in step 2:

• Path A (atc bootstrap, systemd): bootstrap creates an atc-hooks user and bakes --hook-run-as-user into the unit. Hooks run as atc-hooks. That user is in the daemon’s group, so it can read the 0750 store but cannot corrupt it. You do not create this user by hand.
• Path B (manual foreground atcd): no hook user is configured, and no atc-hooks user exists. Hooks run as the user that launched the daemon, which is your SSH login user. Do not pass --hook-run-as-user here. A hook uid switch needs a root or CAP_SETUID daemon, which the un-sudo’d foreground recipe deliberately avoids.

The hook runs on both paths. The paths differ in which uid it gets, not in whether it fires. Full reference: atc-toml.md.

Two limits on what a hook can do with systemd. Both come from the device or from the sandbox, not from your hook.

• systemctl --user needs two packages the device may not ship. On Debian or Ubuntu, install libpam-systemd and dbus-user-session. Without libpam-systemd, loginctl enable-linger starts no user manager and each systemctl --user call answers Failed to connect to bus. Without dbus-user-session there is no /run/user/<uid>/bus, so the DBUS_SESSION_BUS_ADDRESS in the table above names nothing, and a hook that speaks D-Bus fails. atc bootstrap atcd install reads the device and warns when either one is absent. atcd installs and runs without both.
• A hook writes its units under its own home, not under /etc. atcd’s unit sets ProtectSystem=full, which each hook inherits. So /usr, /boot, and /etc are read-only to it. The hook user’s home is in ReadWritePaths, so put unit files in ~/.config/systemd/user/ and drive them with systemctl --user. That is the pattern the isolated hook user exists for. It holds no sudo, so a root-owned system unit is out of reach from a hook either way.

Your program prints no version of its own

Some programs carry one version and no more. An autotools C project is the common case. jq --version prints the jq release, and the project has no second version to bump per push. You do not need one. atc owns the version label, and you set it with --version at push time (step 4). atc ota status reports the live label, so “which version is on the device” never depends on the program.

Add an on-device version marker only when the DEVICE itself must report the answer, such as from a log line or a health endpoint. Two patterns work, and neither one patches the program:

• Stage a VERSION file beside the binary. Write the same string you pass to --version into the source directory before each push. The file ships with the payload, so it lands in the version’s own directory on the device. A hook runs in that versioned directory, so it can read the file beside itself. Give the file one writer, your build script, so the file and the push label cannot drift.

  Do not treat the hook’s directory and atc shell -a <name> as one place. atc shell -a <name> enters delivery/current/<name>, which is the ACTIVE version. A hook runs in the versioned tree of the version the push installs. After a stage-only push the two differ. The hook reads the new VERSION file, and the shell still reads the old one. ATC_HOOK_CONTEXT.current_root hands a hook the active pointer when it wants that one instead.

• Ship a wrapper script and point the service at it. The wrapper prints a labelled line, then runs the real binary. Use this when something already starts the program and you cannot add a file read.

Either way, print the label with the value. Bare version output makes a poor marker, for the reason step 5 gives.

Shipping a container image instead?

A [[bundle]] ships a directory. To ship a container image, import it into atc’s local store first, then declare an [[oci]] artifact next to (or instead of) the bundle:

atc oci import alpine:latest              # pull from a registry into atc's store
# or import what your local engine already holds:
#   atc oci import --docker my-app:latest   (--podman for podman)

The registry pull goes through skopeo, which reads your machine’s /etc/containers/registries.conf. On a host whose config is still in the v1 format, that pull fails with a registries.conf format error before atc sees the image. You don’t need to fix the host config — pull the image with your local engine and import from there instead:

docker pull alpine:latest && atc oci import --docker alpine:latest

[[oci]]
name     = "app"
source   = "alpine:latest"   # the ref you imported
platform = "linux/amd64"     # REQUIRED; match the DEVICE's CPU, not your machine's

Shipping an image instead of a bundle? Then that block is the whole file. There is no [[bundle]] to keep:

[[oci]]
name     = "app"
source   = "alpine:latest"
platform = "linux/amd64"

platform is required because an image ref can carry several architectures, and atc must know which one belongs on the device. It is the same wrong-arch trap as the target note in step 2: a wrong value looks fine at push time. Take the value from the device, not from your machine. ssh deploy@device 'uname -m' prints x86_64 for linux/amd64 and aarch64 for linux/arm64. OCI names the arch differently from the kernel, so linux/x86_64 and linux/aarch64 are not the values to use. Pushing and activating then work exactly like the bundle flow in steps 4-5, and [[oci]] artifacts take lifecycle hooks the same way a bundle does (under [oci.lifecycle.*]), so a commit.exit hook can verify an image just as mark.sh verifies the bundle.

Verifying which platform landed is the one step that differs: the ELF-byte check in step 5 reads a single binary, and an image has no equivalent. The arch lives in the image’s config blob, two hops from the version dir’s index.json (index → the single manifest blob → its config.digest). Walk it on the device:

ssh deploy@device '
  d=/var/atc/channel/local/default/delivery/current/app
  m=$(jq -r ".manifests[0].digest" "$d/index.json" | cut -d: -f2)
  c=$(jq -r ".config.digest" "$d/blobs/sha256/$m" | cut -d: -f2)
  jq -r ".architecture, .os" "$d/blobs/sha256/$c"
'

That prints the arch and OS baked into the image that actually landed — expect amd64 on an x86-64 device, arm64 on aarch64, and your laptop’s arch here is the wrong-platform bug.

This check needs jq on the device, and nothing else in this guide does. The hook script reads its context with awk for that reason. So install jq before you run the block above, and only for this check:

ssh deploy@device 'sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y jq'  # Debian/Ubuntu
# Alpine: sudo apk add jq   ·   Fedora/RHEL: sudo dnf install -y jq

Don’t hand-parse these blobs with grep/sed instead: the config digest sits in a "config" object that also holds a "digest" key for every layer, so a line-oriented match picks up a layer digest and the walk dies on a missing blob.

atc ota status and atc ota ls --artifacts deploy@device confirm the version activated. For an OCI-only delivery, their Arch column reads unknown: the column reports the first ELF arch found among the active delivery’s members, and OCI members are not ELF files on disk. Check an OCI payload’s arch from its image config instead.

Full [[oci]] schema and OCI lifecycle hooks: atc-toml.md.

Shipping a module instead?

A bundle and an OCI artifact are payloads the device unpacks. A module is an executable the device also runs, each time a delivery ships it — including a re-push at an unchanged SHA. A delivery that carries a module forward untouched does not re-run it. Write start to be safe to run again. Modules are declared in atc.kdl rather than atc.toml, and much of the module vocabulary is still refused at push time, so read modules-today.md before you design around one. A repo you can copy, with an e2e test that pushes it to a real container, is examples/heartbeat-module/.

Optional: run the whole push on your own machine first

atc ota push --local starts a throwaway atcd on this machine and pushes to it. It needs no device, no SSH and no root, and it runs the real daemon. So a manifest and a hook that pass here ran against the real executor. Use it to check atc.toml and mark.sh before you involve the device.

Get an atcd for this machine first. The installer places only atc, so a fresh machine has no atcd and the push stops on ATC-E0463. One command provisions one, and it needs no device:

atc bootstrap atcd fetch

It reads the same signed release origin atc upgrade reads. It checks the ed25519 index signature and the sha256 artifact digest. Then it writes the binary where --local looks for one: beside your atc when that atc is a plain binary, and ~/.local/bin otherwise. It also says whether --local can reach the result, and names the fix when it cannot.

Add --track <track> to follow another channel, or --source <name> to fetch from a private release mirror registered in sources.toml. Both resolve the way atc upgrade resolves them. Then:

atc ota push --local --switch --local-root /tmp/atc-local-test

Give --local-root an absolute path, and keep it short. A relative path makes every hook fail to spawn (spawn failed: No such file or directory), and the failure vetoes the push. The root also holds atcd’s Unix socket, and a socket path over 103 bytes is refused with ATC-E0463.

• Linux only. The released atcd assets are Linux x86_64 and aarch64 binaries. On any other host atc bootstrap atcd fetch refuses and says so.
• --local needs an atcd for this machine. atc bootstrap atcd fetch is the short way. atc also accepts one you supply: put it next to atc, put it on PATH, or name it with ATC_LOCAL_ATCD. The Path B fetch block in step 2 downloads one too, but pick your OWN target there, not the device’s.
• Pass --switch or --no-switch. The command refuses to choose. --switch runs the activate and commit hooks. --no-switch stages, which runs the download and install hooks and then stops.
• --local-root <DIR> keeps the run: the store, atcd.log, and each hook’s stdout and stderr under hooks/. Without the flag --local uses a temporary directory and removes it. A directory that already holds something else is refused.
• This is not a sandbox. Hooks run under your uid, in your home, against your services. --local relocates the store and nothing else.
• Push the same content twice into one --local-root and the daemon treats it as already present. No hooks run at all. Change something between runs.

Your hook writes /tmp/atc-app-ran on this machine, so read it here:

cat /tmp/atc-app-ran

4. Push and activate

Every artifact by default — a bare atc ota push pushes every artifact declared in atc.toml. (Omit --version and the push gets an auto-generated <host>-<timestamp>-<entropy> label.) Give the push a version and add --switch to activate it:

atc ota push --version v0.1.0 --switch deploy@device

• You don’t have to be in the manifest directory. atc discovers atc.toml by walking up from the current directory, so any subdirectory of ~/first-deploy works. Pass -c/--config <FILE> to point at a manifest elsewhere and skip that search entirely.

• --only <name> (repeatable) narrows the push to a subset of artifacts; a bare push (no --only) pushes all of them.

• -t app@v0.1.0 is shorthand for --only app --version v0.1.0.

• Without --switch the version is staged on the device but it keeps running the old one.

• The push log shows the commit.exit hook firing. The log prints one line per hook, when it finishes. For a hook that prints nothing on success, like the mark.sh above, which only writes a marker file, the log shows:

    hook logs: <cache-path>
    ✓ commit.exit  app  scripts/mark.sh  8ms
    i commit.exit produced no output
  

  The log prints hook logs: once per push, under the push-started line. Under --non-interactive, which prints no push-started line, it prints above the first hook that records output to disk. A failed hook reads ✗ and its exit state, for example exit 3 or timed out.

  The interactive view draws the same hook as └ ✓ commit.exit app scripts/mark.sh under the Commit row, and names hook logs: <cache-path> once, under the first line of the push.

  <cache-path> is $XDG_CACHE_HOME/atc/hooks/<timestamp>-<target>/, or ~/.cache/atc/hooks/<timestamp>-<target>/ when XDG_CACHE_HOME is unset. It holds one <state>.<phase>__<hook-run-id>.stdout/.stderr pair per hook run. These are captured on your machine by atc, not on the device. Recording is on by default for every push; --capture-hooks <DIR> pins a different directory, and --json is the one mode that records nothing.

  The ✓ commit.exit line is your confirmation the commit.exit hook ran and succeeded — it’s the one stable signal to grep for. A failed hook prints ✗ and the reason instead: exit 3, signal 9, timed out or spawn error. Each hook run also carries a unique opaque id, which the recorded filenames use. Don’t string-match against that. The hook logs: and i lines are normal, not a warning: atc captures every hook’s stdout/stderr to disk by default, and notes when a hook was silent. The hook runs after the atomic version flip, exactly as the [bundle.lifecycle.hook.commit.exit] comment says.

• atc ota status’s lifecycle log names the same hook by its journal label, commit/pre_hook_exit, which does not match the commit.exit you wrote in atc.toml. That’s the daemon’s internal journal name for the gate, and pre_ means “about to run the exit hook”, not “before the commit”. commit.exit, commit.post, and commit/pre_hook_exit all name the same gate; commit.enter / commit.pre map to commit/pre_hook. The push log uses commit.exit. Match on ✓ commit.exit, never on the journal label.

• Did the push stop with ATC-E0460: atc and atcd are incompatible or ATC-E0016: unsupported protocol version before any payload moved? An older atc can show ATC-E0443: store: push failed first, with the ATC-E0016 nested below it. Then atc and the device’s atcd speak different push wires. This happens on a device that an older release bootstrapped. Upgrade the binary that the error names. Rollback carries the full rule.

5. Verify it landed

Prove it with commands — don’t assume:

atc ota status deploy@device                        # active version should now be v0.1.0
ssh deploy@device 'grep "^atc update app @" /tmp/atc-app-ran'  # hook marker + content
ssh deploy@device 'od -An -tx1 -j18 -N2 /usr/local/bin/atcd'  # ARCH that landed (see below)

If ota status shows v0.1.0 current and the marker line is present, the update is live. Grep the marker’s content (not just cat it) — a hook that ran with a broken parse could leave an empty file that a bare cat would still show.

Label every version string you check. A bare version marker is a common cause of a verify step that reads as a failure. A program’s own --version often prints 1.0.0 and nothing else. That string holds no stable word to anchor a grep, so one grep cannot separate three cases:

• the new version ran
• an older version ran
• the command printed nothing at all

Print a label beside the value instead. The label proves the command ran, and the value proves which version ran:

ssh deploy@device 'echo "myapp-version=$(myapp --version)"'

Then check the label and the value together:

ssh deploy@device 'echo "myapp-version=$(myapp --version)"' \
  | grep -q 'myapp-version=v0.1.0'

A command that fails prints the label with an empty value. The paired grep rejects that line, so a broken run cannot read as a pass. Use the same pattern for a hook marker file: write label=value, never the value alone.

Confirm the architecture that shipped. ota status reports version and sha, atcd --version reports atcd <semver> with its existing prefix, and atcd version (the subcommand, which also takes --json) reports semver, git sha, and build date. None of them report target arch, so a wrong-arch binary (e.g. a linux-x86_64-musl build on an aarch64 device) looks identical there. Read the deployed binary’s ELF header directly — byte 18 is e_machine, the CPU the binary was built for:

ssh deploy@device 'od -An -tx1 -j18 -N2 /usr/local/bin/atcd'
#   3e 00  → x86-64      ·   b7 00  → aarch64      (ELF e_machine, little-endian)

e_machine is a little-endian 16-bit field, so the two bytes read low-byte-first: 3e 00 is 0x003e (62, x86-64) and b7 00 is 0x00b7 (183, aarch64). Bytes in the other order would mean a big-endian ELF, which no device here targets.

od is coreutils, so this works even on the stripped, file-less container images Path B targets. If file is installed it’s friendlier — ssh deploy@device 'file /usr/local/bin/atcd' prints … x86-64 … statically linked — but it’s absent on minimal images, so don’t lead with it. Don’t fall back to uname -m: it reports the device’s CPU, not the binary’s arch, so it can’t catch a wrong-arch push onto a matching-CPU host — the ELF-byte check above is the real proof. (Default daemon path is /usr/local/bin/atcd; adjust if you set [daemon] remote_path.)

Where your files land on the device. In the default (delivery-pinned) mode, point on-device services at the authoritative delivery pointer — atc flips it atomically on every switch and rollback:

/var/atc/channel/local/default/delivery/current/<name>

6. Rollback

Rollback is a flip back to an earlier version, so it needs at least two on the device — if you’ve only pushed v0.1.0 so far, repeat step 4 with --version v0.2.0 first. List what’s on the device, then switch:

atc ota ls deploy@device                      # list on-device versions
atc ota switch deploy@device <older-version>  # re-activate an earlier one

switch needs a current atcd. Whole-delivery switch is its own wire operation, so an atcd that predates it doesn’t recognize the request and the switch fails with ATC-E0455: atcd is too old for the whole-delivery switch. Nothing was activated — the old daemon refuses before touching the store — so the device is still on the version it was already running. The fix is forward only: upgrade atcd on the target (atc bootstrap atcd upgrade deploy@device on a systemd device, or the Path B binary swap in step 2), then re-run the switch. Downgrading atc to match an old daemon is not supported.

An old atc against a newer atcd is refused too. The push wire is a single generation per build. atc and atcd exchange a protocol version at handshake and accept only an exact match. A push across a skew in either direction is therefore rejected before anything transfers. There is no “old-client-still-works” window for a push. status and ota ls still answer, as the paragraphs below show.

Keep the pair in step: upgrade atc with atc upgrade, and the device’s atcd by Path A or Path B above.

The refusal happens at the push’s first stage, before any payload transfers, and it names the binary to upgrade:

• A current atc against an older atcd fails with ATC-E0460: atc and atcd are incompatible, followed by upgrade this daemon (atcd) and the fix command atc bootstrap atcd upgrade <target>.
• An older atc against a newer atcd fails with ATC-E0016: unsupported protocol version, followed by upgrade the pushing client (atc). A release that predates ATC-E0460 wraps it in ATC-E0443: store: push failed.

atc ota status does not refuse a skewed pair. Status and ota ls use the attach handshake, which has its own version and does not move with the push wire. Measured: an atc 0.1.3 (atc/5) read status and listed deliveries from an atcd 0.1.5 (atc/6) with no error, then failed its first push with ATC-E0016.

A newer atcd reports its push wire in the status reply, and atc ota status then flags a skew itself. The output ends with warning: push wire mismatch: this atc speaks atc/N, the daemon speaks atc/M. and a fix: line that names the binary to upgrade. --json carries the same answer as "push_wire": {"atc", "atcd", "match"}. An atcd that predates the field reports "atcd": null and "match": null, and prints no warning. For that daemon, compare the Daemon column of atc ota status deploy@device with atc --version. The same release always speaks the same wire. Different releases may still match, so a push is the final answer.

A switch re-runs the target version’s activate and commit lifecycle hooks (with trigger: "user_activate" in $ATC_HOOK_CONTEXT). They are not “fire once at first install.” The order is activate.enter, deactivate.enter, deactivate.exit, the pointer flip, activate.exit, commit.enter, commit.exit. The two deactivate cells run the declaration of the OUTGOING version, which is the one the switch leaves. Your marker is rewritten on rollback, carrying that version’s sha. A hook that restarts a service or runs a health check fires on the rollback too. The store itself is only a pointer flip — nothing is rebuilt or re-downloaded — but the hooks do run.

Prove the rollback the same way you proved the push (step 5) — one verification per delivery, whether it arrived by push or by switch:

atc ota status deploy@device                    # active version is the older one again
ssh deploy@device 'grep "^atc update app @" /tmp/atc-app-ran'  # marker rewritten with THAT version's sha

.version.new_sha is the artifact’s content sha, not the delivery version string. It tracks the payload, so it changes only when the payload changes. Roll back to a version whose payload is byte-identical and the marker gets rewritten with the same sha it already held. An unchanged marker is therefore not evidence that the hook was skipped. To prove a re-run for any payload, make the hook append instead of overwrite, or record a timestamp with the sha:

echo "atc update $name @ $sha ran at $(date -Is)" >> /tmp/atc-app-ran

The ✓ <state>.<phase> hook line in the push output is the authoritative signal that the hook ran; the marker proves it did its work.

Pass a bare delivery version (v0.1.0) — in the default delivery-pinned mode that re-activates the whole delivery. The name@version form (or -a <name>) is a per-artifact escape hatch, valid only when assignment_mode = per_artifact; under the default it is rejected with ATC-E0454: switch refused: … device is delivery-pinned.

Open the hatch with atc ota assignment-mode:

atc ota assignment-mode deploy@device per_artifact --for 1h   # lease the opt-in
atc ota status deploy@device                                  # MODE reads per-artifact
atc ota assignment-mode deploy@device delivery_pinned         # back to the default

--for is the safer form. The daemon re-reads the mode at the instant it activates. An expired lease therefore refuses the next lone switch, and the device returns to the delivery-pinned posture on its own. Omit --for and the override has no expiry.

The ota switch argument order is <TARGET> <VERSION> — host first, version second. A reversed form like atc ota switch v0.1.0 deploy@device parses v0.1.0 as the hostname and fails with ATC-E0303: could not resolve hostname.

Appendix: release channels

dev gets you the tip of dev within about an hour of a merge, instead of waiting for the nightly.

Installing a specific version

A track always serves its current release: https://atc.dev/agency/<track>/index.json is overwritten in place on every publish, and the install script takes ATC_TRACK but no version pin. So the curl | sh one-liner cannot give you v0.1.2, and neither can the Path B fetch script below.

Pinned stable and nightly versions come from the GitHub release page instead. Those releases keep their own binaries, so prior versions stay available:

gh release list -R Agency-Tool-Company/agency-code --exclude-pre-releases   # stable releases
gh release download v0.1.2 -R Agency-Tool-Company/agency-code \
  -p 'atc-linux-x86_64-musl' -p 'SHA256SUMS'
grep 'atc-linux-x86_64-musl$' SHA256SUMS | sha256sum -c   # verify before running
chmod +x atc-linux-x86_64-musl
./atc-linux-x86_64-musl --version                         # atc 0.1.2 (6b17235ca …, track stable)

A pinned release prints the wire generation it speaks as a trailing [wire atc/N]. Releases older than that banner print no such segment. v0.1.2 above is one of them.

The push wire accepts only an exact match, so a pinned atc older than the device’s atcd cannot push to it. See An old atc against a newer atcd is refused too in step 6.

A bannerless binary does not tell you its generation, and the release notes do not state one. For a stable release, use this list. It was read from each tag’s source and checked against the published binaries:

A bannerless nightly may speak either generation. The banner arrived after the atc/6 bump, so its absence proves nothing. Do not use atc ota status deploy@device as the probe: status uses a separate handshake and answers across a push-wire skew. A push is the probe that answers. A mismatch is refused at the push’s first stage, before any payload transfers. If the versions match, the push uploads and stages a real delivery. With autoswitch = true in atc.toml, it also activates it. Use this probe only when you intend to push that delivery.

Assets are named <binary>-<target> for the same targets the index carries (atc/atcd × linux-x86_64-musl, linux-aarch64-musl; atc also darwin-arm64), alongside SHA256SUMS and the signed index.json. Pick the target matching the machine that will run the binary. For atcd that is the device, not your laptop.

Nightly builds are published as pre-releases (dropped by --exclude-pre-releases above); list them with a bare gh release list.

The internal dev channel is the exception. Its per-merge publishes record only a lightweight tag, with no release object and no assets, to keep the Releases page clear. So gh release download has nothing to fetch for a dev build. Pin a nightly release instead, or build the binary yourself from the tag’s commit.

This is the path for testing CLI/daemon compatibility against an older release. Note that a pinned binary is outside the self-update flow: a later atc upgrade moves it back to the track’s current release.

Opting in

The dev track is internal and token-gated: the index is served only to requests carrying X-Atc-Channel-Token, which atc reads from ATC_CHANNEL_TOKEN.

# 1Password → Software vault → item "atc-dev-channel" → field "token"
export ATC_CHANNEL_TOKEN=…        # put this in your shell profile
atc bootstrap track dev           # persists in sources.toml
atc upgrade

On a machine with no atc yet, the installer takes the same variable:

export ATC_CHANNEL_TOKEN=…
curl -fsSL https://atc.dev/install.sh | ATC_TRACK=dev sh   # or ATC_TRACK=pr-<number>

The installer reads the token only from the environment, and it sends the token only to https://atc.dev/agency/dev/ and /agency/pr-*/. With no token, it stops before the first download and names ATC_CHANNEL_TOKEN.

The value must carry no trailing newline or surrounding whitespace. The edge compares the header byte-for-byte and atc sends what you exported verbatim — it rejects a blank token but never trims one — so a stray newline from a copy or a $(cat file) produces a 403 while the 1Password value looks perfectly correct. The current value is 64 hex characters — nothing enforces that length, it is just how this one was minted, so treat the count below as a fingerprint of today’s token rather than a rule:

printf '%s' "$ATC_CHANNEL_TOKEN" | wc -c   # 64 for the current token

What is actually required is that the value match the deployed one exactly, with no leading or trailing whitespace.

atc --version should now print track dev.

To go back: atc bootstrap track stable && atc upgrade. Use atc bootstrap track, not a one-shot --track — the flag does not clear a track recorded in sources.toml, so the next bare update would return to dev.

Five things that will confuse you otherwise

No token means 403 on every update. atc names ATC_CHANNEL_TOKEN in the error rather than surfacing a bare HTTP status, so it should be obvious. A 403 with the token set is one of two things: a stale value (re-pull from 1Password) or stray whitespace in it (check the byte count above). The second is the likelier one and the harder to see, since the 1Password UI shows nothing wrong.

Version numbers barely move. Dev builds are 0.1.3-rc.<minute>.<sha>, all sharing one base until the next official release. Self-update compares the content DIGEST, not just semver (the channel is published rolling), so you do get new builds. The version string is just not how you tell them apart. Use the sha.

No GitHub releases. Each publish records a bare git tag (dev-<date>-<sha>) and nothing on the Releases page. Deliberate: per-merge prereleases nobody consumed are what got this channel retired the first time. Release notes live at the URL the index carries.

Not every merge publishes. A docs-only or terraform-only merge does not trigger a release, and a burst of merges collapses into one. Expect roughly one release every couple of hours on a busy day, not one per merge.

The macOS atc arrives after the Linux binaries. Each dev release publishes the index twice. The first publish carries only the Linux targets, so a device gets its binaries without a wait for the macOS runner. When the native macOS build finishes, the release republishes the index with the darwin-arm64 entry. Between the two publishes, the index has no darwin-arm64 atc. On a Mac, an update in that window fails with release index has no artifact for package agency/atc on target darwin-arm64.

The gap is usually one or two minutes. It is longer when the macOS runner is queued or the macOS build is slow. The timeout-minutes of the macOS build (build-atc-macos-release.yml) and of the release-macos job (release.yml) set how long each job can run.

If you see that error on the dev track, wait a few minutes and run atc upgrade again. If the error stays, open the newest dev-release run in GitHub Actions and read the status of two jobs. These are build atc (macOS / Apple Silicon, after the Linux publish) and republish index with macOS atc. If either job is queued or in progress, wait for it. If either job failed, the index has no darwin-arm64 until the next dev release. Use nightly meanwhile, or build atc locally with AGENCY_BACKEND=local ./agency build -c opt //crates/atc-cli:atc.

Before you rely on it

The gate covers the index, not the binaries: those live in the shared content-addressed blob prefix and stay publicly readable by digest. It keeps customers off the channel rather than keeping the bytes secret.

Rotating the token is a tofu apply plus an env update everywhere, with no overlap window — it is baked into the CloudFront function, which cannot read a secrets store. It is a long random value precisely so it does not need a schedule.

Going deeper

• building-from-source.md — the same journey but building atc/atcd from source via ./agency (the contributor / CI path).
• deploy-your-first-update.md — end-to-end quickstart with the no-systemd / container recipe and a troubleshooting table.
• atc-toml.md — the atc.toml manifest schema and the full lifecycle-hook (ATC_HOOK_CONTEXT, @self/@bundle, state/phase) reference.
• lifecycle-hook-cutover.md — which gate each hook belongs at, and what to check if you already have commit.* hooks in the field.
• verifying-the-binary.md — independently verify the atc/atcd binary you’re running is genuine (sha256 + ed25519-signed index).
• console-config-publish-readiness.md: run this checklist before you promise a customer config publish from the web console. Until it is green, use the CLI path above.

For an ATC-Exxxx error, see docs/development/error-codes.md.

Deploy your first update

A self-contained operator path from nothing to a shipped, verified update on a device: install atc, bootstrap the on-device daemon, declare what to ship, push it, and roll back if you have to.

You do not need any of the .claude/skills/ material, Notion, or an external URL to follow this. Every command below was run against the real CLI; discover any subcommand’s full options with atc <cmd> --help.

Two pages cover this journey. This one is the short version, and it ends with a troubleshooting table. onboarding.md is the complete path: both device types, lifecycle hooks, container images, and the traps each step hides. Read one of them, not both.

Mental model (30 seconds)

atc is a controller CLI you run on your machine. It delivers artifacts to a device over SSH. A daemon, atcd, runs on each device and manages a content-addressed store at /var/atc. Every push is versioned and atomic: artifacts are staged, then the active version is flipped only when you ask (--switch). Rollback is just flipping back to an older version.

You declare what to ship in atc.toml and run atc from that directory.

What you need

• A controller machine with atc installed (Step 1). You do not need a checkout of this repo.
• A device you can SSH into (key auth preferred) with passwordless sudo for the one privileged bootstrap step. --non-interactive forces key-only SSH.
• The device’s CPU arch must be aarch64 or x86_64 (ssh user@device uname -m). Bootstrap probes it and picks the matching atcd; anything else needs a binary supplied with --from.
• The device’s init system (ssh user@device 'ps -p 1 -o comm='). If PID 1 is not systemd, skip Step 2 and use the no-systemd recipe instead. Check PID 1, not for a systemctl binary — systemctl can be present (and even answer) on devices where systemd is not running.

Step 1 — Install atc

atc is the controller CLI you run on your own machine:

curl -fsSL https://atc.dev/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"   # the installer drops atc here
atc --version

The installer verifies what it downloads (sha256 against a signed release index) before writing it. See verifying-the-binary.md to check that yourself.

You do not need to install atcd, the device daemon, by hand. Step 2 downloads the arch-matched, signature-verified atcd and installs it for you.

atc reads atc.toml from the directory you run it in.

Building from source instead? If you have a checkout of this repo and want the binaries you built, follow building-from-source.md. It covers the ./agency build path, then rejoins this guide at Step 2.

Step 2 — Bootstrap atcd onto the device (one-time)

Bootstrap probes the device’s arch, downloads the matching atcd from the same signed release origin atc self-updates from, installs it into the store at /var/atc, and enables a systemd service:

atc bootstrap atcd install --yes --non-interactive user@device
# → bootstrap: atcd started on user@device (socket /run/atcd/atcd.sock)
# → ✓ atcd <version> installed and running on user@device

The downloaded atcd goes through the same verification as a self-update: the ed25519 signature over the release index is checked against the pinned trust keys, and the binary’s sha256 against the signed digest, before anything is written or copied to the device.

There is exactly one atcd on the device: the store-managed binary the systemd unit execs, at /var/atc/channel/agency/atcd/delivery/current/atcd/atcd. /usr/local/bin/atcd is a symlink to it.

So atcd --version reports the version that symlink resolves to, which is not always the version the daemon RUNS. atcd upgrade re-points current, and the symlink follows it, but the daemon keeps serving the old process until a restart. In that window atcd --version reports the STAGED version. The reply from atcd upgrade names the running version, which it reads from the daemon on its socket.

Any target form works: a hostname, an IP address or localhost. You do not name the device. atcd names itself from a hash of its hardware identity, or from its device id when the hardware has none.

To ship a specific binary instead (one you built, or a patched build), pass --from ./path/to/atcd — it must match the device’s arch. --from is explicit intent, so a bad path is an error rather than a silent fall back to the download.

Verify the daemon is up and the controller can reach it:

ssh user@device 'systemctl is-active atcd'   # → active
atc ota status user@device                    # controller ↔ daemon; lists the (empty) store

A status (or any other command) can occasionally fail with ATC-E0308: ping ... timed out after 5s — a transient SSH-connection timeout, common on the first contact after starting atcd but possible on any contact. Retry once; it succeeds immediately (see Troubleshooting).

Want the device to install itself, with no atc? For provisioning scripts and Ansible, run atcd install on the device: curl -fsSL https://atc.dev/install-atcd.sh | sudo sh -s -- --socket-owner user. It is safe to run again, and --check changes nothing. Flags, exit codes and an Ansible task: onboarding.md, Path A on the device.

No systemd on the device? (a bare container, a minimal image) bootstrap aborts up front with ATC-E0419. Skip to No-systemd / container devices for the foreground recipe, then come back to Step 3.

Step 3 — Declare what to ship (atc.toml)

atc ota init scaffolds a minimal manifest; or write it by hand. The smallest valid atc.toml ships one bundle (a directory tree — your built app plus any service files / hooks):

[[bundle]]
name   = "app"             # on-device identity; unique across artifacts
source = "dist"            # directory to ship — must exist at push time

Put your built application under source (here, ./dist/). There is no version field — the version is a label assigned at push time (Step 4). A manifest is validated when you push; a missing source directory fails with ATC-E0504: bundle source not found.

Bundles can also carry lifecycle hooks — scripts run on the device at specific lifecycle states — see atc-toml.md for the hook syntax and the full manifest reference.

(For container images use a [[oci]] block with a source registry ref and atc oci import/oci build first — see atc oci --help.)

Step 4 — Push and activate

A bare push sends every declared artifact (the default); narrow to a subset with --only <name> (repeatable). A push stages by default; add --switch to make it the active version. The version label comes from --version (or -t name@version):

atc ota push --version v0.1.0 --switch user@device

atc ota push --dry-run user@device uploads and mutates nothing, but its report is terse: it previews only peripheral reconciliation — printing peripheral reconcile: no removals, or a table of peripherals it would remove. It does not summarize the bundles, versions, or bytes a real push would send.

Step 5 — Verify

atc ota status user@device     # current + prior version and daemon health
atc ota ls     user@device     # all on-device versions
atc ota netcheck user@device   # connectivity / daemon-reachability preflight

atc ota status showing your version as current means the update shipped and is live. The device table at the top reads:

• STORE — the on-device store root (/var/atc).
• DAEMON — the version of the atcd running on the device.
• MODE — how versions are assigned: delivery-pinned (the default and the normal mode for push-based delivery — the device tracks one pinned delivery across all artifacts) or per-artifact (each artifact resolved independently). Change it with atc ota assignment-mode <target> per_artifact, which takes an optional --for <DURATION> so the opt-in expires on its own.
• CHANNEL (in the artifact tables) — the channel each artifact tracks, e.g. local/default.

Where your files land on the device: the active version’s files are materialized at /var/atc/channel/local/default/artifacts/<name>/versions/<sha256>/. In the default delivery-pinned mode the stable pointer is not a per-artifact current symlink — it is the delivery pin: /var/atc/channel/local/default/delivery/current/<name>, a symlink that resolves (through the active delivery dir) to that artifact’s active versions/<sha256>/ dir and is flipped atomically on every switch/rollback. Point on-device services at delivery/current/<name> for a stable path. (A per-artifact artifacts/<name>/current symlink is written only in per-artifact MODE, never in the delivery-pinned default — see the MODE row above.)

Emergency rollback

Flip the active version back to an older one. Push output does not tell you the previous version — find it with atc ota ls, then switch. The signature is atc ota switch <TARGET> <VERSION> — host first, version second:

atc ota ls     user@device            # find the previous version to revert to
atc ota switch user@device v0.1.0     # re-activate it on user@device

(Order matters: a version-first form like atc ota switch v0.1.0 user@device is parsed as “host v0.1.0” and fails ATC-E0303: could not resolve hostname.)

Anti-rollback floor (opt-in)

A device can be told to refuse a delivery older than the one it already accepted. That is the anti-rollback floor. It is OFF by default. Turn it on per segment, with policy in the deployment doc:

{
  "root": {
    "policy": { "anti_rollback": true },
    "children": [
      { "kind": "rule", "name": "canary",
        "when": { "all": [ { "attr": "custom.tier", "op": "eq", "value": "canary" } ] },
        "policy": { "anti_rollback": false } }
    ]
  }
}

The root value is the deployment default. A segment that sets nothing inherits it. A segment that sets anti_rollback overrides it for itself and for every segment below. Read the resolved value back before you roll out:

atc fleet device resolve <DEVICE_ID> --channel stable
# anti-rollback   armed

The server cannot turn it off

The device reads the bit as an arm-only ratchet. The bit can arm the floor. It can never disarm one. A server that sends false to a device that already armed does nothing. This is deliberate: the floor defends against a compromised server, so the server must not hold its off switch.

To disarm a TEST robot, delete the floor on the device itself:

ssh user@device 'rm -rf /var/atc/.atc/delivery-floors/local/default'

The last two path components are the channel’s <namespace>/<name>. The device arms again on the next stamped delivery it accepts.

Limit: legacy devices never arm

The bit rides the desired-state object the server publishes for each device. An atcd that predates that object reads its desired state from POST /v1/checkin, and the server no longer serves that route. Such a device receives no channel policy and never arms. The deployment cannot change that. Upgrade the device to arm it.

Health timeout of a multi-host device

A device with several hosts switches to a new delivery on every host at once. After the switch, each host must confirm the new delivery within the health timeout. If a host does not confirm in time, the server rolls back every host. The default is 15 minutes. If your devices take longer to reboot, set health_timeout_secs in the segment policy:

{ "root": { "policy": { "health_timeout_secs": 2400 } } }

The value is from 1 to 86400 seconds. The server refuses a publish with any other value. It inherits like anti_rollback. The server also raises the value to twice the longest check-in interval of the hosts, so that one late check-in cannot cause a rollback.

No-systemd / container devices

atc bootstrap installs atcd as a systemd service. On a device whose PID 1 is not systemd (a bare container, a minimal image), bootstrap stops up front with ATC-E0419 and prints a foreground recipe. Here is the full end-to-end version of that recipe.

Critical: the binary must live at /usr/local/bin/atcd on the device. In this no-systemd recipe that path is a real binary you install, not the symlink-into-the-store the systemd bootstrap creates — there is no store-managed current/atcd to point at, because nothing seeds or manages a store here. The controller’s liveness check and auto-bootstrap reach the daemon by SSH-exec’ing /usr/local/bin/atcd attach (the [daemon] remote_path default). A daemon started from any other path (e.g. /tmp/atcd) is running and healthy, yet atc ota status / netcheck report the daemon as not present — because the controller probes the fixed path, not the running process. Install it there.

Equally critical: run atcd as the same user you SSH in as. The controller SSH-execs atcd attach as your SSH login user (the user in user@device), and the socket defaults to 0660 owned by the user that started the daemon. So if you start atcd as root (sudo atcd run) but SSH in as a non-root user, every atc ota status / push fails with connect to atcd at /run/atcd/atcd.sock: Permission denied (os error 13) — the daemon is up and healthy but unreachable. (Worse, that permission error is currently masked as ATC-E0401 atc.toml is malformed or an ATC-E0415 release-download 404 — see Troubleshooting.) The recipe below starts atcd without sudo so the socket is owned by your login user. Only steps 1–2 (install the binary, create+chown the dirs) need sudo.

On the device (the user you SSH in as, with passwordless sudo for steps 1–2):

# 1. Install the device-arch atcd at the path the controller probes. Bootstrap
#    needs systemd. `atcd install --no-service` can seed a store instead (see
#    onboarding.md, Path B); this recipe stages the binary by hand. Fetch it
#    with install-atcd.sh. It checks the signature on the release index and
#    the binary's sha256, and ATC_DOWNLOAD_ONLY makes it install nothing.
#    Set TRACK to the track that `atc --version` prints on your machine.
#      curl -fsSL https://atc.dev/install-atcd.sh | ATC_TRACK=$TRACK ATC_DOWNLOAD_ONLY=/tmp/atcd sh
#    It needs curl, OpenSSL 3, awk, and sha256sum or shasum on the device.
#    Without them, use the fetch
#    block in onboarding.md (Path B), or scp up a binary you built yourself.
sudo install -m 0755 /tmp/atcd /usr/local/bin/atcd
/usr/local/bin/atcd version   # arch check — 'exec format error' means the
                              # binary is for another CPU

# 2. Create the store root and the socket's parent dir (atcd does NOT create
#    them — on the systemd path the unit's RuntimeDirectory= does), and hand
#    both to your login user so the un-sudo'd daemon in step 3 can write them
#    and owns the socket the controller attaches through. Use chown -R:
#    if atcd ever ran as root it left root-owned subdirs a shallow chown misses.
sudo mkdir -p /var/atc /run/atcd
sudo chown -R "$(id -un):$(id -gn)" /var/atc /run/atcd

# 3. Start the daemon headless (NO sudo) so the SSH session returns AND the
#    socket is owned by your login user. setsid + redirecting all three streams
#    detaches it; a plain `nohup ... &` leaves the SSH channel hanging open.
setsid /usr/local/bin/atcd run \
  --store-root /var/atc --socket /run/atcd/atcd.sock \
  </dev/null >/tmp/atcd.log 2>&1 &
# Fresh-device sanity check: if the ssh gets wrapped in a `timeout` (so it exits
# 124/143 rather than returning), `DAEMON_UP` on stdout is a quick signal the
# daemon came up. This is only trustworthy on a FIRST bootstrap — on a re-run an
# OLD atcd may still hold the socket, so both the socket test and `pgrep -x atcd`
# can pass before (or even without) the new daemon binding. If you're re-running,
# stop the old daemon first (`sudo pkill -x atcd; sudo rm -f /run/atcd/atcd.sock`)
# before the start line above. The AUTHORITATIVE proof either way is the
# controller handshake in the next step (`atc ota status`), which only succeeds
# when the controller can actually attach to the live socket.
for i in $(seq 1 10); do
  [ -S /run/atcd/atcd.sock ] && pgrep -x atcd >/dev/null && { echo DAEMON_UP; break; }
  sleep 1
done

Release/opt builds log nothing by default, so an empty /tmp/atcd.log is normal — use pgrep -x atcd as the liveness check. To debug startup, prefix the run line with RUST_LOG=info.

Running atcd as your login user makes the socket owned by the identity the controller attaches as — the only setup that works out of the box. If you must run the daemon as a different user than you SSH in as (e.g. a dedicated atcd service account), grant that login user the socket with --socket-owner <login-user>, or add the login user to the daemon user’s group. Do not reach for --socket-mode 0666. That mode opens an admin control socket to every local process.

Started this way, lifecycle hooks run as atcd’s own uid — your SSH login user (the atc-hooks default in atc-toml.md applies to the systemd/bootstrap install, not this recipe). --hook-run-as-user <user> cannot work here: switching the hook uid needs a root (or CAP_SETUID/SETGID) daemon, and atcd refuses a hook user that resolves to its own uid. On this recipe, omit the flag.

A hook on this recipe cannot drive systemctl --user either, because the device has no user session. For a hook that must reach a root-owned file, use a supplementary group for the daemon’s user, then restart atcd. See path 1 in Root work on a device with no user session. Path 2 in that section needs systemd as PID 1, so it does not apply here.

/run is tmpfs — it does not survive a container restart. /run/atcd and the running daemon are both gone after the container restarts, so steps 2 and 3 (mkdir -p /run/atcd + the atcd run start) must run on every container start — bake them into your image’s entrypoint / init script. For a durable socket dir that survives restarts, put it under the (persistent) store root, e.g. --socket /var/atc/run/atcd.sock (and set the matching [daemon] socket_path in your atc.toml so the controller probes the same path).

The controller then talks to it exactly as a bootstrapped device — no bootstrap needed, plain SSH:

atc ota status user@device      # controller SSH-execs /usr/local/bin/atcd attach
atc ota push --version v0.1.0 --switch user@device

The first contact after starting atcd commonly exceeds the 5s ping window and fails with ATC-E0308: ping ... timed out — the recipe worked; retry once and it succeeds immediately.

If atc ota status reports the daemon as absent here, check pgrep -x atcd on the device (and /tmp/atcd.log, if you started it with RUST_LOG) and confirm the binary is at /usr/local/bin/atcd (step 1).

Troubleshooting

Next steps

• atc-toml.md. The full manifest schema and the lifecycle-hook reference.
• examples/heartbeat-module/. A repo that ships a module: an executable the device runs after the delivery commits, rather than a payload it only unpacks. Its README says what the example proves and what it cannot do. modules-today.md carries the whole map.

Getting started: human identity (atc auth login)

This is the quickstart for a person authenticating to an atc-server control plane with the atc CLI — as opposed to ../development/building-from-source.md, which walks through shipping software to a device. It assumes an atc-server is already running somewhere and you know its URL; if you’re standing up the server itself, see docs/runbooks/workos-identity-setup.md (WorkOS) or docs/runbooks/local-identity-setup.md (WorkOS-free).

Mental model

A server exposes exactly one of four auth methods, discovered automatically — you never configure the CLI to match it:

atc never accepts a token on the command line — only an interactive prompt, stdin, or a file. That’s deliberate: a token on argv ends up in shell history and ps.

First login

atc auth login --server https://atc.example.com

• If the server is workos_device: atc prints a URL and a short code, then opens the URL in your default browser. Approve in the browser, and atc finishes automatically once you approve — no need to press anything locally. atc does not open the browser over SSH, on Linux with no graphical display, or with --no-browser (or ATC_NO_BROWSER=1). Then open the printed URL yourself. If the browser cannot start, the login continues and prints the reason.
• If the server is a token mode: atc prompts for the token on your terminal (input hidden, never echoed). Get the token from whoever administers the server (a shared admin token, or one minted for you personally in local mode).
• If the server is none: you’re done immediately — there was nothing to authenticate.

A successful login activates a profile (default name default) that every later atc command uses automatically. Confirm it:

atc auth status

Working with more than one server

Name a profile per server so you can switch between them:

atc auth login --server https://staging.example.com --profile staging
atc auth login --server https://prod.example.com --profile prod
atc auth profiles         # see everything you have stored
atc auth use staging      # switch which one is active

Everything after login (push, ota status, etc.) uses whichever profile is active — there’s no need to repeat --profile on every command.

Logging out

atc auth logout

For a WorkOS login this actually revokes the session server-side (not just a local delete) — the CLI confirms the revocation before wiping local state. If it can’t confirm (network blip, server hiccup), it says “logout incomplete” rather than silently claiming success; run atc auth logout --retry to resume. Full recovery paths (including the un-revocable case) are in docs/runbooks/workos-identity-operations.md.

When something goes wrong

Full day-to-day operations (session revocation, incident recovery, mode transitions) live in docs/runbooks/workos-identity-operations.md.

See also

• docs/runbooks/workos-identity-setup.md — stand up a WorkOS-backed server.
• docs/runbooks/workos-identity-operations.md — login/logout deep dive, quarantine recovery, session revocation.
• docs/runbooks/workos-identity-rotation.md — key/JWKS rotation (plus the fail-closed readiness kill switch: a signed lease or the explicit ATC_WORKOS_READINESS_INSECURE=1 opt-out) (server operators).
• docs/runbooks/local-identity-setup.md — the WorkOS-free local mode.

atc.toml reference

atc.toml declares what you ship to a robot and how it behaves across the delivery lifecycle. One file feeds both delivery planes: atc ota push sends it over SSH, and atc ota publish --deliver and atc channel deliver embed the same hook tables in the signed manifest a polling device fetches for itself. This page documents the lifecycle-hook schema, because it is the part the parser validates most strictly and the part that has caused the most field confusion.

The schema below is derived from the parser (crates/atc-lifecycle/src/config.rs), which is the source of truth. If this doc and the parser ever disagree, the parser wins — file a docs bug.

There is no [[module]] table. A repo declares modules in atc.kdl, and no TOML syntax reaches one. What a declared module can and cannot do is mapped, with citations, in modules-today.md; a working repo is examples/heartbeat-module/.

Which file atc reads. With no -c, atc reads the nearest atc.toml or atc.kdl in the current directory or a parent. When one directory holds both, atc reads atc.kdl and prints a note that names the atc.toml it did not read. Set ATC_KDL_CONFIG=0 to read the atc.toml instead. The full rule is in Where atc finds the file.

What a bundle’s source ships

[[bundle]]
name = "myapp"
source = "dist"

source names a directory. atc ships every file and every resolvable symlink under it, subdirectories included, at any depth. The two exceptions are the bundle’s exclude patterns and Python bytecode, both described below. Nothing declares that layout, and every push reads the directory again.

Three things the walk does not carry:

• An empty directory has no entry of its own, because a directory is implied by the paths under it.
• A FIFO, socket or device node is skipped, and the push still succeeds. A bundle that needs one must have a hook create it on the robot.
• A dangling symlink is a hard error, not a skip. The push stops with ATC-E0510 and names the link, because a broken link shipped verbatim fails confusingly on the device. An unbuilt node_modules/.bin is the usual source. If the robot does not need that directory, add it to exclude.

Leave paths out with exclude

[[bundle]]
name = "myapp"
source = "dist"
exclude = ["**/.git", "**/*.log", "tmp"]

Each pattern is a glob that matches the path relative to source. * stays inside one directory, so *.log matches only at the top of source. ** crosses directories, so **/*.log matches at any depth. A directory that matches is left out with everything under it, and the walk does not enter it. So a dangling symlink inside an excluded directory does not stop the push.

A pattern that can never match stops the parse with ATC-E0531. That is an empty pattern, a leading or trailing /, a . or .. segment such as ./build, or a glob that does not compile.

Python bytecode is left out by default

The push leaves out every __pycache__ directory and every .pyc and .pyo file. The robot cannot use them. The push does not keep a file’s modification time, and CPython checks a timestamp-based .pyc against the time of its source. So on the robot CPython rejects the .pyc and compiles the source again. On a read-only store it cannot write the result either.

When the default left something out, atc ota push prints one line:

atc: note: bundle web: left out 2 Python bytecode files. Set `ship_python_bytecode = true` to ship them.

Set ship_python_bytecode = true on the bundle only if you build hash-checked bytecode (python -m compileall --invalidation-mode checked-hash). CPython accepts that form whatever the file times are.

In atc.kdl, both keys are children of the bundle node:

bundle myapp {
    source "dist"
    exclude "**/.git" "**/*.log"
    ship-python-bytecode #true
}

On any other artifact kind, loading the file fails with ATC-E0776.

Point source at a clean build output

Apart from exclude and the bytecode default, nothing filters the tree. So a directory you also work in ships what your tools left there. A checkout used as source ships its whole .git history, and a test run leaves .pytest_cache.

Nothing fails on the robot. The push still sends the bytes, and the store holds them for every version it keeps.

atc ota push names what it found before it uploads:

atc: warning: bundle web: 2 staged files are in a tool cache: .pytest_cache/v/cache/nodeids, .pytest_cache/v/cache/lastfailed.
atc:   The robot never reads a tool cache. The store holds these bytes for every version it keeps.
atc:   Point `source` at a clean build output. Delete the cache directory before the push.

It reports two kinds: a tool cache (.mypy_cache, .pytest_cache, .ruff_cache, .tox, node_modules/.cache) and a .git repository. Each warning names up to three paths and counts the rest. The push continues: a bundle that ships one of these on purpose is legitimate, and a refusal would break it. To leave one out for good, add it to exclude.

So reorganizing inside source costs no atc.toml edit. A flat dist/myapp that becomes dist/bin/myapp plus dist/share/VERSION needs no change here, and the next push carries both new directories. atc ota push --dry-run previews the upload size per artifact, though not the paths it would send.

The dist directory that excludes itself

A build that stages into ./dist/ usually copies the project tree first. An unanchored exclude then removes more than the staging directory:

rsync -a --exclude=dist ./ ./dist/     # WRONG: drops every nested dist/
rsync -a --exclude=/dist ./ ./dist/    # anchored: only the top-level one

rsync matches an unanchored pattern against every path component, so --exclude=dist also drops node_modules/<pkg>/dist/. Several npm packages keep their runtime files there. The bundle still validates and still pushes, and the program fails on the device with a missing module.

Anchor the pattern with a leading /, or stage into a directory whose name appears nowhere else in the tree.

A binary that reads files beside itself

A delivery path carries a content hash, so it changes on every version. A bundle that ships bin/ and share/ together must resolve share/ from the directory of its own executable, and must ask the platform for that directory:

Do not assemble the path by string concatenation, such as exe + "/../share". That form breaks as soon as a caller reaches the program through a symlink or a relative path. It breaks on the device, not on the build machine.

The shell row resolves the link first. $0 is the path the caller typed, so a wrapper reached through a convenience symlink would otherwise look for share/ beside the LINK. readlink -f also resolves delivery/current. That lands on the version directory the program runs from, where bin/ and share/ are siblings.

A hook is a different case. A hook runs in a sibling of delivery/, so it must read current_root out of ATC_HOOK_CONTEXT. See “current_root: the path a service unit must name” below.

A third-party binary whose version you do not own

A cloned upstream project bakes its own version into its --version output, and a pin to one commit cannot change it. The bundle version and the program version are two labels, and atc owns only the first.

atc ota push --version v1.0.0 sets the bundle version. That is the label atc ota ls reports and atc ota switch takes, and it is independent of what the shipped program prints.

To report it from the bundle as well, ship a VERSION file beside the binary and a wrapper that reads it:

dist/
  bin/jq             # the upstream binary, untouched
  bin/jq-wrapped     # the wrapper below
  share/VERSION      # v1.0.0, written by your build

#!/bin/sh
# bin/jq-wrapped - report the bundle version, then hand off to upstream.
set -eu
self=$(readlink -f -- "$0")
here=$(cd -P -- "$(dirname -- "$self")" && pwd)
if [ "${1-}" = "--bundle-version" ]; then
  cat "$here/../share/VERSION"
  exit 0
fi
exec "$here/jq" "$@"

Write share/VERSION from the build step that passes --version to atc ota push, so the two labels cannot drift.

Shipping a compiled binary

This section is about compiled binaries. A node or python bundle ships source, so skip it.

Run file on the artifact, then ldd. An autotools or libtool build writes a wrapper shell script under the binary’s own name, and leaves the real ELF under .libs/. ldd tells you nothing useful about that wrapper, which is why file comes first. Configure with --disable-shared to get one self-contained binary instead.

ldd then lists what the binary loads at start. The robot must carry every library it names, or the process dies with error while loading shared libraries.

A second failure reads differently. A library that resolves by soname, but is not the one you built against, loads without complaint. The process then dies at symbol-binding time:

/app/bin/myapp: symbol lookup error: /app/bin/myapp: undefined symbol: jv_free

The file was found, so the “cannot open shared object file” message never appears. A non-static autotools build reaches this often: the real ELF under .libs/ links the soname, and the build host already carries an older copy of it. The binary runs on that host and dies on the robot, or the reverse. readelf -d <file> | grep NEEDED names the sonames it will accept.

The toolchain decides how you link statically, and no flag is a guarantee. For autotools, configure with --disable-shared --enable-static. That governs the project’s own libraries, not libc, so add LDFLAGS=-static when you want libc in the binary too. For Go, build with CGO_ENABLED=0 go build. A plain go build looks static and often is not, because the standard library pulls cgo in for DNS lookups and for os/user. For Rust, build for a musl target.

Check the result rather than trusting the flag. file should say statically linked, and ldd should say not a dynamic executable.

The glibc risk runs one way. glibc keeps older binaries working, so a binary built against an older glibc than the robot’s runs there. Only the reverse needs a static link or vendored libraries: a newer build machine, an older robot. Run ldd --version on both machines to see which case you have, rather than deploying to find out.

atc ota push reads the staged binaries. It refuses a binary that binds a newer glibc than the robot reports, and it warns about a libtool wrapper script. It says nothing about shared libraries, because a dynamic link is the right answer when the robot carries them.

Loader search order, $ORIGIN rpaths, and an install.exit hook that refuses a tree whose binaries cannot resolve: shared-library-resolution.md.

A device with several hosts

A robot can carry several computers, each with its own atcd, for example a nav computer and a compute computer. Each computer is a host, and its role is its job in the robot. Set a host’s role when you install atcd: atc bootstrap atcd install --role nav.

Give each artifact for one role a role. An artifact with no role goes to every host. The [device.hosts] table maps each role to the SSH target of its host:

[[bundle]]
name = "planner"
source = "dist/planner"
role = "nav"

[[oci]]
name = "team/vision"
platform = "linux/amd64"
role = "compute"

[[bundle]]
name = "common"
source = "dist/common"

[device.hosts]
nav = "ops@nav-14"
compute = "ops@compute-14"

A role is 1 to 63 of a-z, 0-9, - and _, and starts with a letter or a digit. atcd refuses an artifact whose role is not its own.

atc ota push --device --version v2 moves every host to v2 as one cutover:

1. It reads each host, and refuses a host whose role is not its key in [device.hosts]. A host that cannot be reached stops the push before any host changes.
2. It pushes v2 to each host with no switch, with only the artifacts for that host’s role and the artifacts with no role.
3. It switches every host to v2 at the same time. Each switch runs the activate and commit hooks.
4. If one host fails, or does not finish its switch within --health-timeout (default 15m), the push rolls back. Every host that runs v2 switches back to the version that every host ran before. A new device, whose hosts share no earlier version, has nothing to roll back to. The error then names each host that runs v2.

--device needs --version or -t, because a second run finds the cutover by its label. For the same reason it refuses --overwrite: push changed content under a new label. The push keeps no state. If it stops in the middle, run the same command again: it reads each host and finishes the cutover. A host that cannot go back to the old version is named in the error, with the atc ota switch command that moves it.

A host that reaches the robot through a gateway needs a ProxyJump entry in your ssh_config. Only a host whose atcd reports the device-cutover-v1 capability can take part. An older atcd is refused with a message to upgrade it. atc.kdl spells an artifact role (role "nav") and this table (device { hosts { nav "ops@nav-14" } }), so atc ota push --device works from either file.

Files signed with each version: attachments

A [[bundle]] or [[oci]] table can name files to keep with each version that atc ota publish uploads. Examples are a wiring diagram or a test report. Devices never download them.

[[bundle]]
name = "field-config"
source = "dist"
attachments = ["docs/wiring.txt", "reports/bench.txt"]

• Each path is relative to the manifest and stays inside its directory: no absolute path and no ... The file is stored under its file name, which must be 1 to 128 of A-Z a-z 0-9 . _ - and start with a letter or a digit. Two files with one name are refused.
• atc ota publish reads each file before it uploads anything. It refuses a file over 256 KiB, and more than 32 files for one artifact.
• The hash of each file is signed into the version’s metadata, so it is part of the version’s identity. A changed file is a changed version, and needs a new --version.
• The publish uploads each file after it creates the version. The server takes only bytes with the signed hash, and nobody can delete the file. A rerun uploads the files again, so it finishes an interrupted publish.
• A delivery does not carry the hashes, so devices see no change.
• atc ota attach ls --version <artifact>@<version> lists the files. A signed name that has no upload yet shows not uploaded.

atc.kdl has no spelling for attachments yet, so a manifest that uses them stays atc.toml.

[ownership]

atc ota publish --deliver reads this table. One manifest owns the delivery config of a server channel: its hooks, workloads, peripherals, state_dirs and member roles. Another manifest that ships artifacts into that channel defers to the owner. The ownership block of atc.kdl is the same table.

[ownership]
id = "robot-base"
owns = ["stable", "beta"]
defers = { canary = "robot-core" }

The channels are server channels, the --channel of atc ota publish. They are not the [channel] table that atc ota push targets. An id and an owner id follow the rules of an artifact name. A file with no [ownership] table makes no claim.

atc ota push, atc ota publish and atc config lint refuse these tables:

atc config lint also warns when owns names a channel more than once (ATC-E0588). That changes no delivery.

The manifest does not list its server channels. So the lint cannot find a channel name that the org does not have. atc channel owner <channel> shows the owner that the server keeps for a channel. Release through atc-server tells what the server does with the claim.

Lifecycle hooks

A lifecycle hook runs a script on the robot at a specific (state, phase) during a delivery. The table key is exactly:

[<scope>.lifecycle.hook.<state>.<phase>]

The .hook. segment is required. Omitting it (e.g. [lifecycle.commit.exit]) is rejected at push time — it does not silently wire nothing.

Body keys

timeout_secs = 0 is push-only. A signed (pull-plane) delivery carrying it is refused on the device — the daemon stays on the prior delivery and reports hook_config_invalid — because an unbounded pre-flip hook wedges the poller with no retry budget to fall back on.

No other keys are allowed in a hook body. A stray run / command / exec (or any other key) is rejected. A [delivery.hook.*] or [lifecycle.hook.gc.*] bookend takes script and timeout_secs only.

A hook script that the artifact installs

If the artifact can carry the script in its own tree, use @self instead. Before the install, a @self hook is the active version’s own hook, and a first delivery runs none (Which version’s script @self runs). This section is for a script that must live at a fixed system path.

Sometimes the artifact itself installs the script its hook runs. For example, a .deb installs /usr/lib/agent/pre-download.sh, and the download.enter hook runs that path. On the first delivery to a device the file does not exist yet. Without a policy, the hook fails with ENOENT and atcd refuses the delivery that would install the script.

Set allow_missing_initial = true for this case:

[[bundle]]
name = "agent"
source = "dist"

[bundle.lifecycle.hook.download.enter]
script = "/usr/lib/agent/pre-download.sh"
allow_missing_initial = true

The first delivery of agent to a device skips the hook. Each later delivery runs it. If the script is missing on a later delivery, the hook fails the delivery as usual, because the package that owns it is already installed.

Set allow_missing = true to skip a missing script on every delivery.

The rules:

• Set one key or the other. A hook that sets both is refused, even when one of them is false.
• Both keys need an absolute script. On @self/… or @<name>/… they are refused, because atc places those trees itself.
• Only a missing file is skipped. A script that exists but has no +x, a path that escapes its allowed root, and every other pre-spawn check still fail the hook.
• “No earlier version” means the device has no prior version of this artifact. The uninstall gate and atc ota run-hook are not deliveries, so allow_missing_initial never skips there. allow_missing does.
• A [[peripheral]] hook refuses both keys. A peripheral hook is the flash, and a skipped flash would still record the peripheral as flashed.
• A symlink at the path whose target does not exist is not a missing script. It fails the hook.
• A stage hook (top-level [lifecycle.hook.*]) takes allow_missing only. It belongs to no one artifact, so it has no first delivery.
• A skipped hook passes its gate. The push output prints a → <state>.<phase> <script> skipped (script missing) line for it. atc ota status lists it in a Skipped Hooks table, and the lifecycle_log rows of atc ota status --json show the outcome skipped_missing. The lifecycle.hook span carries outcome="skipped_missing".
• A device whose atcd predates these keys ignores them, and a missing script fails the hook as before.

States and phases

enter refuses, exit rolls back. On a push, on a pulled delivery and on an atc ota switch, a hook that exits nonzero at enter refuses the step. The state’s work never started, so atc undoes nothing. It runs no rollback.*, no failure.*, and no activate.* for the outgoing version. current does not move, and the push or switch reports refused, the device did not change. A hook that exits nonzero at exit answers after the work, so atc rolls the delivery back.

commit.enter is the one exception. It runs after the pointer flip, so a refusal there rolls back like an exit refusal. delivery.pre refuses with no ladder, as before. A switch crosses two of these gates, activate.enter and deactivate.enter. See Which of these run on an atc ota switch.

A refusal is clean only when every hook that ran at that gate exited nonzero. Some gates run more than one hook, for example a deactivate.enter with a stage hook and one hook per member. If one of them passed, it may have done its work, so atc rolls back as before. A hook that times out, is killed, or cannot start also rolls back, because it may have stopped part way.

A stage hook that refuses at activate.enter or deactivate.enter stops that gate, on a push and on a pulled delivery. No artifact hook runs beside it at that gate. A stage hook that exits nonzero there stays a clean refusal. One that times out, is killed, or cannot start still rolls back. At deactivate.enter the outgoing version’s hooks do not run, so atc records no teardown. Before issue #7215 a push still ran the artifact hooks at both gates.

Write an enter hook so that it changes nothing when it exits nonzero. Its refusal tells atc that the device is as it was.

enter and exit are the only two. There is no error phase to declare — nothing dispatches one, at any state, so <state>.error is rejected at push.

The authoritative list is atc_lifecycle::State::ALL (crates/atc-lifecycle/src/state.rs).

The order they run in

check.enter → check.exit
  → download.enter → download.exit
  → install.enter  → install.exit
  → activate.enter                    (may the new version go live?)
      → deactivate.enter              (the outgoing version stands down)
      → deactivate.exit
      → [the current pointer flips]
  → activate.exit                     (the new version confirms it is live)
  → commit.enter → commit.exit

deactivate runs nested inside activate, which puts every step that can fail ahead of the one destructive step. A veto at activate.enter has torn nothing down.

A name shared across versions

Read this order before you write a hook that claims a name the device holds only once. Examples: a systemd unit name, a TCP port, a lock file, a PID file. Every version claims the same name. The order above decides who holds it at the end.

Start the service at activate.exit. Never at activate.enter. activate.enter runs before the outgoing version’s deactivate.*. A start placed there is stopped again by the version it replaces. The push then reports success with the device on the new version and the service down. activate.exit runs after both deactivate gates, so the order pays the stop back.

Make the start a restart, not an enable --now. deactivate.* reads the OUTGOING version’s declaration. So the hook that stops the service is the one already on the device. Add the pair in v2 and push it over a v1 that declares none, and nothing stops v1. systemctl --user enable --now then exits 0 against a unit that is already active, does not restart it, and leaves v1’s process holding the name. The delivery commits green, atc ota status names v2, and the robot runs v1. A first deploy dispatches no deactivate.* at all, because there is no outgoing version to tear down.

Stop the service at deactivate.enter or deactivate.exit. Both run before the flip, and the incoming activate.exit runs after them. So a stop there needs no test for “am I still the current version”. The order answers that question.

To ask “may this version stand down now?”, put the stop at deactivate.exit. Then deactivate.enter is free for that check, and it runs before anything stops.

Know what a refusal there does. On a push, on a pulled delivery and on a switch, a refused deactivate.enter leaves the device as it was: the outgoing service keeps running, and nothing restarts it. A push does not try again, so push again when the device is ready. A poll tries again on the next poll. To ask the poller to retry sooner or later, exit 21 instead; see How long a pulled delivery defers.

A refused deactivate.exit starts the unwind ladder, because the stop already ran. The ladder runs the outgoing version’s activate.* again to bring the service back.

A worked pair of hooks is in Worked example: a long-running service.

The unwind track is rollback.enter → rollback.exit, and failure.* after that if a rollback hook itself fails.

A rollback restores the delivery that the vetoed one displaced. A first delivery displaced nothing, so a veto after its flip removes delivery/current instead. The push footer then reports “no earlier delivery to restore”, and a service unit that starts from that pointer fails until the next successful push. Give such a unit a ConditionPathExists= line; see install_unit.sh in the worked example.

The pointer flip used to happen between commit.enter and commit.exit. If you have existing commit.* hooks, read the hook cutover runbook: they all still run, but three gates changed which side of the flip they sit on.

uninstall.* is not on this graph at all. The periodic housekeeping sweep and atc ota rm both dispatch uninstall.enter then uninstall.exit for each version they are about to reclaim, and delete only if both return 0. Both run before the deletion, because an uninstall.exit script lives in the tree that is about to go.

Which of these run on a pulled delivery

atc ota publish --deliver and atc channel deliver --lifecycle-config author the same three hook tables atc ota push does (artifact, stage, delivery) from the same atc.toml, using the same parser, and sign them into the manifest. A polling device re-validates those bytes against the manifest signature before it runs anything, but the declaration it runs is yours, unchanged.

All fourteen forward cells now run on a pulled delivery. A pulled delivery walks delivery.pre, then per member check.enter → check.exit → download.enter → download.exit → install.enter → install.exit, then activate.enter → deactivate.enter → deactivate.exit → [flip] → activate.exit → commit.enter → commit.exit → delivery.post. That is push’s order and the whole FORWARD pull walk. The unwind ladder is a separate walk. It is not narrower either.

Read this before you publish to a pull-fed fleet. Until now those six cells were accepted, signed, delivered, and never run. Your channel may already publish a check.*, download.* or install.* hook. A device that upgrades past this change starts running it. Check what those scripts do first.

@self resolves the same way on a pull as on a push. At check.*, download.* and install.enter the hook is the ACTIVE version’s own, with that version’s atc.toml entry and script. From install.exit on, it is the incoming version’s (Which version’s script @self runs).

check.*, download.* and install.* cross one gate per member, on both planes. They have no delivery-wide barrier, so they run at artifact scope only. A top-level [lifecycle.hook.check.enter] table is refused at publish, and a device that somehow receives one warns about it on atcd.pull.undispatched_hook_total. Every other state runs at both scopes.

A member the device already holds crosses check.* and neither of the other two. It fetches nothing and places nothing, so a download.* there would say bytes arrived when none did. Push makes the same skip, member by member, for a version the device holds when that member’s check.exit runs, active or not (#7217). The push transfer itself does not change.

delivery.pre now runs before any byte is fetched. That is where push runs it. On this plane it used to run after the poller staged every member. Two things follow.

A hook that read the incoming version’s files in delivery.pre no longer finds them. Move that work to install.exit or activate.enter.

A poll whose download then FAILS has run delivery.pre already. So a channel stuck on an unfetchable blob re-runs the hook at poll cadence. The same failure on a push costs one operator-driven attempt.

The delivery bookend closes on every ending that ends the walk. delivery.post runs at the end of a pulled delivery whose delivery.pre passed. That holds for a staging failure, for a refusal before the flip, for a refusal after it, and for a spent retry budget. Push pairs the two bookends the same way. A delivery.pre that refuses or defers opens nothing, so the poller runs no delivery.post for it.

One case is neither. A poll superseded between its hooks and its commit leaves the delivery RESUMABLE, so it runs no delivery.post of its own. The poll that finishes the delivery runs it. If that delivery is never advertised again, its delivery.pre stays unpaired.

A deferred poll is not an ending, so it runs no delivery.post either. The poller has a retry scheduled for the same delivery. So a hook that exits 21 at activate.enter, after delivery.pre passed, leaves that delivery.pre unpaired for that poll. The poll that ends the delivery closes the bookend: the commit, a refusal, or an operator-set retry cap that runs out. Do not read the pair as 1:1 across a chain of deferred polls.

A delivery.post that refuses still runs once. The bookend that reports a failure can be the failing hook. The poller records that refusal, keeps the reason the walk already reported, and starts no second unwind.

What a refusing hook costs is not identical on the two planes. A hook that refuses after the flip un-flips the delivery on both. The prior delivery becomes the active one again, and the incoming version’s modules are drained.

The compensation ladder then runs, on all three planes. It dispatches rollback.enter, then the compensating activate.* for the outgoing version, then rollback.exit. If a gate refuses, the compensating activate.* does not bring the outgoing version back, or the take-back cannot restore the prior delivery, it also dispatches failure.enter and failure.exit. The compensating activate.* runs no stage-scope hook. So if a stage deactivate.* stopped the workload and no forward stage activate.* passed after it, the workload stays down, and failure.* follows. The ladder’s own cells honour no --skip-hook and no --hook-env: an operator flag may not suppress a recovery hook.

If the compensating activate.enter refuses, its activate.exit does not run. That is the rule for every refused enter gate (#6389): the state’s work did not start, so its exit has nothing to close. It is a refusal only when every hook that ran exited non-zero, and every member that deactivate.* stopped ran an activate.enter hook. Otherwise activate.exit still runs. A member with no activate.enter did not refuse, and its activate.exit may be what restarts it. After a timeout, a signal or a dispatch fault, the device state is not known.

A ladder that completes ends the delivery rolled_back. A ladder that does not complete ends it failed, and failure.* runs. Each member’s push row (pushes) takes the same status. This applies to a push whose pointer flip landed, and to a pull whose flip landed. It also applies to a push or a pull that a hook refused before the flip, when the ladder brought back members that deactivate.* stopped. A push that fails before any teardown has nothing to roll back, so it ends failed. On a pull, a compensation before the flip that does not bring the outgoing version back runs failure.*. If a member of that version declares no activate.*, nothing can bring it back, so the delivery ends failed and the channel accepts the next pulled delivery (#8055). Otherwise, for example after a refusal by the outgoing version’s own activate.enter, the delivery stays in_progress, and the next poll or the next start tries again.

Before #6750 and #6939, a ladder whose hooks refused could still report rolled_back, and a clean ladder before the flip reported failed. Before #6977, a pull ladder ran no compensating activate.* after the flip, and a dirty pull ladder before the flip still ended rolled_back.

The compensating activate.* names the version it restores as version.new_sha. So a hook that pins or marks new_sha acts on the version that comes back. On the push ladder version.old_sha is the failed version. A pull compensation, a switch unwind and a boot recovery name the restored version in both fields. So do not tear down old_sha unless it differs from new_sha. On rollback.* and failure.*, new_sha is still the failed incoming version.

An unwind hook cannot stop the unwind. A refusal at any of those four gates is recorded and the take-back continues, because the ladder is already the failure path. This differs from a forward gate, where a refusal is a veto.

Before #3298 the poller dispatched none of the ladder, so a rollback.* hook on a pull-fed device parsed, published without error, and never fired.

A hook that refuses before the flip at an exit gate is compensated on both planes, in the poll or push that heard the refusal. If the outgoing version was already torn down, the poller re-runs that version’s activate.* to bring the workload back, under the auto_rollback_veto trigger. A refused enter gate owes no compensation, because nothing was torn down (#6389).

Nothing bounds a permanent pre-flip refusal on the pull plane. A veto writes no retry-budget row, so the next poll dispatches the whole pre-flip half again. A deactivate.exit hook that refuses forever therefore stops and starts the outgoing version once per poll. Each of those polls also runs that version’s two activate gates. A deactivate.enter hook that refuses forever stops nothing: the poll ends at the refusal. A defer (exit 21) is not bounded either, by design (#6431): see How long a pulled delivery defers.

No forward state runs on one plane only. check.*, download.* and install.* used to. The poller fetched and staged bytes without walking them, so such a hook parsed, published without error, and never fired. It fires now (#3412 §1). See Which of these run on a pulled delivery for what changes on a fleet that already publishes one.

Read your peripheral hooks before you upgrade a pulling fleet. An unattended device on a firmware channel writes to hardware on a poll. No operator is present. ATC_HOOK_CONTEXT.trigger reads pull on those runs. It read push on every peripheral hook before this change, on both planes.

Peripheral firmware converges by pull. A bundle a peripheral subscribes to used to be refused at the commit with ATC-E0156. That refusal told the operator to use atc ota push. The poller now runs the flash ladder push runs. It admits the firmware-payload.toml and parks every subscribed peripheral BEFORE the version flip. Then it runs each peripheral’s commit.exit and records last_flashed_sha after it. A flash hook that refuses before the flip leaves the device on the prior version.

A peripheral subscription is device-local and names no channel. The ladder used to read subscriptions in the channel the delivery lands on. Every operator route writes the seeded local/default channel, so the read returned no rows on any other channel. No pulled delivery flashed anything. Every subscription read now keys on the peripheral and the artifact. So the ladder finds the operator’s subscription whichever channel the delivery lands on. atc ota peripheral subscribe and atc ota peripheral apply still take no channel argument.

A peripheral hook must be idempotent. A poll that flashes and then loses its commit re-flashes on the next poll. A re-poll that skipped the flash would flip the pointer over hardware still holding the prior firmware. A crash between the two does the same. Push has the same rule for the same reason: a re-push of a member that is not yet active re-runs the ladder. Write the flash so that running it twice against one payload sha is the same as running it once.

The ladder runs only where the device can run it: pull_plane_hooks_enabled = true and a signature-verified release. Anything else refuses with ATC-E0156 and leaves the prior firmware live. That covers the flag off and an unsigned release. Pull never flips firmware it did not flash.

A release does NOT have to declare a lifecycle hook of its own. It used to. That was a limitation rather than a rule. A release can carry the [[peripheral]] declarations of its source file, hooks included, in the signed manifest. A release that carries none flashes with the device’s fallback: the [[peripheral]] blocks of atcd’s --config file, which atcd imports once into its store at the first start of this release. The old condition refused such a channel.

Such a release now converges. The device resolves the walk the ladder hangs off when a member of the release has a subscribed peripheral. It dispatches no delivery hook, because the release declares none.

That refusal could not fire on a pulled delivery while the subscription read was channel-scoped. It fires now. Such a delivery stops instead of flipping a subscribed member unflashed.

--skip-hook and --hook-env reach no pull cell, and that includes peripheral hooks. This is deliberate, not an omission. Both flags scope to one operator command: atc ota switch --skip-hook commit.exit skips that gate for that switch only. The next command runs the hook again. A poll has no operator on the other end. There is no invocation for an override to scope to.

A pull-plane override has two possible homes, and neither one is safe. In the signed manifest, a skip would let whoever holds the publish key disarm a device-owned safety gate. The device already refuses that shape of privilege for root hooks: the daemon.pull_plane_root_hooks setting in atcd’s store is device-owned for that reason, and the manifest cannot set it.

In atcd’s settings store, a skip would be a standing disarm that no fleet-side surface reports. A device-local --hook-env is worse. The daemon applies extra_env after every framework variable. A standing entry would therefore override ATC_TRIGGER for every later delivery on that device.

To stop hook dispatch on one pull-fed device, run atcd settings set daemon.pull_plane_hooks_enabled=false on the device. Read the comment on that key first. A device that ran with the key off can wedge a channel when you turn the key back on.

One smaller difference: timeout_secs = 0 is refused on the pull plane. See Body keys.

uninstall.* runs on both planes. A version carries its own declaration in an .atc/lifecycle.pb sidecar next to its files, and both planes write one. Push writes it at that artifact’s install.exit. The poller writes it when it commits the delivery that activates the version.

On a pulled version the poller writes a sidecar only when three things hold. The delivery is signature-verified, it carries hook declarations, and the device has pull-plane hooks enabled. An unsigned or legacy delivery declares no hooks at all, so there is nothing to write.

One further case has no sidecar even when all three hold. The server can ask a device to keep a version resident without activating it. Such a version gets its declaration only when a later delivery makes it active, so housekeeping reclaims it silently if it never becomes active.

Read the daemon log if a teardown does not run. The poller warns whenever it commits a delivery whose declared hook config it could not read back.

The sidecar feeds more than the sweep, so expect two other hooks to start running. A version’s own declaration is also what a later ota switch reads for its activate.* hooks, and what a push reads for the outgoing version’s deactivate.* hooks. Before this, a pulled version answered “no declaration” to both, and both were skipped. Now they run, and a deactivate.* hook that refuses can veto the push that displaces a pulled version. That is the intended behaviour on both planes. Check your deactivate.* and activate.* declarations before you publish to a pull-fed fleet.

How long a pulled delivery defers

A hook that exits 21 on a pulled delivery defers the activation. The device stays on its running delivery and reports deferred, which the server reads as in progress. A deferring device is not a failed one, so it does not count toward a rollout’s failure rate.

The poller tries again for as long as both of these stay true:

• the hook keeps exiting 21, and
• the server keeps the delivery as the device’s desired state.

It stops when the hook exits 0 (the delivery activates), or when the server advertises a different desired state. A refusal (a veto) ends the deferral too and reports failed, but a pre-flip veto is not final: the next poll runs the walk again. A new desired state is a new delivery, and the poller applies it at once. A deferral survives an atcd restart, because its state is in the device store.

A deactivate.* hook can stop the running workload before a later gate defers. The poller then starts that workload again in the same poll, so the device runs its prior delivery while it waits. The next attempt stops it again, once per attempt.

A deferring hook can set the wait. To do this, print a line retry-after: <seconds> on stdout before you exit 21, for example retry-after: 120. The key is case-insensitive, and the first such line counts. The poller waits that many seconds, plus up to 10 % jitter, but never longer than hook_retry_max_secs. A value of 0, or one that is not a whole number, is ignored. When several per-artifact hooks defer at the same gate, the largest value wins, so no hook runs again before the time it asked for.

Without a retry-after: line, each retry waits longer than the one before. The first wait is hook_retry_default_secs (default 30 s), and each later wait doubles it, up to hook_retry_max_secs (default 300 s). Every deferral counts as an attempt, with a hint or without one. So after a hinted deferral, the next deferral without a hint takes the next backoff step, not the first. Once the backoff reaches the ceiling, a deferral that lasts all day runs the hook about once every five minutes.

A retry that is due runs at the next check-in, so the wait is never shorter than the check-in interval.

Each retry walks the delivery again from the start. The hooks that run before activate.enter, such as download.exit and install.*, run again on every attempt, for as long as the deferral lasts. Write them so that a second run is safe.

The [daemon] keys hook_retry_max_attempts and hook_retry_max_total_secs are unset by default. Set one to make the device give up: when the cap runs out the poller stays on the running delivery, reports exhausted, and stops retrying that delivery. A cap of 0 gives up on the first defer; to remove a cap, unset the key. Before #6431 these caps defaulted to 10 attempts and 3600 s.

atcd exports the histogram atcd.pull.hook_defer_age (Prometheus: atcd_pull_hook_defer_age_seconds). It records one sample each time a deferral defers again, and the sample is how long that deferral has lasted. Its wait label tells what set the next wait: hint for a retry-after: line, backoff for no line.

Which of these run on an atc ota switch

A switch is a third walk. It re-points the device at a delivery already in the store, so it moves no bytes. atc ota switch <version> walks delivery.pre -> activate.enter -> deactivate.enter -> deactivate.exit -> [flip] -> activate.exit -> commit.enter -> commit.exit -> delivery.post. That walk drops one thing a push does, and it is a gap rather than a carve-out.

No check.*, download.* or install.*. Those three states put bytes on the device. A switch moves none, so it crosses none of them. This is the one walk that still omits them: the pull walk dispatches all six (#3412 §1).

A clean refusal at activate.enter or deactivate.enter leaves the device as it was, as on a push: no rollback.*, no failure.*, and no activate.* for the outgoing version. Any other refusal at activate.enter or after it enters the unwind ladder described above, in the same order push and pull run it. One case still unwinds: a clean deactivate.enter refusal whose teardown record merged into one an earlier switch left standing. That earlier switch still owes the outgoing workload a restart, so this refusal runs the ladder and tries the restart again. If the restart fails, the record stands for the next boot. A stage hook that refuses before the record is written leaves that earlier debt as it was. A switch does not run delivery.post after a refusal.

deactivate.* DOES run, from the outgoing version’s declaration. It did not until #3698. A switch owned no durable record of a teardown. So a daemon that died between the teardown and the flip left the outgoing workload stopped, with nothing able to start it again. The switch now writes that record before the first teardown hook. A boot re-runs the outgoing version’s activate.* from it. All three walks dispatch the same deactivate.* cells. They also dispatch the same rollback.* and failure.* cells: check.*, download.* and install.* stay the switch’s gap, for the reason above.

This changes what a device does. A fleet that switches versions starts running deactivate.* hooks that never ran there before. So does a rollback, because a rollback IS a switch. Read the deactivate.* declarations of the versions already on your fleet before you roll out an atcd carrying this.

A refused teardown VETOES the switch, and the device stays on the version it was already running. The switch un-does nothing, because it had not yet switched. A clean deactivate.enter refusal stops there. A refused deactivate.exit, or a deactivate.enter refusal that is not clean, runs the compensation: the outgoing version’s activate.* is dispatched again, so write both halves to tolerate running twice.

A version whose members declare no activate.* cannot be brought back that way. A teardown that refuses there leaves those workloads stopped until an operator pushes, and the reply says so by name.

A refused switch now runs your rollback.* scripts, and your failure.* scripts when the recovery itself fails. A clean activate.enter or deactivate.enter refusal runs neither. Those hooks ran on a push and on a poll and never on a switch, so a fleet that rolls back starts running a script that has never run there. Read them before you roll out an atcd carrying this. rollback.enter runs before the device is put back, rollback.exit after, and both run twice if you retry a switch that keeps refusing — write them to tolerate that. failure.enter and failure.exit fire only when the outgoing version’s workload is still stopped after the recovery ran, which is the one condition worth paging on.

Stage-scope hooks DO run, at each of the six gates. The scope table says a stage hook runs once per delivery, at that barrier. That holds on a switch as it does on a push and a poll. So a top-level [lifecycle.hook.activate.exit] runs on the switch that makes the delivery live. The switch reaches activate.enter, deactivate.enter, deactivate.exit, activate.exit, commit.enter and commit.exit. A stage hook at rollback or failure runs only if the switch enters that ladder — a refusal at activate.enter or after it, except a clean refusal at activate.enter or deactivate.enter — and the ladder’s cells honour neither --skip-hook nor --hook-env.

The stage arm changes what a device does too. A fleet that switches versions starts running stage hooks that never ran there before. Read your top-level [lifecycle.hook.*] tables as well. A non-zero exit at any of the six gates VETOES the switch. A stage hook that passed at activate.enter or deactivate.enter makes a later member refusal there not clean, so that refusal still unwinds. After the flip a veto un-switches the device back to the version it came from.

Only a delivery committed by an atcd new enough to archive the table has one to read. A delivery committed before that runs no stage hook on a switch until the next push or poll re-commits it.

atc ota switch -a <artifact> is a fourth walk, narrower again. It activates one artifact and not a delivery, so it mints a synthetic single-member delivery. The device must also be in per_artifact assignment mode to accept -a at all.

The -a form runs the per-artifact arm and nothing else. It walks activate.enter -> deactivate.enter -> deactivate.exit -> [flip] -> activate.exit -> commit.enter -> commit.exit. So it runs no delivery.pre, no delivery.post and no stage hook at any gate, at a forward gate or at an unwind cell. It crosses the same six gates the whole-delivery form crosses, and it walks the same unwind ladder when one of them refuses. Read every rule above as the <version> form’s rule. The -a form is the carve-out from the bookends and from the stage arm, and from those two only.

That is a property of the TARGET and not of the walk. The bookends and the stage arm both read the target delivery’s .atc/delivery.pb archive. No push or poll ever committed the synthetic delivery, so it has no archive, and it declares no delivery-scope or stage-scope hook. To read the originating delivery’s archive instead would run one delivery’s hooks over another delivery’s activation. The whole-delivery form runs both arms because its target IS a committed delivery.

deactivate.* is the arm the two forms share, and it did not run on -a until #4211. It reads the OUTGOING version’s own .atc/lifecycle.pb sidecar, which every version carries, so it is a property of the version being replaced and not of the synthetic target. Everything the deactivate.* note above says holds here: a refused teardown vetoes the activation, the device stays on the version it was already running, and a refusal that is not a clean deactivate.enter refusal dispatches the outgoing version’s activate.* again first. A daemon that dies between the teardown and the flip is covered by the same durable record. That record names the synthetic delivery as its target, so a boot that finds the pointer already on it reads the walk as finished and clears the record.

This changes what a device does. A fleet that runs atc ota switch -a on a per_artifact device starts running deactivate.* hooks that never ran on that form before. Read the deactivate.* declarations of the versions already on your fleet before you roll out an atcd carrying this.

The unwind ladder brackets that compensation. #4257 is where it started to run on -a. A refused -a activation dispatches rollback.enter, then the compensation that brings the outgoing version back, then rollback.exit. If a rollback hook itself fails, it also dispatches failure.enter and failure.exit. A take-back that could not restore the device does the same.

These cells read the TARGET version’s own sidecar, which every version carries. So they run at the per-artifact arm, like every other cell of this walk. A hook there cannot veto anything. The unwind track IS the failure path. A non-zero exit is recorded, and the ladder walks on.

One ordering differs from atc ota switch <version>. On that form rollback.enter runs while the pointer still names the target. On -a the pointer is already back, because putting it back is what tells this walk’s background task the activation was refused. Nothing may be awaited before that. So an @name/... cross-ref in a -a rollback.enter hook resolves against the outgoing version.

A refused atc ota switch -a now runs your rollback.* scripts. It runs your failure.* scripts too, when the take-back did not restore the device. These are the same cells a push and a poll already dispatch. So a fleet that declares them for either of those planes starts running them on this form too. Both cells bracket the workload restart, and the pointer is back before either of them runs.

Which hooks run after atcd crashes during a push

If atcd dies during a push, for example on a power loss, the next start of atcd finishes the delivery. What it does depends on where the push stopped. Here n is the version that was active before the push, and v is the pushed version.

Every hook in the table reads daemon_restart_recovery in ATC_HOOK_CONTEXT.trigger. That is how a hook, and an operator reading atc ota status, tells a crash from a hook that refused: a refusal’s rollback reads auto_rollback_veto. Write an activate.* hook so that it can run again on a version that is already active, because these re-runs do exactly that.

A crash after the pointer flip is treated as an activate.exit failure. atcd puts current back on n, then runs these hooks in this order:

1. v’s rollback.enter.
2. n’s activate.enter and activate.exit, if n’s deactivate.* ran during the push. If activate.enter refuses, activate.exit does not run. The refusal rule is the one above: every stopped member ran an activate.enter hook, and each one exited non-zero.
3. v’s rollback.exit.
4. If a hook in steps 1 to 3 exited non-zero, v’s failure.enter and failure.exit.

If every hook passes, the delivery is rolled_back. If one does not, the delivery is failed. atcd runs this rollback at most once. If atcd dies again during the rollback, the next start marks the delivery failed and runs no hook. atcd never pushes v again by itself, because an activate.exit that crashes the device would then crash it on every start. To try v again, push it again.

The delivery-wide delivery.post hook, stage-scope hooks and the workload restart step do not run during this rollback.

A pulled delivery that a hook refuses after the flip gets the same rollback from the next start if atcd dies after the un-flip and before n’s compensating activate.* ran. If atcd dies during that compensation, the next start marks the delivery failed and runs no hook.

A live push whose activate.exit refuses ends with the same hooks and the same status (#6750).

After a crash before the flip, n’s hooks set the delivery status:

• If every member that went down runs an activate.* hook and no hook exits non-zero, the delivery is rolled_back.
• If a hook exits non-zero, or atcd cannot dispatch the hooks, the delivery stays in progress and atcd tries again on its next start. v’s failure.enter and failure.exit run, if v declares them.
• If a member that n’s deactivate.* stopped declares no activate.*, atcd cannot restart it. The delivery is failed, and v’s failure.* run, if v declares them.

While atcd runs these hooks, it refuses a new push or pull into that channel.

Which state’s hooks come from which version

Almost every hook runs the declaration of the delivery arriving. Two do not.

So a deactivate.* hook you add today does not run on today’s delivery. It runs on the next one, when the version you ship now becomes the outgoing version. An atc ota switch that leaves that version reaches it too. A first deploy traverses deactivate and dispatches nothing, because there is no outgoing version to tear down. So does a delivery that flips nothing: a stage-only push, a re-push of the version already active, or a pulled delivery re-cut over unchanged bytes. The rule is the same on both planes.

A push that flips nothing also runs no activate.*, because it activates nothing. That covers a stage-only push and a re-push of the active version. A pulled re-cut still runs activate.*, because it moves current to the new delivery. An atcd older than v0.1.5 did run activate.* there; see Behavior changed.

A pulled re-commit of the delivery already active runs no activate.*, because it moves nothing. A revision that only adds present-only deliveries makes that re-commit. So does the first poll after an atcd restart, when the last walk finished. A restart that resumes an unfinished walk still runs activate.exit.

An atc ota switch to the delivery already active runs no activate.* and no deactivate.*. It moves nothing, so it activates nothing. commit.* and the delivery bookends still run. An atc ota switch -a that repeats the per-artifact activation already current also runs no activate.* and no deactivate.*. An -a of a version that is live under a whole delivery still runs activate.*, because the pointer moves to a new delivery.

Scope

The Stage row holds on all three walks. An atc ota switch <version> runs the stage arm at each of the six gates it crosses, and crosses no other. The atc ota switch -a <artifact> form runs no stage arm at all. See Which of these run on an atc ota switch.

check.*, download.*, install.* and uninstall.* have no stage scope. The first three are artifact-major. The daemon crosses one gate per artifact, not one per delivery, so a stage table there would parse and never run. Push preflight rejects all four rather than accept one silently.

Declare them per artifact instead. That is where they run on a push and on a poll alike, once each, in push-list order. A pull walk reached none of the three until #3644; it dispatches all six cells now (atcd_pull::pull_poller::walk_acquire), and the rest of this page says so at “Which of these run on an atc ota switch”. The atc ota switch walk is the one that still reaches none of them, because it moves no bytes.

The refusal above is about SCOPE and not about a plane. A check, download or install cell declared at STAGE scope is rejected at preflight on every plane, and the device warns about each such cell once per delivery. It still runs it zero times.

uninstall.* is a separate case. It is dispatched per artifact of each version being removed, read from that version’s own sidecar. There is no per-version stage table for a delivery-wide cell to read. Put cleanup for one version in an artifact-scope uninstall.* hook; see Per-version cleanup: uninstall.*. gc.* brackets a whole sweep and names no version.

So the stage-scope states are activate, deactivate, commit, rollback and failure.

The daemon runs the same check over a signed manifest it pulls. Any invalid prong refuses the whole delivery. So a delivery published with one of these tables before the rule tightened stops activating on a pulling device. It used to activate with the hook silently skipped. Move the table under the artifact and publish again. The device cannot be repaired locally.

That move buys a pulling fleet what it buys a pushing one. The delivery activates again, and the hook runs. The pull walk reaches all three states at artifact scope (#3412 §1). Neither plane has a delivery-wide barrier at any of them, which is why the top-level table is refused rather than supported.

A carve-out worth knowing about. A stage-scope deactivate.* is legal and it runs, but it reads the incoming manifest, not the outgoing version’s. Stage hooks are delivery-wide and bound to no artifact, while the outgoing world is per-version, so there is no delivery-wide outgoing table to read from. If you want a teardown veto over the version already running, declare it at artifact scope.

Declare it at artifact scope for a second reason. The durable teardown record above covers the artifact-scope teardown only. A stage-scope deactivate.* can stop a workload, and a crash after it leaves nothing on the device that says so. That is true on all three planes.

Per-version cleanup: uninstall.*

Some work must follow one version off the device. Example: a hook loaded an OCI image into podman storage, and the tag must go when atc deletes that version. Declare that cleanup as an artifact-scope uninstall.* hook, not as gc.*:

[[bundle]]
name = "myapp"
source = "dist"           # ships scripts/release_image.sh

[[oci]]
name = "myapp-image"
source = "registry.example.com/myapp:1.4"

[oci.lifecycle.hook.uninstall.exit]
script = "@myapp/scripts/release_image.sh"
timeout_secs = 60

Both removal paths run uninstall.*: the housekeeping sweep and atc ota rm. gc.* runs on the sweep only, and it names no version.

What the hook receives. atc runs the gate once for each version it removes, before it deletes the version directory. On atc ota rm the delivery is already gone at that point, so the gate holds back the version and nothing else. ATC_HOOK_CONTEXT holds:

The hook runs in the directory of the version being removed. That is true for a @self script and for a @<peer> cross-ref.

A non-zero exit keeps the version. A non-zero exit, a timeout, or a script that cannot start withholds the removal. exit 21 is a refusal here too, not a defer. The housekeeping sweep asks again while the artifact has an active version. After that, the next atc ota rm or purge in the channel asks. So a cleanup hook that fails on every run keeps its version on disk for ever.

If the cleanup is best effort, report the failure and end the script with exit 0:

#!/bin/sh
# uninstall.exit on myapp-image. Release the podman tag of the version that
# atc removes. ctx() is the helper from "How hooks run".
sha=$(ctx version.new_sha) || exit 0
podman rmi "localhost/myapp-image:$sha" > /dev/null 2>&1 ||
  printf '%s\n' "release_image.sh: no tag for $sha" >&2
exit 0

The tag must be the one your load hook wrote. This example assumes the image’s install.exit ran podman tag <image-id> "localhost/myapp-image:$sha" with the same version.new_sha. A different tag makes podman rmi miss, and the script still exits 0.

Ship both artifacts in one push. An OCI version holds an image layout and none of your scripts, so its hook names a script that a bundle ships. A @myapp/... ref to a bundle in the same push is accepted. An uninstall.* hook runs at GC, after the push, so it puts no order on the push. See “Same-push dependencies” under “Script references”.

At uninstall.* and gc.*, @<peer> finds the ACTIVE version, not a copy kept with the image. No push is in flight when GC runs. So @myapp resolves to the myapp version that is active in the channel when the hook runs. The delivery pointer delivery/current/myapp names the same version. For an uninstall.* hook, two cases leave no script to run, and each one keeps the image version:

• The active myapp version does not ship the script. An older version that you switched back to is the usual cause.
• myapp is not in the active delivery, for example because you removed it first.

So ship the script at one fixed path in every bundle version, and remove the image versions before the bundle. To remove that dependency too, install the script at a fixed path outside the store, such as /usr/local/libexec/myapp/release_image.sh. atc does not manage or roll back that file.

gc.enter / gc.exit

The housekeeping sweep’s bracket. Declared at the top level under [lifecycle.hook.gc.*], because that is where you expect to write a hook, and carried in .atc/delivery.pb alongside the [delivery.hook.*] bookends, because a sweep bracket has a bookend’s shape and lifetime rather than a state’s. An artifact-scoped [bundle.lifecycle.hook.gc.*] is rejected at push.

[lifecycle.hook.gc.enter]
script = "@ops/scripts/pause_reclaim.sh"   # @self is forbidden here
timeout_secs = 30

gc.enter is your opt-out. A non-zero exit skips that sweep’s delivery GC and per-artifact version GC, and nothing else. The partial-transfer, orphan-staging, stream-cache and pull-partial reapers always run, whatever gc.enter says. Those reclaim disk to stop the device filling up, and a wedged operator hook that could block them would be a worse failure than the one the opt-out prevents.

gc.exit is reporting only. The sweep is over by the time it runs, so its exit code gates nothing.

Each channel runs its own gc.* bookend. The sweep is device-wide, but the declaration belongs to a delivery. So the sweep runs the bookend once for each channel whose active delivery declares it. On a device with one channel, that is once per sweep. On a device with agency/atcd beside your own channel, a hook declared in both deliveries runs twice in one sweep. At gc.enter, a non-zero exit from any one of them skips the delivery GC and version GC for the whole sweep.

A @<peer>/... script in gc.* resolves to the peer’s ACTIVE version when the sweep runs, the same rule as uninstall.*.

atc ota rm runs no gc.* hook. The pair brackets the housekeeping sweep only. rm is an operator command, not a sweep. It runs uninstall.enter and uninstall.exit for each version it removes, and it skips both gc.* gates. So a gc.enter opt-out does not stop an rm.

gc.exit does not run after the rm. It runs after the uninstall pass of a later sweep, and only on a sweep whose gc.enter let that pass run. For cleanup that must run when one version goes, use uninstall.*. That gate runs on both removal paths.

uninstall.* and gc.* both write a lifecycle_log row. A dispatched uninstall.enter or uninstall.exit is journalled like any other hook. A dispatched gc.* bookend is journalled under the id of the active delivery that declares it, with scope and trigger both gc and the uninstall cell it brackets. A refusal at any of the four also goes to the daemon log and the atc.housekeeping.uninstall_veto span. An uninstall.* refusal reads uninstall.<phase> refused for …; withholding its removal this sweep.

What a gc.* hook reads. A sweep is not a delivery, so the bookend context names no artifact and no version. ATC_HOOK_CONTEXT holds:

• push_id: the id of the active delivery whose [lifecycle.hook.gc.*] declaration runs. A sweep has no session of its own, so this can name a delivery pushed long before the sweep.
• state uninstall, phase enter or exit, and trigger gc.
• null for artifact, version, self_root.new, self_root.old and current_root.

An uninstall.* hook also reads state uninstall and trigger gc. Its artifact is not null, which is how a shared script tells the two apart. ATC_DELIVERY_CONTEXT carries hook_scope gc as well.

[lifecycle.hook.gc.error] and uninstall.error are both rejected at parse, as is every other .error cell: enter and exit are the only gates anything dispatches, so a .error cell would parse, push green, and never run. You get an error naming the two phases that do dispatch, rather than a hook that silently never fires.

Progress hooks: reporting DURING a state

Every hook above fires at a barrier. [lifecycle.progress.<state>] fires between two, on a cadence you set, while the state’s work runs:

[lifecycle.progress.download]
script = "@ops/scripts/report-progress.sh"
interval_secs = 5

The script gets two extra environment variables on top of the usual hook environment: ATC_PROGRESS_PCT, the percentage at that tick, and ATC_HOOK_POINT, which reads download.progress.

Read ATC_HOOK_POINT, not ATC_HOOK_CONTEXT.phase. progress is not a phase, so the context document carries the placeholder "phase":"enter", the same placeholder a delivery bookend carries. No gate waits on a progress hook.

Rules, all enforced at atc ota push:

It runs on push and on pull. atc ota push and a polled delivery both tick at the declared interval while a member transfers. atc ota push refuses to send the table to a device whose atcd predates the feature.

The two planes differ in two places:

• The exit code. On pull, a non-zero exit cancels the delivery. The member in flight stops, later members are not staged, and nothing activates. A timeout, a signal or a spawn failure does not cancel. On push, the exit code is logged and nothing acts on it.
• The first tick. Pull waits one full interval after a member’s transfer starts. Push arms its timer when the push starts, so the first tick in a transfer lands somewhere inside the first interval. A push tick that comes before the daemon knows the member’s total size runs no script, so the first run can come one interval later.

What it does NOT do, today:

• It runs once per member. The task brackets one artifact’s transfer. A three-bundle delivery runs the hook three times over, not once across the delivery.
• A slow script does not queue. A tick that arrives during a live run is skipped. Nothing is killed for being slow.
• It does not run a script in a member that is not installed yet. A tick resolves @<name>/<path> against the channel’s installed <name>. On the channel’s first delivery of <name>, that tree does not exist until the delivery commits, which is after every download. So atcd skips the hook and logs the skip, once per push or once per pulled member. For a first delivery, use an absolute path or a member that the channel already holds. atc ota push accepts the script, because later deliveries run it.

Script references

@self is forbidden in top-level (stage-scope) [lifecycle.hook.*] tables, because a stage hook runs once across all artifacts and isn’t bound to one tree. Use an artifact-scoped table ([bundle.lifecycle.*] / [oci.lifecycle.*]) for @self.

A stage hook’s @<name> must name an artifact in the same push or an artifact that is active in the channel. If it names neither, atc ota push fails at preflight, before any blob moves:

stage hook `commit.exit`: script `@sidecar/…` names no artifact in this push or active in channel `local/default`

Which version’s script @self runs

A version’s own scripts run only after that version is installed. Before that, an artifact hook’s @self/<path> belongs to the version that is ACTIVE on the device. This is the version that the delivery replaces. A rollback and a failure report belong to the version that the unwind restores, which is that same version.

At those gates the hook is the active version’s own hook. Its table entry comes from the active version’s atc.toml: whether the hook exists, its timeout_secs and its <path>. Its script comes from the active version’s tree.

An absolute-path hook and a @<name>/<path> hook name no version tree of the artifact. At check.*, download.*, install.enter, rollback.* and failure.*, they keep the INCOMING atc.toml. At deactivate.* and uninstall.*, they use the outgoing version’s declaration, as a @self hook does.

What follows from the rule:

• A guard that a release adds. A @self guard that release N+1 adds at check.*, download.* or install.enter runs first at the upgrade AFTER N+1, from the copy that N+1 installed. It does not run on N+1’s own delivery.
• A release that needs its own prerequisite. Declare it as an absolute-path or @<name> hook. It runs on that release’s delivery, the first delivery included.
• A guard that a release removes or changes. It still runs on that release’s delivery, with the active version’s script, path and timeout.
• Two hooks at one gate. The active version can declare a @self hook at a gate where the release declares an absolute or @<name> hook for the same artifact. Then both run: the active @self hook first, then the release’s hook. If either one refuses, the delivery is refused. A release cannot switch off the guard of the version that it replaces, and it keeps its own prerequisite.
• A first delivery. The device has no active version, so no @self hook runs at these gates and none writes a journal row. An absolute or @<name> hook still runs.
• An unwind with no version to restore. A first delivery that is refused after the switch runs no @self hook at rollback.* or failure.*.
• A version installed before atcd kept declarations. If the active version’s declaration cannot be read, no @self hook runs at these gates. The push journals deactivate_config_unavailable for that member.
• A missing script. The active version’s own @self script must be in its own tree. If the tree does not hold it, the hook fails, as at every other gate. It vetoes at check.*, download.* and install.enter. It marks the unwind dirty at rollback.* and failure.*.

[[bundle]]
name = "agent"
source = "dist"

# Runs the ACTIVE version's guard before any byte of the new version moves.
# A release that adds this table runs it first at the next upgrade.
[bundle.lifecycle.hook.download.enter]
script = "@self/hooks/pre-download.sh"

# A prerequisite of THIS release. It runs on this release's own delivery.
[bundle.lifecycle.hook.check.exit]
script = "/usr/local/libexec/agent/battery-ok.sh"

A pulled delivery that declares no hooks still runs the active delivery’s hooks: its @self guards at these gates and its deactivate.* hooks.

atc ota run-hook is not a delivery, and this table does not apply to it. It runs the hook that the ACTIVE version declares, from that version’s own tree, at every gate. It cannot run the guard of a version that is not active yet. Without -o, if no artifact declares the cell, it runs the active delivery’s stage hook. With -o <name>, it never runs a stage hook.

The hook runs in the active version’s tree, and ATC_HOOK_CONTEXT says so. At these gates self_root.old is the active (or restored) version’s tree, which holds the script and is the working directory. self_root.new names the incoming tree only if that tree is already on the device, and the hook must not run scripts from it. version.old_sha is the active version, and version.new_sha is the incoming one. On a first delivery the hook does not run.

A device whose atcd does not advertise active-hook-table-v1 reads the INCOMING atc.toml for a @self hook at these gates. It skips a script that the active tree does not hold.

Requires atc and atcd from nightly-2026-10-08 or later. No stable release has this rule yet: 0.1.5 does not. A stable atc refuses @self at download.enter itself, before it connects to the device. It prints self-dependency at download_enter is not allowed (circular), then ATC-E0401: atc.toml is malformed.

On a stable release, move that hook to install.exit, activate.* or commit.*, where @self is the incoming version’s tree in every release. Or give it an absolute script path that is already on the device before the first push. allow_missing_initial is also newer than 0.1.5, so a stable release cannot skip a missing script.

On the daemon side, the rule needs the self-active-tree-v1 capability. An older atcd refuses @self at download.enter and fails a @self hook at check.*. At download.exit, install.enter and rollback.*, it runs the incoming script. A @self hook at failure.* also requires the self-failure-active-tree-v1 capability. An atcd that has only self-active-tree-v1 runs the incoming script at failure.*.

So atc ota push refuses to send such a declaration to a daemon that does not advertise the capability, and names the capability and the gates. A pulled delivery has no handshake. Upgrade every device on a channel before you publish such a declaration to it.

Same-push dependencies

One push can carry an artifact and the artifacts it depends on. A dependency is a requires entry or an @<name>/... hook script. atc orders the push so that each dependency that an order can satisfy resolves. atcd refuses the two that no order satisfies: a cycle, and a check.* hook that names another artifact of the push.

Which version @<name> names. When <name> is in the same push, an artifact hook’s @<name> resolves to the version of <name> that this push installs, at check.* through commit.*. That is true before and after the flip, on a first deploy and on an upgrade. Other hooks resolve it to another version, the same release @self names there:

• deactivate.*, and the activate.* that a rollback runs again, run the OUTGOING version’s declaration. @<name> names the version active before the push.
• rollback.* and failure.* restore the earlier version. @<name> names the version of <name> they restore, or none on a first deploy. The incoming <name> may never have been placed: a download.exit veto unwinds before the install.
• uninstall.* and gc.* run at GC, after the push. @<name> names the version active when the hook runs.
• Delivery bookends, stage hooks and progress hooks resolve through the delivery pointer, as before.

An @<name> or a requires entry that names the artifact itself is not a same-push dependency. @<name> keeps resolving to its active version at check.* through commit.*, as before, so the artifact must be active, or atcd refuses with missing dependency. Use @self for the tree this push installs.

When <name> is not in the push, @<name> resolves to its active version, and <name> must be active, or atcd refuses with missing dependency: <name>.

The order. A push downloads its artifacts one at a time, in the order it sends them. Every state after install.* runs once all of them are installed. So atc ota push sends each artifact after the same-push artifacts that it requires, and after the artifacts that its download.* hooks name. Other artifacts keep the order of atc.toml.

An install.* hook runs after every download, so its @<name> resolves in any order. So does an @<name> at a later cell, for example activate.enter or uninstall.exit.

A push cannot order a check.* hook. A push runs check.* for every artifact before it downloads any. So a check.* hook that names another artifact of the same push finds nothing, in any order, and atcd refuses it. Move the hook to download.* or install.*, or push the other artifact first.

A pulled delivery stages in the same order. A device that pulls a delivery takes each artifact through check.*, download.* and install.* before the next, and stages an artifact after the artifacts it depends on. So on a pull, a check.* hook can name another artifact of the same delivery.

Both planes refuse a cycle, because no order can install a after b and b after a. A push and a pull report the same message. A device refuses a pulled cycle before it downloads anything, stays on its current release, and reports the rollout failed. An atcd from before this check can have installed such a delivery, or started to and parked it on a hook retry. The device keeps the first and can finish the second. The message:

same-push dependency cycle: a -> b -> a. No order of the push installs each artifact after the ones it depends on. Remove one dependency.

Older versions. An atc that predates the ordering sends artifacts in atc.toml order. atcd cannot reorder a push, so it refuses one that lists an artifact before its dependency: artifact "a" is listed before "b", which it depends on. Upgrade atc. An atcd that predates the ordering refuses every same-push dependency with depends on b which is in the same push — split into separate pushes. Upgrade atcd, or split the push as it says.

How hooks run

Write hooks so a second run is harmless — including a second run that OVERLAPS the first. atcd will not re-run a hook for a delivery it knows it already started, but it can lose that knowledge: if the daemon restarts partway through an activation, a retry starts the delivery again from the top. The same applies if atc ota switch reconnects after a drop early enough that it has not yet seen any progress from the daemon.

The first instance may still be running. Hooks are spawned into their own process group (process_group(0)), so they do not receive signals sent to atcd. atcd kills that group when a hook exceeds its own timeout_secs, but a daemon that is SIGKILLed, OOM- killed, or panics runs no cleanup at all — the hook is reparented and keeps going. A retry can therefore put two instances of the same script on the device at once.

Requires a daemon whose unit grants CAP_KILL. With hook_run_as_user set (the default), the hook runs as a different uid than atcd, and POSIX allows kill(2) only from root or a matching uid. A unit rendered before that capability was granted therefore takes EPERM on both SIGTERM and SIGKILL: timeout_secs still fires on time, and the hook still runs to completion while being reported as a timeout. Re-bootstrap (atc bootstrap atcd install) to re-render the unit; systemctl show atcd -p AmbientCapabilities should list cap_kill. See #3242.

Sequential safety is not enough for that. Do both:

• Idempotent steps, so a re-run is a no-op: mkdir -p not mkdir; write to a temp path and rename(2) over the target rather than appending; check whether a unit is already enabled before enabling it; make “already done” an exit-0 path, not an error.

• Mutual exclusion, so two instances cannot interleave: take an flock on a well-known path for the duration of the hook — blocking with your own bound, e.g. flock -w 60 /run/atc/myhook.lock -c '…'. Wait for the lock, then do the work (your idempotent steps make the second pass cheap) and report your own real outcome; if the wait expires, exit non-zero, because at that point you genuinely do not know whether the other instance is progressing or wedged.

  Bound the lock wait yourself — do not rely on timeout_secs. A hook declaring timeout_secs = 0 runs unbounded (that is the documented rule), so a plain blocking flock behind an orphan that never exits stalls the deployment forever instead of failing it. -w is what makes the failure mode legible on every configuration.

  Do not flock -n and exit 0 on losing the race. Exit 0 tells atcd the hook succeeded and the lifecycle advances — but the instance holding the lock is a leftover from a dead daemon that may still fail or leave its work half-finished. You would be reporting completion for work you neither did nor observed. Exiting non-zero is no better: it fails a deployment that may well be fine. Blocking is the only option whose exit code describes something the hook actually knows.

• Prefer single atomic operations over read-modify-write sequences; two concurrent read-modify-writes lose an update no matter how idempotent each looks alone.

A hook that only works on a clean, solitary first run will eventually fail a deployment on a device that rebooted at the wrong moment.

Background and the design decision behind this: docs/internal-site/src/design/2026-07-27-switch-wait-and-resume-design.md (“Accepted limit: one window can still run hooks twice”).

atcd exec’s the script directly — there is no sh -c wrapper. The file needs the executable bit and a shebang; a script without +x is rejected on the device just before it runs (not executable: mode=…). Neither is pre-checked when you author the manifest or at push preflight — a bad script only surfaces once its hook fires during a delivery — so chmod +x and check head -1 for the shebang in source before pushing. An absolute script that does not exist fails the hook with ENOENT, unless the hook sets allow_missing or allow_missing_initial (A hook script that the artifact installs). Those keys never excuse a script that exists without +x.

For an ARTIFACT-scope hook, the working directory is the tree that the @self script came from, if that directory exists. For most cells this is the new version. At check.*, download.* and install.enter it is the ACTIVE version, and at rollback.* it is the restored version, which is the same one (Which version’s script @self runs). For deactivate.*, for uninstall.*, and for the ladder’s compensating activate.*, it is the OUTGOING version. Those cells read their declaration from that version too. So a hook that ships data beside itself always gets the copy of its own version, even through a relative path.

Two shapes have no declaring version, and they keep the old rule. They run in the artifact’s new version directory, or in the script’s own directory when there is none. A @<peer>/... cross-ref names a script in another artifact’s tree. An absolute path names a script the operator owns.

A STAGE-scope or DELIVERY-scope hook is bound to no artifact. @self is refused there, so such a hook has no version tree at all. Its working directory is the script’s own directory.

The hook does not inherit atcd’s environment. The env is cleared, PATH is pinned to /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, and atcd exports the whole delivery context as a single JSON variable, ATC_HOOK_CONTEXT (see ATC_HOOK_CONTEXT below), plus the hook user’s login environment (HOME/USER/LOGNAME from the passwd entry; SHELL is the resolved bash on PATH, not the passwd login shell, which stays nologin; plus XDG_RUNTIME_DIR, DBUS_SESSION_BUS_ADDRESS). Parse the context with awk:

set -eu

# Read one dotted field out of ATC_HOOK_CONTEXT. A second argument gives the
# value to use when the field, or an object above it, is null. With no second
# argument, a field under a null object prints "". An unknown key or a null
# value exits nonzero, so a typo stops the hook instead of giving "".
ctx() {
  awk -v path="$1" -v nargs="$#" -v dflt="${2-}" '
    function fail(why) { print "ATC_HOOK_CONTEXT." path ": " why > "/dev/stderr"; exit 1 }
    function find(t, k,   j, c, d, q) {
      k = "\"" k "\":"
      for (j = 1; j <= length(t); j++) {
        c = substr(t, j, 1)
        if (q) { if (c == "\\") j++; else if (c == "\"") q = 0 }
        else if (c == "{") d++
        else if (c == "}") { if (--d == 0) return 0 }
        else if (c == "\"") { if (d == 1 && substr(t, j, length(k)) == k) return j + length(k); q = 1 }
      }
      return 0
    }
    BEGIN {
      s = ENVIRON["ATC_HOOK_CONTEXT"]; n = split(path, key, ".")
      for (i = 1; i <= n; i++) {
        if (substr(s, 1, 4) == "null") { print dflt; exit 0 }
        if (substr(s, 1, 1) != "{") fail("not an object")
        p = find(s, key[i])
        if (p == 0) fail("no such field")
        s = substr(s, p)
      }
      if (substr(s, 1, 4) == "null") { if (nargs < 2) fail("null"); print dflt; exit 0 }
      if (substr(s, 1, 1) != "\"") fail("not a string")
      for (j = 2; j <= length(s); j++) {
        c = substr(s, j, 1)
        if (c == "\"") { print v; exit 0 }
        if (c == "\\") { j++; c = substr(s, j, 1); if (c != "\"" && c != "\\" && c != "/") fail("holds a control escape") }
        v = v c
      }
      fail("not a string")
    }'
}

sha=$(ctx version.new_sha)
state=$(ctx state)
name=$(ctx artifact.name)
old=$(ctx version.old_sha "")   # null on a first install

artifact.* and version.* are empty in a hook bound to no artifact. A stage hook, a delivery.* bookend and a gc.* bookend get null for artifact and version. The helper stops at that null, prints "" or its second argument, and exits 0. A per-artifact hook always has a non-empty artifact.name, so [ -z "$(ctx artifact.name)" ] is true only in a hook bound to no artifact.

A ctx copied from an older version of this page exits 1 with not an object on those reads. Under set -eu, a delivery.pre that does so vetoes the delivery, and a gc.enter skips the sweep. Paste the helper above in its place.

A stage hook and a gc.* bookend can read push_id, state, phase and trigger. A delivery.* bookend gets placeholders in state and phase (commit, enter), so read its real phase from ATC_DELIVERY_CONTEXT. Its trigger is real, and it matches ATC_DELIVERY_CONTEXT: pull on a pulled delivery, user_activate on a switch. Before #7223 it read push on every plane.

Use awk, not jq or python3. The meta-atc device image carries neither of them. atcd pins the hook PATH to the standard system directories, so the hook sees only what the image already installed. awk is on every base image, busybox included.

The helper handles the fields atcd writes: strings, nulls and nested objects. It decodes \", \\ and \/. It refuses any other escape, such as a control character, and names the field. A hook that reaches for a missing reader gets an EMPTY value, not an error. It writes a broken file while the push reports success. set -eu plus the null check above turn that silent case into a failed state.

There are no discrete ATC_PUSH_ID / ATC_NEW_VERSION_SHA / ATC_STATE / ATC_PHASE / … variables — those were removed. Every field lives inside ATC_HOOK_CONTEXT. (There is likewise no ATC_VERSION; read .version.new_sha.)

Hook user: hooks inherit atcd’s own uid unless hook isolation is configured. On the bootstrap/systemd path it always is: bootstrap creates the atc-hooks user and bakes --hook-run-as-user into the unit (from [daemon] hook_run_as_user, default atc-hooks) — a separate user in the daemon’s group that can read the 0750 store but cannot corrupt it, and drives its own systemctl --user without sudo. A manually started atcd (the no-systemd foreground recipe) has no hook user configured — no atc-hooks user exists on such a device — so hooks run as whatever user launched the daemon (your SSH login user). Neither --hook-run-as-user <user> nor [daemon] hook_run_as_user can isolate them on that recipe: switching the hook uid requires a root (or CAP_SETUID/SETGID) daemon, which the un-sudo’d foreground recipe deliberately avoids — set neither there (see the no-systemd section of deploy-your-first-update.md). A hook user that resolves to the daemon’s own uid is rejected at startup, never silently accepted.

A hook that must read a root-owned file, a K3S kubeconfig for example, reaches it through a supplementary group on atc-hooks. Do not reach for hook_run_as_user = "root". atcd resolves that group set at every spawn, so the grant needs no ATC change and no daemon restart. The recipe and its cost are in the atcd non-root runtime model.

That is why writing hook_run_as_user into the file is what turns isolation on for a hand-launched daemon, and leaving it out is not (#3366). The key’s DEFAULT reaches only atc bootstrap atcd install, which creates the account and grants the unit the capabilities before it bakes the flag. A hand-launched atcd run --config <file> applies the key when the file states it, and keeps hooks on its own uid when the file does not — so a config written for a bootstrap install does not turn a foreground daemon into one whose every hook dies in its pre-exec. --hook-run-as-user still wins over the file, and atcd logs a warning when it overrides a different stated value.

[daemon] socket_mode reads the same way: --socket-mode wins, then the file, then 0660. So does [daemon] socket_owner: --socket-owner wins, then the file, then no grant at all.

Socket owner: the daemon’s own user owns the control socket. The controller opens it by SSH-exec’ing atcd attach as your SSH LOGIN user, so those two accounts must line up. Under the default install they do. The installer adds the login user to the daemon’s group, and the group bits of the 0660 socket then reach it.

Under run_as_user = "root" there is no such group. An operator may not join the root group, so the login user takes ATC-E0420 on every command while the daemon stays healthy.

[daemon] socket_owner closes that gap. atcd chowns the socket to the named account right after bind, and grants exactly that one uid. atc bootstrap atcd install --root <user>@<host> fills the key in from the target’s own user@. The same install also puts the store root (/var/atc, mode 0710) in that account’s primary group, so the account can traverse to the atcd binary the remote_path symlink resolves into; without that grant every atc ota verb fails with ATC-E0470. State it yourself when your target names no user, or when the account that runs atc is not the account the daemon expects.

Do NOT reach for socket_mode = "0666" instead. That mode hands an admin control socket to every local process on the device.

atcd REFUSES to start when the named account does not resolve, or when the chown fails. A grant that quietly did not happen leaves a socket nobody can open, behind a systemctl status atcd that still reads green.

On timeout_secs expiry the hook’s whole process group gets SIGTERM, then SIGKILL after a 2-second grace.

A hook that calls setsid() leaves that process group, so no signal reaches it. atcd therefore also puts each hook in its own cgroup (v2) and kills that cgroup on the same ladder. A process cannot leave its cgroup, and every descendant inherits it. The cgroup is the backstop. The process group stays the first move.

That backstop needs a cgroup subtree atcd may write to. The daemon probes for one at startup and says which mechanism is in force:

lifecycle hook bound: cgroup v2 at /sys/fs/cgroup/system.slice/atcd.service/atc-hooks …
lifecycle hook bound: process group only (…)

The second line is not an error. atcd runs in containers and under atcd run with no systemd, and a hook still runs and is still bounded by its process group there. It says only that a hook which daemonizes can outlive its timeout_secs. To get the backstop on a device installed before #3562, re-bootstrap (atc bootstrap atcd install). The regenerated unit carries Delegate=. Do not add the directive by hand: bootstrap rewrites the unit whole, so a hand edit is erased by the same command that would apply it. Where there is no systemd, point ATC_HOOK_CGROUP_ROOT at a writable cgroup-v2 directory instead. The same answer reads across a fleet as the atc.lifecycle.hook_cgroup_delegation metric. See #3321.

With timeout_secs = 0 there is no expiry: the hook runs until it exits or the delivery preempts it, so a wedged hook wedges the delivery. Use it only for work whose duration you genuinely cannot bound (a long flash, a large migration).

File modes: the packer records each source file’s mode, but the device materializes every bundle file read-only (0o440), adding u+x,g+x only when the source file had any execute bit. So only the exec bit survives the trip. Run chmod +x on your hook script in source before pushing. Write, “other”, and setuid/setgid bits are always stripped.

Directory modes are not part of that guarantee. The directories that hold delivered files keep the daemon user’s own mode, so a manual write as that user can still succeed. Do not predict which way a write goes. hook_run_as_user picks the account (see Hook user above), and the directory’s mode comes from atcd’s umask when the store was built. A permissive umask leaves the directory group-writable, and atc-hooks is in the store group, so even a non-owning hook can write. Neither input is a guarantee, so only the file modes are.

Keep writable state in a directory that the bundle declares with state_dirs. See Writable state: state_dirs below.

A write beside the binary also does not follow the delivery pointer. Each version lands in a directory of its own, so the file stays with the version that was current when it was written. Retention prunes it later, and a rollback to that version brings it back.

Phase names in the journal: each state’s journal labels are the six internal sub-phases (pre_hook → hook_done → pre_side_effect → side_effect_done → pre_hook_exit → hook_exit_done). An enter hook runs at <state>/pre_hook; an exit hook at <state>/pre_hook_exit. The push output names the hook as you wrote it, so a commit.exit hook reads commit.exit there. The pre_ means “about to run this hook”, not “before the state”.

Only enter and exit gate hook dispatch, so those are the only two that surface as a pre_hook* phase in the journal. Match on the hook line, ✓ commit.exit, not on the journal label.

<state>.error is not a declaration point at any state. Nothing dispatches an error gate — not for commit, not for any other state — so a .error cell would parse, push green, and never run. atc ota push rejects it and names the two phases that do dispatch, enter and exit.

It does not name a replacement cell, because which states you may use depends on the scope you declared at. deactivate.exit is valid under [bundle.lifecycle.*] and refused under [[peripheral]]. Pick enter or exit on a state your scope supports.

At artifact and stage scope that is usually <state>.exit, or the unwind hooks rollback.* and failure.*, which run after a delivery fails. Under [[peripheral]] those are not all available — failure.* and rollback.exit are refused there too, and rollback.enter is the peripheral’s one unwind hook. See the plane table below.

Which states a [[peripheral]] may declare. A peripheral’s hooks run on a shorter ladder than an artifact’s, and atc ota push refuses a cell the ladder never reaches. The whole plane:

Use check.enter, check.exit or install.enter to refuse a flash before it starts. All three run before any byte reaches the peripheral, and a non-zero exit from any of them stops the delivery with nothing written. check.enter is the earliest of the three. Declare the rejected states at artifact or stage scope instead.

Writable state: state_dirs

A bundle declares the writable directories its program needs. atcd creates them before every install.* and activate.* hook, so a check.* or download.* hook of a first delivery runs before they exist. They persist across versions.

[[bundle]]
name = "myapp"
source = "dist"
state_dirs = ["data", "cache"]

In KDL, the key is state-dirs "data" "cache" on the bundle node.

Rules. Each entry is one path segment: not empty, not . or .., and with no /. A list must not name one directory twice. Only a bundle may declare state_dirs. A bad entry fails the load with ATC-E0533, at atc ota push and at atc channel deliver. A device checks a signed list again and refuses a bad one.

Location. The root is <hook-home>/.local/state/atc/<bundle>/, where <hook-home> is the home directory of the hook user (see Hook user above). Each entry is a subdirectory of that root. atcd creates the root and each entry as the hook user, with mode 0700. The step does not depend on a hook: a bundle with no hooks gets its directories too.

Not in the delivery tree. atcd does not link, mount or copy the root into any version’s directory. A data/ that a program creates beside its own binary is a plain directory of that version, and the next version starts with an empty one. Give the program the root path: a hook reads ATC_STATE_DIR and passes it on, for example in the systemd unit that an install hook writes. Run the program as the hook user. No other account can open the 0700 root, so a program that you start over ssh as a different user cannot use it.

A bundle with no hooks. A one-shot CLI that you run by hand has no hook to pass the path on. Run it through atc shell instead:

atc shell user@robot -a myapp -- sh -c './myapp --data "$ATC_STATE_DIR/data"'

atc shell -a <bundle> runs the command as the hook user, in the bundle’s current version. When atc.toml declares state_dirs on that bundle, it also sets ATC_STATE_DIR to the same root that a hook gets. The program can read the variable itself, or take the path as an argument. The single quotes stop your local shell from expanding the variable. Without state_dirs, the variable is unset.

The shell sets the variable only when the root exists in the home of the user it runs as. With hook isolation off, hooks run as atcd’s own user, and the shell becomes atc-hooks. The root is then in another home, so the shell leaves ATC_STATE_DIR unset and prints the path it looked for on stderr.

What a hook sees. A hook of that bundle gets ATC_STATE_DIR, the root, and a state_dirs map in ATC_HOOK_CONTEXT from each name to its full path. A hook that atc ota run-hook, an uninstall or a recovery starts gets the same values. These read the root on disk, because the request carries no list.

Shared across channels. The root is keyed by bundle name alone. Two channels that ship a bundle of one name share its state. This is decided behaviour, not an accident.

Deletion. atc ota rm deletes the root when it removes the last installed version of that bundle name on every channel. A version switch, keep_n pruning and the housekeeping sweep never delete it. A failed delete does not fail the removal. atcd logs it and counts it in atcd.state_dirs.remove_failures.

Version skew. On a push, an atcd without the bundle-state-dirs-v1 capability cannot create the directories. atc ota push prints a warning for each bundle with state_dirs. On the pull plane, the list rides on the signed delivery manifest and there is no handshake. An atcd that predates the field ignores the list and creates nothing. So atc channel deliver prints a note for each bundle with state_dirs.

ATC_HOOK_CONTEXT

Every hook receives one atc variable, ATC_HOOK_CONTEXT, holding a compact JSON object. Field presence is stable: artifact, version, self_root.new, self_root.old, current_root, and peripheral are always present. JSON null means “absent”, so a parser never has to special-case a missing key. A hook bound to no artifact gets null for the whole artifact and version values: a stage hook, a delivery.* bookend and a gc.* bookend. When version is an object, version.old_sha is always present.

{
  "push_id": "01J…",                       // session id: a ULID on push,
                                           // "pull-activate-<delivery-id>" on pull,
                                           // "manual-run-hook-<conn-id>" for `atc ota run-hook`
  "artifact": { "kind": "oci",             // "oci" | "bundle"; the whole object is
                "name": "tempo" },         // null for a hook bound to no artifact
  "version":  { "new_sha": "abc…",         // version being installed/activated; the whole
                "old_sha": null },         // object is null like `artifact`. old_sha: the
                                           // previously-active version, or null on first install
  "state":  "install",                     // any name from the states table above
  "phase":  "enter",                       // enter | exit
  "trigger": "push",                       // push | pull | user_activate | gc | manual_run_hook | …
  "artifacts_dir": "/var/atc/…/artifacts", // artifacts umbrella dir on the device
  "self_root": { "new": "/var/atc/…/versions/abc…",  // new version's root dir, or null
                 "old": null },            // previous version's root dir, or null.
                                           // At check.*, download.*, install.enter,
                                           // rollback.* and failure.*, @self and the
                                           // working dir are "old"
  "current_root": "/var/atc/…/delivery/current/tempo",  // stable pointer, or null
  "peripheral": null                       // peripheral-flash context, or null
}

Read fields with awk, through the ctx helper in How hooks run. The document arrives in the ENVIRONMENT, not on stdin, so ENVIRON["ATC_HOOK_CONTEXT"] is the read; a per-artifact hook gets /dev/null on stdin. The same call works from #!/bin/sh and from #!/usr/bin/env bash. The schema is defined in crates/atc-lifecycle/src/hook_context.rs, which is the source of truth.

current_root: the path a service unit must name

current_root is the artifact’s delivery/current/<name> pointer. atc flips it atomically on every push, switch and rollback, so a systemd ExecStart that names it follows every later version with no edit. Take the field; never derive the path yourself:

set -eu
# Print one dotted field of ATC_HOOK_CONTEXT, or the second argument when
# it or an object above it is null. With no second argument, a null field
# exits 1 and a field under a null object prints "". A missing key exits 1.
ctx() {
  awk -v path="$1" -v nargs="$#" -v dflt="${2-}" '
    function fail(why) { print "ATC_HOOK_CONTEXT." path ": " why > "/dev/stderr"; exit 1 }
    function find(t, k,   j, c, d, q) {
      k = "\"" k "\":"
      for (j = 1; j <= length(t); j++) {
        c = substr(t, j, 1)
        if (q) { if (c == "\\") j++; else if (c == "\"") q = 0 }
        else if (c == "{") d++
        else if (c == "}") { if (--d == 0) return 0 }
        else if (c == "\"") { if (d == 1 && substr(t, j, length(k)) == k) return j + length(k); q = 1 }
      }
      return 0
    }
    BEGIN {
      s = ENVIRON["ATC_HOOK_CONTEXT"]; n = split(path, key, ".")
      for (i = 1; i <= n; i++) {
        if (substr(s, 1, 4) == "null") { print dflt; exit 0 }
        if (substr(s, 1, 1) != "{") fail("not an object")
        p = find(s, key[i])
        if (p == 0) fail("no such field")
        s = substr(s, p)
      }
      if (substr(s, 1, 4) == "null") { if (nargs < 2) fail("null"); print dflt; exit 0 }
      if (substr(s, 1, 1) != "\"") fail("not a string")
      for (j = 2; j <= length(s); j++) {
        c = substr(s, j, 1)
        if (c == "\"") { print v; exit 0 }
        if (c == "\\") { j++; c = substr(s, j, 1); if (c != "\"" && c != "\\" && c != "/") fail("holds a control escape") }
        v = v c
      }
      fail("not a string")
    }'
}
root=$(ctx current_root)
[ -n "$root" ] || { echo "install_unit.sh: current_root is null" >&2; exit 1; }

The set -eu line and the emptiness test are part of the recipe. A reader that is absent from the device gives an empty $root, and the hook then writes a broken ExecStart and exits 0. The push reports the hook as successful.

It is a path, not a directory the field checked. At install.exit on a first install, nothing under delivery/ exists yet. That is exactly the hook which writes the unit file. So write the value out, and do not test it first. self_root.new is the field filtered by existence, and current_root is not.

Do not read the incoming version through it before the flip. On an upgrade, current_root names the OUTGOING version at every cell up to the flip, install.exit included. A test such as [ -x "$root/bin/myapp" ] there inspects the old tree. To inspect the version being installed, read self_root.new. An @self hook at install.exit also runs in the incoming tree, so a path relative to its working directory reads that tree too.

Do not compute it from the hook’s working directory. A hook runs in artifacts/<name>/versions/<sha>, which is a sibling of delivery/, not a path under it. Counting levels up from $PWD and stripping from /delivery/ both produce a plausible string that resolves to nothing.

current_root is null for a peripheral-flash hook. That plane’s artifacts_dir is the artifact’s own version dir, not the channel umbrella. No channel pointer can be named from it, and a firmware flash has no service pointer to follow.

What each field names at each cell

The table gives, for each delivery cell of an ARTIFACT-scope hook, whose declaration runs, which tree a @self script runs in, and which version each field names. It uses three words for two versions:

• incoming: the version that the delivery or the switch brings.
• outgoing: the version that was active before it.
• active and restored: the outgoing version again. It is “active” before the install, and “restored” on the unwind ladder, which brings it back.

How to read the columns:

• A declaration is the atc.toml that decides whether the hook exists, its timeout_secs and its script path. “active” and “restored” name the version that was active before this delivery, so they are the outgoing version’s declaration. A @self hook at those cells is that version’s hook: a release cannot add, remove or change it there.
• A @self/<path> script joins the tree of the version whose declaration it is. That tree is also the hook’s working directory. A @<name>/... or an absolute script names no version tree. That hook runs in self_root.new if that directory exists, else in the directory of the script.
• The trigger column gives the value the hook reads. It does not say which walk dispatches the cell. A switch runs no check, download or install. A push that flips nothing runs no activate and no deactivate; see Behavior changed.
• On a first deploy there is no outgoing version. old_sha and self_root.old are null, no @self hook runs at an “active” or “restored” cell, and no deactivate.* runs.

A stage hook (top-level [lifecycle.hook.*]) is bound to no artifact. It gets null for artifact, version and current_root, and it reads the INCOMING manifest at every cell, deactivate.* included. A peripheral-flash hook gets null for current_root.

The table pins the version pair that a PUSH dispatches. Three other callers pass a different pair to the compensating activate.*:

• A pulled delivery, an atc ota switch and a boot recovery pass the restored version as both new_sha and old_sha. Only a push passes the failed version as old_sha.
• A boot recovery runs the same cells as the compensation, under the trigger daemon_restart_recovery, with the outgoing declaration in the outgoing tree.

uninstall.* and atc ota run-hook are not deliveries. An uninstall.* hook runs the declaration of the version being deleted, in that version’s tree. atc ota run-hook runs the declaration of the version that is active now, in that version’s tree, at every cell. It gives new_sha as that version, old_sha as the version before it, and trigger as manual_run_hook. Without -o, if no artifact declares the cell, it runs the active delivery’s stage hook, which gets null version fields.

version.new_sha has three meanings

So new_sha names the failed version at rollback.* and the restored version one step later, at the compensating activate.*. To tell a rollback from a forward activation, read trigger. Do not compare new_sha with the version that current_root names:

case "$(ctx trigger)" in
  auto_rollback_veto | daemon_restart_recovery) action=restore ;;
  *) action=forward ;;
esac

The compensating activate.* runs the outgoing version’s own script, so a version’s activate.* must also bring that same version back after a failed upgrade. Write it to run twice without harm.

What current_root resolves to

current_root is <channel>/delivery/current/<artifact>. delivery/current is a symlink to the active delivery’s directory. In that directory, <artifact> is a symlink with the relative target ../../artifacts/<artifact>/versions/<sha>. So sha below is the sha of the version that current names at that moment:

root=$(ctx current_root)
sha=$(basename "$(readlink -f "$root")")

The last column of the table says which version that is at each forward cell. Before the flip it is the outgoing version. After the flip it is the incoming version. On a first deploy the link does not exist before the flip, so readlink -f prints nothing. Test [ -e "$root" ] before you read the sha.

On the unwind ladder:

• At rollback.enter, current_root still names the version that current held when the hook refused. After the flip, that is the incoming version. atc ota switch -a is the exception: it puts the pointer back first.
• At the compensating activate.*, at rollback.exit and at failure.*, it names the restored version if atc put the pointer back. If that revert failed, or a newer delivery took the channel, it names whatever current names then. Read the link. Do not assume.
• A first deploy refused after the flip has no version to restore, so atc removes delivery/current.

A progress hook ([lifecycle.progress.download]) gets the pointer of the artifact being DOWNLOADED, not of the artifact that ships the script. Its new_sha is the version being downloaded. Its old_sha, self_root.new and self_root.old are null.

An outgoing version with no deactivate.* hooks

Warning: no artifact hook stops the outgoing version when it declares no deactivate.* hook. deactivate.* reads the OUTGOING version’s declaration, not the incoming one. A version shipped before you added the hook declares none. So does a version installed by an atcd too old to keep the declaration. On a push, a pull or a switch away from such a version, a workload that only your hooks start and stop keeps running while the incoming activate.exit runs.

Two things can still stop it: an incoming stage-scope deactivate.*, and the stop that atc runs for a workload that atc itself manages.

So write the incoming activate.exit to take over a workload that still runs. Use systemctl --user restart, not enable --now. See A name shared across versions.

A refusal later in that delivery has no teardown to undo either. If no member’s deactivate.* ran, the unwind runs no compensating activate.*.

Behavior changed: a push without --switch runs no activate.*

Changed in atcd v0.1.5 and nightly-2026-10-02 (#6361, 2026-10-01). A push that flips nothing dispatches no activate.* and no deactivate.*, at any scope. Two pushes flip nothing: a push without --switch (stage-only), and a re-push of the version already active. An artifact-scope commit.* does not run either. A stage-scope commit.* still runs.

Before that change, such a push ran activate.enter and activate.exit with current still on the old version. So an activate.exit hook written for that behavior often carries a guard: exit early when new_sha is not the version that current_root names. On atcd v0.1.5 and later, a forward activate.exit runs only after the flip, so that guard never fires there. Keep it only while a device runs an older atcd.

Do not move the same guard to activate.enter. There it fires on every upgrade, because current_root still names the outgoing version.

What a hook does not receive

ATC_HOOK_CONTEXT is the only atc variable. atc shell -a <name> also sets ATC_ARTIFACTS_DIR, ATC_ARTIFACT_KIND, ATC_ARTIFACT_NAME, ATC_SELF_ROOT and ATC_NEW_SELF_ROOT for interactive work, and it starts you in delivery/current/<name>. A hook receives none of those. It starts in the versioned tree instead. So a script written in that shell fails on its first real push with unbound variable. Debug against the document, or against atc ota run-hook, which dispatches through the real hook path.

A hook’s exit status is its last command’s

There is no separate success signal. The shell’s usual rule applies, and a non-zero status vetoes the delivery. A diagnostic hook that ends in a find over one unreadable directory therefore fails a push whose real work already succeeded, and the reported failure names the hook rather than the find. End any script that reports rather than gates with an explicit exit 0.

Worked example: a commit.exit hook

Drop a marker file once a bundle has been committed (made active) on the robot:

[config]
autoswitch = true        # commit/switch on push without an explicit --switch

[[bundle]]
name = "myapp"
source = "dist"          # directory ATC chunks + sends

# Runs AFTER myapp is committed (made active) on the device.
[bundle.lifecycle.hook.commit.exit]
script = "@self/scripts/mark_committed.sh"   # path inside myapp's own tree
timeout_secs = 60

scripts/mark_committed.sh lives inside the bundle source (dist/), so it is shipped with the artifact and resolved via @self. It must be chmod +x with a shebang in dist/ — see How hooks run.

A marker is a report, so commit.exit is the right gate for it. If the script were instead the thing that brings myapp into service, activate.exit is the better home: it is the first gate after the pointer flips, so the window where the device points at the new version but is not yet running it stays as short as possible. See the hook cutover runbook.

With multiple [[bundle]] entries, a [bundle.lifecycle.hook.*] table binds to the [[bundle]] defined immediately above it (TOML array-of-tables semantics) — keep each bundle’s hooks directly under that bundle.

Worked example: a long-running service

This is the pattern behind “ship a bundle and keep it running across reboots”. atc has no [service] key and starts no process of its own. A systemd unit does that, and three lifecycle hooks install it and drive it.

Bootstrap already prepares the device for the rootless half of this. It creates the atc-hooks user, gives it the home directory /var/lib/atc-hooks, and runs loginctl enable-linger for it. Linger is what starts that user’s systemd manager at boot with no login, so a --user unit the hooks enable comes back after a reboot. Every hook already receives XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS for that manager, so the scripts below need no sudo and no root daemon.

The declaration

[[bundle]]
name = "myapp"
source = "dist"

# Write the unit and reload systemd BEFORE the pointer flips. A unit file that
# systemd refuses fails the delivery here, instead of after the robot is live.
[bundle.lifecycle.hook.install.exit]
script = "@self/scripts/install_unit.sh"
timeout_secs = 60

# Start the new version. activate.exit is the first gate after the flip.
[bundle.lifecycle.hook.activate.exit]
script = "@self/scripts/start_service.sh"
timeout_secs = 60

# Stop the outgoing version. This declaration runs on the NEXT delivery, when
# the version you ship now becomes the outgoing one.
[bundle.lifecycle.hook.deactivate.enter]
script = "@self/scripts/stop_service.sh"
timeout_secs = 60

dist/scripts/install_unit.sh

The unit points at delivery/current/myapp, not at this version’s own directory. delivery/current is the pointer atc flips atomically on every push, switch and rollback. A unit pinned to one version’s SHA cannot follow it.

The hook reads that path out of ATC_HOOK_CONTEXT.current_root rather than hard-coding it. The same script then works on any channel and any store root. It also never walks up from its own working directory, which is a sibling of delivery/. See current_root.

ConditionPathExists= guards the one case where the pointer is absent. A rollback restores the earlier delivery, and the pointer follows it. A first delivery has no earlier one, so a veto on it removes delivery/current. With Restart=always and no condition, the unit then restarts every few seconds into “No such file or directory”. The launcher’s one useful log line is buried under the repeats. The condition makes systemd skip the start and report the unit as inactive instead. The next successful push creates the path, and the activate.exit hook starts the unit as usual.

#!/bin/sh
# install.exit — pre-flip. The new tree is on the device and readable, and
# `current_root` still points at the OLD version (or at nothing, on a first
# install). Either way the path is the one the unit must name.
set -eu

# awk, not jq or python3: the device image carries neither, and a missing
# reader gives an EMPTY root rather than an error. `set -eu` and the test below
# make that case stop the hook.
# Print one dotted field of ATC_HOOK_CONTEXT, or the second argument when
# it or an object above it is null. With no second argument, a null field
# exits 1 and a field under a null object prints "". A missing key exits 1.
ctx() {
  awk -v path="$1" -v nargs="$#" -v dflt="${2-}" '
    function fail(why) { print "ATC_HOOK_CONTEXT." path ": " why > "/dev/stderr"; exit 1 }
    function find(t, k,   j, c, d, q) {
      k = "\"" k "\":"
      for (j = 1; j <= length(t); j++) {
        c = substr(t, j, 1)
        if (q) { if (c == "\\") j++; else if (c == "\"") q = 0 }
        else if (c == "{") d++
        else if (c == "}") { if (--d == 0) return 0 }
        else if (c == "\"") { if (d == 1 && substr(t, j, length(k)) == k) return j + length(k); q = 1 }
      }
      return 0
    }
    BEGIN {
      s = ENVIRON["ATC_HOOK_CONTEXT"]; n = split(path, key, ".")
      for (i = 1; i <= n; i++) {
        if (substr(s, 1, 4) == "null") { print dflt; exit 0 }
        if (substr(s, 1, 1) != "{") fail("not an object")
        p = find(s, key[i])
        if (p == 0) fail("no such field")
        s = substr(s, p)
      }
      if (substr(s, 1, 4) == "null") { if (nargs < 2) fail("null"); print dflt; exit 0 }
      if (substr(s, 1, 1) != "\"") fail("not a string")
      for (j = 2; j <= length(s); j++) {
        c = substr(s, j, 1)
        if (c == "\"") { print v; exit 0 }
        if (c == "\\") { j++; c = substr(s, j, 1); if (c != "\"" && c != "\\" && c != "/") fail("holds a control escape") }
        v = v c
      }
      fail("not a string")
    }'
}
root=$(ctx current_root)
[ -n "$root" ] || { printf '%s\n' "myapp: current_root is null" >&2; exit 1; }

# An ALLOWLIST, not a list of the characters that happen to break today. The
# artifact-name validator permits every one of them, so this is the only gate.
case $root in
  *[!A-Za-z0-9/._+:@%\ -]*)
    printf '%s\n' "myapp: unsafe character in the delivery path" >&2; exit 1 ;;
esac

# `%` is inside the allowlist, because a real path can hold one. Double it, or
# systemd reads a specifier: a name holding `%n` resolved to the UNIT name.
esc=$(printf '%s' "$root" | sed 's/%/%%/g')
unit_dir="$HOME/.config/systemd/user"
mkdir -p "$unit_dir"
cat > "$unit_dir/myapp.service" <<UNIT
[Unit]
Description=myapp
# A vetoed first delivery removes delivery/current. Without this line the
# unit restart-loops on "No such file or directory" until the next push.
ConditionPathExists=$esc/bin/myapp

[Service]
ExecStart="$esc/bin/myapp"
Restart=always

[Install]
WantedBy=default.target
UNIT

systemctl --user daemon-reload

One allowlist, and one escape. An enumeration of the characters that break is the wrong shape here. It is only ever as complete as the last person to look. Two rounds of review found %, then "/\/newline, then asked about $. The allowlist above ends that: a character must be added to it to reach the unit file.

% is the one allowed character that still needs work. A real path can hold one, and systemd expands it. Measured with systemd-analyze verify: a path holding %n resolved to …/pct<unit-name>/…, which names no directory. Doubling it to %% resolved the literal path.

$ needs no escaping, which is the counter-intuitive part. systemd expands no variable in the FIRST argument of a command line. The executable is that argument. Measured the same way, with Environment=HOME=/pwned set. A path holding $HOME resolved to the literal $HOME path. $$HOME resolved to a literal two-dollar path, so doubling would have broken a path that worked.

In an argument AFTER the executable a $ does expand, and $$ is the escape there. The ExecStart above has no such argument, and $ is outside the allowlist, so the question does not arise for a delivered path.

The allowlist is deliberately stricter than systemd. Measured over 26 characters, systemd fatally rejects only a single quote and a tab in an executable path. The message is “Executable path contains special characters”. It treats `, ;, &, | and the rest as literal, because it runs no shell, so those would work. They are refused here anyway. A bundle named a;b is a typo far more often than an intention, and a narrow allowlist is the version of this rule that cannot rot.

validate_artifact_name permits every character named above. It rejects an empty name, /, :, NUL, ./.., the reserved oci and .atc, and an over-long name. It rejects nothing else, so all of these are pushable today. Tightening it is tracked in #4293. Until then, this guard stands between a pushed name and a generated unit file.

install.exit is the right place to refuse. The hook has written no unit yet, so a non-zero exit vetoes the delivery with nothing installed, and it leaves the running service untouched.

The heredoc is unquoted (<<UNIT, not <<'UNIT') so $esc expands. Every other $ in the TEMPLATE expands too, and the shell gets there before systemd does. So a variable you want SYSTEMD to expand needs a backslash: write \$MAINPID, which reaches the file as $MAINPID. $$MAINPID does not work here, because the shell reads $$ as its own PID and writes that instead.

The interpolated path needs none of that. $esc is a variable VALUE, and the shell does not rescan what it substitutes. So the path is written once, exactly as the allowlist and the sed left it.

dist/scripts/start_service.sh

Use restart, not enable --now. enable --now exits 0 against a unit that is already active, and starts nothing. So a delivery whose outgoing version declared no deactivate.* keeps the old process alive under a green push. See A name shared across versions.

#!/bin/sh
# activate.exit — post-flip. delivery/current already names this version.
set -eu

systemctl --user enable myapp.service      # survive the next reboot
systemctl --user restart myapp.service     # always replace the running process
systemctl --user is-active --quiet myapp.service

The is-active line is the hook’s own check. restart reports success for a start job that is only queued, so without it a service that starts and dies still commits a green delivery.

dist/scripts/stop_service.sh

#!/bin/sh
# deactivate.enter — pre-flip, and the incoming activate.exit runs after it.
set -eu

systemctl --user stop myapp.service

The stop needs no test for “is my version still the current one”. The order guarantees the answer: deactivate.* runs inside activate, and the incoming version’s activate.exit runs after both deactivate gates.

The three scripts bind to the versions that ship them

A hook is resolved out of the version tree its declaration came from. @self/scripts/stop_service.sh means that version’s copy of that file. So stop_service.sh here is not the script that stops the version in this push. It is the script that stops the version already on the robot, and that copy shipped with it.

Editing atc.toml or a script therefore reaches the deliveries after this one, never the way out of a version already installed. Two consequences bite in practice:

• The pair does nothing on the push that first adds it. The outgoing version declared no deactivate.*, so nothing stops the old process, which is why start_service.sh must restart rather than enable --now.
• A relabelled push of unchanged content still runs the old stop_service.sh. That copy can be wrong: it can stop a unit the incoming version just started. Fixing the file does not fix that push. Push a version carrying the corrected script, then push again.

The push output does not tell you whose copy ran. A hook line reports the declared ref, @self/scripts/stop_service.sh, which reads the same for every version. So a stale deactivate.enter looks exactly like a current one. Reason it out from the order above, not from the output.

A wrapper that sources someone else’s environment

A ROS node, a Python venv, and most other environments need a small wrapper. The wrapper sources a setup script, then execs the program. Point ExecStart at the wrapper, not at the program.

Give that wrapper a #!/bin/bash shebang, not #!/bin/sh. On a device image /bin/sh is dash. Catkin’s install/setup.bash and ROS 2’s /opt/ros/<distro>/setup.bash both need bash. They read $BASH_SOURCE to find their own directory, and they use bash-only syntax elsewhere. Under dash they report Bad substitution or builtin: not found, and neither names the shell.

Source setup.bash, not setup.sh. This is a second fault with its own symptom, and the right shebang does not fix it. A POSIX shell cannot tell a sourced script its own path, so colcon’s install/setup.sh carries the prefix of the machine that built it. The robot has no such directory, and the script says so: The build time path '<path>' doesn't exist.

atc ota push catches neither mistake. It ships your script and never reads it, so the push reports success and the unit is broken.

Ship a ROS 2 install space built without --symlink-install. With that flag, colcon gives an ament_python package an editable install. Its install/<pkg>/lib/python3.X/site-packages/ then holds a <pkg>.egg-link and no .py files. The .egg-link holds an absolute path to build/<pkg> on the build machine, and that directory links into src/. The bundle holds neither directory.

The console script under install/<pkg>/lib/<pkg>/ looks the package up by its metadata, and the .egg-info with that metadata also stays in build/. So the push succeeds, and the node fails on the robot with importlib.metadata.PackageNotFoundError or ModuleNotFoundError.

Run a plain colcon build for the install space you ship. A ~/.colcon/defaults.yaml with symlink-install: true under build: sets the flag on every build, so check that file too. atc ota push warns when a bundle carries a .egg-link, or a __editable__ pointer from pip install -e. It does not refuse the push.

Do not set -u in that wrapper. ROS 2’s /opt/ros/<distro>/setup.bash reads AMENT_TRACE_SETUP_FILES without setting it, and a venv activate does the same with variables of its own. Under nounset the wrapper dies before your program starts, and the failure names a variable you never wrote. set -e alone is fine.

#!/bin/bash
# Sourced environment first, strict mode after it. Never around it.
set -e
. /opt/ros/humble/setup.bash
exec "$(dirname "$0")/../bin/my_node"

Checking a --user unit’s state

The unit belongs to the hook user’s manager, not to yours. So systemctl --user status myapp.service, run over ssh as your login user, answers Unit myapp.service could not be found. The push can be correct, and the service can run. Ask with the hook identity:

atc shell robot@device -- systemctl --user status myapp.service

atc shell reconstructs the hook environment, XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS included. A plain sudo -u atc-hooks systemctl --user ... sets neither variable, and answers Failed to connect to bus. Set XDG_RUNTIME_DIR yourself where atc shell is not available:

sudo -u atc-hooks env XDG_RUNTIME_DIR="/run/user/$(id -u atc-hooks)" \
  systemctl --user status myapp.service

Reading a --user unit’s logs

journalctl --user -u myapp.service, run as atc-hooks over atc shell robot@device, can answer No journal files were found while the unit is actively logging. The user session’s journal is not always reachable from a shell that never logged in. The records still reach the system journal, where each one carries the user unit name. Read them from outside:

sudo journalctl _SYSTEMD_USER_UNIT=myapp.service _UID="$(id -u atc-hooks)" -f

Name the two fields. journalctl also has a --user-unit= flag, which matches on the caller’s own session, so a root shell needs the explicit form above. For every unit of that user at once, ask for the manager instead:

sudo journalctl -u "user@$(id -u atc-hooks).service" -f

An empty journal is not a dead service. systemd gives the unit a pipe, not a terminal. A Python or C runtime buffers stdout in blocks on a pipe. A line then sits in the buffer until the buffer fills or the process exits. A rospy node that prints one line per message can show one startup line for hours. It handles every message correctly.

Which answer applies depends on how the node writes the line.

• print() and a bare write reach stdout, and nothing else. Set Environment=PYTHONUNBUFFERED=1 in the unit. The journal then gets each line as the node writes it. For a C program that the unit starts, wrap the command in stdbuf -oL. stdbuf cannot change a Python program, because Python sets its own buffering.

• rospy.loginfo and the other framework loggers reach the framework’s own files instead. ROS writes them under ~/.ros/log/<run_id>/. Read them as the hook user:

  atc shell <target> -- sh -c 'ls -t ~/.ros/log/'
  

  Quote the command. The tilde must expand on the device, and your own shell expands an unquoted one first. Sort by time rather than reading latest. The latest symlink is a roslaunch convenience, so a node the unit starts directly can leave none.

A node can use both, and each line reaches one of the two places. So read both before you call the service broken.

A ROS 1 roslaunch service logs almost nothing there for a second reason. roslaunch redirects each node’s output into its own files under ~/.ros/log/<run_id>/. So output="screen" never reaches systemd’s capture. Read ~/.ros/log/latest/ for the node’s own logs. The journal carries only roslaunch itself. An unbuffered stdout does not change that.

A service that needs time to stop

At shutdown, systemd stops the hook user’s systemd --user manager (user@<uid>.service). The manager then stops its own units. A unit cannot outlive its manager. So the manager’s TimeoutStopSec caps the stop time of every hook-launched user service, whatever the unit’s own TimeoutStopSec says.

Ubuntu and Jetson L4T set that cap to 5 s. They ship /usr/lib/systemd/system/user@.service.d/timeout.conf with TimeoutStopSec=5. A service that saves state on SIGTERM and needs longer is killed before the save ends.

By default, atc bootstrap atcd install raises the cap to 180 s for the hook user only. It writes /etc/systemd/system/user@<uid>.service.d/zz-atc-hook-user-stop-timeout.conf, where <uid> is the numeric uid of the hook user. The file name sorts after timeout.conf, and systemd applies the drop-in that sorts last. The install then reads the value back with systemctl show. It prints a warning when another drop-in still sets a different value.

Other users’ managers, and the user@.service template, keep their values. A daemon-reload applies the change to a running manager, so the hook user’s manager does not restart.

Bootstrap writes no drop-in when hooks run as the daemon user, as with --root. atc bootstrap atcd upgrade does not write it. Run atc bootstrap atcd install again on a device installed before this change. atc bootstrap atcd uninstall removes the file when it removes the hook user: on a purge, when no other atcd store uses that user. It removes only a file whose first line is the bootstrap header. --keep-store keeps the hook user, and the file with it.

The install records the uid that it wrote the drop-in for, in /etc/atc/<service>.hook-user-stop-timeout-uid. The hook user can change in three ways:

• --set daemon.hook_run_as_user=<user> names another hook user.
• Hooks run as the daemon user again.
• The account is deleted and created again with a new uid.

Then the next install, and a purge, remove the drop-in of the recorded uid. They remove only that one path, and only a regular file with the bootstrap header. They keep it when the record of another atcd unit names that uid. They never remove a drop-in of any other uid.

--keep-store keeps the drop-in and the record. A purge keeps the record while the drop-in it names stays. Three limits remain:

• A failed first install rolls back the unit, but it keeps the drop-in and the record, as it keeps the hook user.
• A device installed before the record has none. Its next install writes one. If that install also changes the hook user, the drop-in of the former uid stays. Remove it by hand.
• Two atcd units can share one hook user. If the other unit was installed before the record, it has no record that names the uid. A change of the hook user of this unit then removes the shared drop-in. The next install of the other unit writes it again.

Two flags of atc bootstrap atcd install change the drop-in:

• --hook-user-stop-timeout <SECS> writes the drop-in with that value. The value must be from 10 to 3600. Above 30 min, poweroff.target stops waiting (JobTimeoutSec=30min) and powers the host off.
• --no-hook-user-stop-timeout removes the drop-in, then runs systemctl daemon-reload, so the vendor’s value applies again. A drop-in that sorts after it still applies. It removes only a file whose first line is the bootstrap header, and it keeps a symlink at the path. With no file, it changes nothing.

The two flags conflict, and each one refuses --root. When hooks run as the daemon user, there is no hook user’s manager. Then a flag changes nothing, and the install prints a note that says so.

The flags are not saved. Each install sets what its own flags say. An install with no flag writes the 180 s drop-in again, also after --no-hook-user-stop-timeout. So give the flag to every install of the device.

atc bootstrap atcd upgrade does not read the flags, and it does not change the drop-in. A repair by atc ota push of an existing unit does not change the drop-in either. When atc ota push installs atcd on a device for the first time ([bootstrap].auto), the install writes the 180 s drop-in.

Check your units against the value you set:

• Set the unit’s TimeoutStopSec to the longest stop you expect. The cap must be at least that value plus a margin of about 30 s.
• A unit with ExecStop= can take about twice its TimeoutStopSec. systemd applies the timeout to each ExecStop= command and again to the wait after SIGTERM.
• Units that are not ordered against each other stop in parallel. The slowest ordered chain counts, not the sum.

If a unit needs more than 180 s, use --hook-user-stop-timeout. You can also add a drop-in for the hook user that sorts after zz-atc-hook-user-stop-timeout.conf, for example zzz-local-stop-timeout.conf. The install keeps your value. It prints the warning only when it runs its script, and an install that finds nothing to repair runs none. atcd install reads the file, not the value systemd holds.

With or without a flag, the install keeps a regular file at the drop-in path that bootstrap did not write. --no-hook-user-stop-timeout also keeps a symlink there. An install that writes a value, with or without the flag, refuses a symlink and stops. The install names a kept file only when it runs its script.

These points are yours to check. atc does not and cannot verify them:

• Late or network mounts. user@<uid>.service stops before local fstab mounts unmount. A unit that writes to a network mount, an automount or a late mount at stop needs ordering that atc does not add.
• Other users’ managers. The drop-in covers only the hook user. A workload that runs in another user’s manager keeps that manager’s timeout.
• Container stop timeouts. podman stop sends SIGKILL after 10 s by default, whatever systemd allows. A unit that runs a container must raise it: podman run --stop-timeout, podman stop -t, or StopTimeout= in a Quadlet file.

systemctl --user needs more than linger

Bootstrap enables linger, which is necessary and not sufficient. A rootless systemd --user manager also needs dbus installed, and cgroup delegation for user slices. A nested container usually has neither. Docker-in-Docker is the common case, and a device image built that way behaves the same. The manager then fails with status=219/CGROUP, however correct the unit is. A missing dbus on its own gives Failed to connect to bus.

A hook cannot fix either. The next section gives the two supported paths for such a device.

Root work on a device with no user session

This section is for a device with no dbus or no systemd --user manager. On such a device a hook cannot drive systemctl --user. Use one of two supported paths, in this order.

1. Give the hook user a supplementary group. Use this path when the hook must read or write a root-owned file, socket, or device node. Add the hook user (atc-hooks by default) to a group that owns the resource. Then give that group the access. When the hook user differs from the daemon user, atcd reads the group set at every spawn, so you need no daemon restart. When hooks run as the daemon’s own user, as on the hand-launched no-systemd recipe, they inherit the daemon’s groups. Then restart atcd after the group change. The recipe, with the K3S kubeconfig as the example, is in the atcd non-root runtime model.
2. Ship a root-owned system unit in a .deb. Use this path when the work must run as root, for example a service that writes to /usr or /etc. This path needs systemd as PID 1, for bootstrap and for the unit itself. Bootstrap the device with atc bootstrap atcd install --root. atcd then runs as root, and its hooks inherit root. The hooks install the .deb and start its unit with systemctl. The complete producer tree is examples/deb-systemd-service/.

A root hook user under a non-root daemon is refused by design. atc refuses hook_run_as_user = "root" while run_as_user is not root. The error is ATC-E0506. This is a decision, not a gap (#4785). There is no opt-in that lifts it. To run hooks as root, run atcd as root with path 2.

A sudoers rule that lets the hook user call sudo is not a supported path for this device shape. atc does not install, test, or check such a rule.

Ubuntu 24.04 and user namespaces

Ubuntu 24.04 sets kernel.apparmor_restrict_unprivileged_userns=1. With that setting, a binary needs an AppArmor profile that grants userns before it can use an unprivileged user namespace. Without one, the kernel confines the process to the unprivileged_userns profile and strips its capabilities.

Ubuntu ships such profiles for podman, buildah and crun. It ships none for skopeo. So podman works for the hook user, and skopeo copy fails with unshare: Operation not permitted. atc oci import from a registry runs skopeo, so a first push can fail this way while every rootless check passes.

Install a profile for skopeo. It is Ubuntu’s podman profile with the name and the path changed:

sudo tee /etc/apparmor.d/skopeo > /dev/null << 'EOF'
abi <abi/4.0>,

include <tunables/global>

profile skopeo /usr/bin/skopeo flags=(unconfined) {
  userns,

  include if exists <local/skopeo>
}
EOF
sudo apparmor_parser -r /etc/apparmor.d/skopeo

A file under /etc/apparmor.d loads again at boot. atcd ctl setup-rootless --user <name> reports check R6 as WARN while the sysctl is 1 and the kernel lists no skopeo profile in /sys/kernel/security/apparmor/profiles. It also warns when the file changed after the kernel loaded it, and when the file does not attach to /usr/bin/skopeo. The file alone is not a pass. A profile the parser refused, or that nobody loaded, leaves skopeo failing. The delivered agency/podman bundle needs no extra step: setup-rootless --provision installs /etc/apparmor.d/atc_podman for it.

Do not set the sysctl to 0. That lifts the restriction for every unprivileged binary on the host, and it hides the missing profile instead of adding it. Ubuntu 20.04 and 22.04 have no such sysctl and need no profile.

What the push checks after activate.exit

A hook can exit 0 and still leave a unit that does not run. So after a push that activates a bundle, atc ota push asks the hook user’s systemctl --user about the unit of that bundle. It polls every half second for up to 10 seconds. A unit counts as running when it stays active for 2 seconds with no restart. A slow starter therefore does not fail the first poll.

A unit that is not running gives a WARNING in the push summary. The warning names the unit and its last state, for example failed, activating (auto-restart) or restarting. The push still succeeds, the exit code does not change, and atc reverts nothing. --json carries the same list as inactive_units.

The push finds the unit in one of two places:

• An atc.kdl systemd <name> { from <bundle>; unit "<unit>" } workload. The push checks that unit when it activates <bundle>. If systemd does not know the unit, the warning reports not-found.
• Otherwise, the unit named <bundle>.service, which is the name this example uses. The push checks it only when the hook user’s manager has it loaded. A bundle with no such unit gets no check and no warning.

The check says nothing when it cannot run: a push over --socket, or a device with no user session. No poll starts with less than 2 seconds of the budget left. When a later poll fails, the push still reports each unit an earlier poll saw failed, inactive, not-found or restarting. It drops a unit last seen activating, because that unit may be a slow starter.

What this example does not cover

• A root-owned system unit, and a .deb. Use examples/deb-systemd-service/, a complete producer tree with its own e2e test. That path needs atc bootstrap atcd install --root, because a .deb writes to /usr and /etc, and the unprivileged hook user cannot.
• A device image with no user-session support. systemctl --user needs libpam-systemd and dbus-user-session. Bootstrap probes for both and reports what it found. On a device that has neither, systemctl --user answers Failed to connect to bus. For the supported paths on such a device, see Root work on a device with no user session.
• The systemd <name> { unit ... } workload syntax. It parses and it persists, but nothing starts it. See modules-today.md. Hooks are the mechanism that works today.

Debugging hooks

Hook stdout/stderr is recorded on the controller once the hook is dispatched: one <state>.<phase>__<hook-id>.{stdout,stderr} pair per hook run, under $XDG_CACHE_HOME/atc/hooks/<timestamp>-<target> (~/.cache/atc/hooks/… by default — the exact path is printed at push start). Pass --capture-hooks <DIR> to pin a specific directory; recording is unavailable only under --json.

spawn_error means the exec itself failed — the hook never ran, so there is no exit status. The OS error is still reported: the CLI prints it as a [stderr] line on the failure replay, and it is written to the hook’s .stderr capture file. Look for spawn failed: Exec format error or spawn failed: Permission denied. The same text is also available after the fact, base64-encoded in stderr_tail_b64, from atc ota status --json. Triage the usual causes on the device:

• missing or bad shebang (ENOEXEC / Exec format error) — check head -1 of the shipped script under the artifact’s version dir;
• missing exec bit on the shipped file;
• hook-user setup failure (see the hook-user paragraph above).

Warning: running the script by hand over SSH masks a missing shebang — bash falls back to interpreting ENOEXEC files as sh scripts, so a script that “works” when run manually can still spawn_error under atcd, which exec’s it directly.

Run a hook out-of-band without a push:

atc ota run-hook user@robot commit.exit

-o NAME narrows to one declared artifact, spelled as ota push’s selector (--artifact stays as a hidden, deprecated alias). Without it, run-hook picks the single artifact declaring the cell and refuses when several do.

Which scope run-hook reaches. A hook declared under an artifact ([bundle.lifecycle.hook.activate.exit]) is read from that version’s .atc/lifecycle.pb. A stage hook — top-level [lifecycle.hook.<state>.<phase>] — is bound to no artifact, so it is read from the active delivery’s .atc/delivery.pb archive instead, the same file the gc.* bookends live in. run-hook consults that archive only when NO artifact declares the cell. So when the same cell is declared at both scopes, run-hook runs the artifact hook, and the stage hook is not directly reachable — a push runs both, run-hook runs one.

A stage hook is archived by the delivery that declared it. A delivery committed by an atcd older than this behaviour wrote no stage table, so run-hook reports no hook configured for its stage cells until the next push or poll re-commits the delivery.

A stage hook run this way sees no delivery members. On a push or a poll, a stage hook’s context lists the artifacts of the delivery being applied, each with the version it is moving from and the version it is moving to. run-hook applies no delivery, so it reports an empty list rather than a made-up one. Write a stage hook that must also be reachable by hand so that it reads the device state it acts on, not the member list.

run-hook requires a bootstrap-provisioned device (the atc-hooks user plus its NOPASSWD sudoers drop-in); on a manual no-systemd install it fails with “device not provisioned for a hook shell” — trigger the hook via a push instead, or exec the script from the version dir with ATC_HOOK_CONTEXT set by hand.

On a no-systemd device (where run-hook is unavailable), confirm a hook ran by triggering a push and checking the hook’s own side effect on the device (e.g. its marker file) alongside the captured stdout/stderr under ~/.cache/atc/hooks/ on the controller.

Testing hooks without a device

atc ota push --local runs the whole push against a throwaway atcd on your own machine. There is no target, no SSH, no bootstrap and no root:

atc ota push --local --switch --hook-output=always

It makes a test root, starts a real atcd on it, pushes, and stops the daemon again. Because it is the real daemon, a hook that passes here ran against the real executor — same ATC_HOOK_CONTEXT, same working directory, same timeout, same exit-code rules. A hook that exits non-zero vetoes the delivery, and the command exits non-zero with it.

--switch or --no-switch is required, and --local will not guess. --switch activates the pushed version, so the activate and commit hooks run too. --no-switch stages, which runs the download and install hooks and stops. Guessing would either report half a hook run as a whole one, or override a project whose own policy is to stage.

Keep the root to look inside it. Give --local-root an absolute path. A relative path makes every hook fail to spawn (spawn failed: No such file or directory (os error 2)), and the failure vetoes the whole push:

atc ota push --local --local-root /tmp/hook-test --switch --hook-output=always
ls /tmp/hook-test/store    # the store atcd wrote
cat /tmp/hook-test/atcd.log  # what the daemon said
ls /tmp/hook-test/hooks    # each hook's recorded stdout and stderr

Keep the path short as well. The root holds atcd’s Unix socket, and a socket path may not exceed 103 bytes. A longer one is refused up front with ATC-E0463.

A root you name is created and kept, and holds everything about the run. A root you do not name is a temp directory that goes away with the command. A root that already holds something else is refused, so --local-root . cannot scatter a store through your project.

To activate a version that a --no-switch push staged in a kept root, run atc ota switch --local-root <DIR> with the version or --delivery-id. The push prints that command. The switch starts atcd on the root for this command only, so the activate and commit hooks run as they do for --switch. A temp root holds nothing after the push, so that push prints no command.

Four things this does not do.

• It is not a sandbox. The test root relocates atcd’s store and nothing else. Hooks run under your uid, in your home, against your user services. A hook that writes an absolute path writes it here, on this machine. The command prints this every time it runs.
• A hook that is still running when the command ends is not killed. Every hook gets its own process group and only atcd knows them, so killing the daemon leaves a slow hook running. It keeps making changes on this machine after the command has returned.
• It is not the device’s hook user. On a robot, hooks drop to atc-hooks. Locally there is no privileged daemon to drop from, so they run as you. A hook that works only because of something your account has can still fail on a device — and one that needs the service account’s home can fail here and pass there.
• Pushing the same content twice runs no hooks at all. The daemon short-circuits a push whose content is already active. Change something between runs into a kept root, or the second run does nothing and reads as a broken feature. A changed second push is also the only way to reach the deactivate hooks on a push, which need a previous version to deactivate. atc ota switch back to the earlier version reaches them too.
• It needs an atcd for this machine. Released atcd builds are Linux only. atc looks for atcd beside itself, then on PATH; ATC_LOCAL_ATCD names one directly.

--local takes no target, no --socket, no --ssh-port and no --dry-run: each of those names a device, and local mode is the absence of one. Two local pushes cannot share one root — the second is refused rather than becoming a second writer on the same store.

Common mistakes (all now rejected at atc ota push)

These shapes used to parse with no error and wire no hook — push still reported success and the hook never ran. They are now rejected with a message that shows the correct shape:

lifecycle = "x"                         # lifecycle must be a table, not a scalar
lifecycle = []                          # ...not an array
lifecycle.commit.exit = 12345           # missing the required `.hook.` segment
[lifecycle.commit.exit]                 # missing `.hook.` — `commit` is unknown
[lifecycle.hook.commit.exit]
run = "do-thing.sh"                     # wrong body key — it's `script`, not `run`

The correct form for the last one:

[lifecycle.hook.commit.exit]            # stage scope: @self forbidden, use @<name>/...
script = "@myapp/scripts/x.sh"

…or, more commonly, artifact-scoped under the bundle:

[bundle.lifecycle.hook.commit.exit]
script = "@self/scripts/x.sh"

atc.kdl reference

atc.kdl declares what a repository ships to a robot. It is a KDL 2.0 document. atc ota push reads it, and atc config lint checks it before anything depends on it. atc.kdl is an opt-in alternative format. atc.toml is the format atc ota init writes by default, and its reference is atc.toml reference. When one directory holds both files, atc reads atc.kdl, as Where atc finds the file says.

This page follows the parser in crates/atc-config/src/kdl_*.rs. Every kdl block on this page is also a test fixture under crates/atc-config/tests/fixtures/kdl/valid/, and a test fails when a block and its fixture differ. So each example lints. If this page and the parser disagree, the parser wins. File a docs bug.

Check a file

atc config lint
atc config lint path/to/atc.kdl
atc config lint path/to/atc.toml
atc config lint --json

With no argument, atc config lint reads the manifest that atc ota push reads. It finds that file as Where atc finds the file shows. A named file ending in .kdl is linted as KDL. Any other named file is linted as atc.toml, through the checks that atc ota push runs on it. An atc.toml lint does not check that a bundle source directory exists, because the build output may not exist yet.

When it finds neither file, lint stops with ATC-E0561 and names both. It prints every finding with an ATC-E#### code, and with a line number when the finding points at a place in the file. It exits non-zero on an error and exits zero when it finds only warnings.

A file can be valid and still be refused by atc ota push. Some constructs parse and have no delivery path yet. For each of them, lint prints the warning ATC-E0597, which starts “no errors, but atc ota push will refuse this file”. What the push refuses today lists them. Each example below says whether the push takes it.

Where atc finds the file

Commands that read the manifest, such as atc ota push and atc ota publish, find it as this table shows. atc config lint with no argument finds it the same way.

These commands look in the current directory, then in each parent directory, and stop at the first directory that has either file. So an atc.toml in a subdirectory is read before an atc.kdl above it.

ATC_KDL_CONFIG accepts 0, false, no or off to prefer atc.toml. Any other value, or no value, keeps the rule in the first rows. atc config lint prints ATC-E0598 for a file that atc will not read: one beside the file that wins, or one that discovery from the current directory does not pick.

Convert an atc.toml

atc config migrate writes an existing atc.toml as atc.kdl. It is optional. atc.toml stays a supported format, and the command never changes or deletes it.

atc config migrate                  # print the KDL on stdout
atc config migrate --write          # write atc.kdl beside the atc.toml
atc -c path/robot.toml config migrate

With no -c, the command reads the atc.toml that discovery finds. Each field goes to one of three places:

• A field that atc.kdl spells is written as KDL.
• A device setting is not written. The command lists it on stderr with the command that sets it on a device: atcd settings set <key>=<value>, or, for a key that the atcd unit holds, atc bootstrap atcd install with its flag (--root, --company) or with this atc.toml as -c. [daemon] keys other than socket_path and remote_path, the [storage] table, and [server] keys other than url, org and ca_bundle_path are device settings. server.url and server.ca_bundle_path are both: they are written to atc.kdl for the CLI and listed for the device.
• Any other field stops the command with ATC-E0792, which names the field. A [[peripheral]] lifecycle table is one: a peripheral’s flash hooks have no atc.kdl spelling. An unknown key is another.

Before it prints or writes, the command loads the KDL back through the KDL loader. It stops with ATC-E0792 if the result differs from the atc.toml in any repository field or in a hook that a push would carry.

--write refuses to replace an existing atc.kdl. After --write, both files are in one directory, so atc reads atc.kdl, as Where atc finds the file says. The command prints a note that says which file a command run from the current directory reads. Set ATC_KDL_CONFIG=0 to read atc.toml instead. atc bootstrap atcd install reads the same file, and atc.kdl does not carry device settings. If the atc.toml has device settings, the note says to pass it to that command with -c.

The grammar in five rules

1. A file is a list of nodes. Each node is a word, then its arguments, then an optional { … } block of child nodes.
2. A bare word is a string. bundle app and bundle "app" are the same node. Quote a value that has a /, a space, or a :, for example a path, an image reference or a platform.
3. Booleans are #true and #false. Numbers are bare: timeout-secs 30.
4. A ; separates nodes on one line: exit { script "@self/a.sh"; timeout-secs 30 }.
5. // starts a comment.

A name can appear only once where it means one thing. A second source in one artifact, or a second daemon block, is an error (ATC-E0562).

A minimal file

schema-version 1

bundle app {
    source "./dist"
}

The push takes this file.

Top-level nodes

Any other word at the top level is an error (ATC-E0542). The word artifact from an earlier draft is removed. Write the kind as the node name instead (ATC-E0571).

keep-n and allow-no-rollback are removed. They never changed how many versions a device keeps. A file that still has one of them parses, and lint prints the warning ATC-E0795. Delete the line. The device setting storage.version_keep_n sets how many versions a device keeps.

Artifacts

An artifact node is the kind, then the name: bundle web, oci camera, module updater. The name of a bundle or a module becomes a directory on the device, so it must be one path component. It can hold letters, digits, - and _ (ATC-E0566). Names are unique across all the artifacts and workloads in a file.

exclude and ship-python-bytecode behave as their atc.toml equals do. See Leave paths out with exclude and Python bytecode is left out by default.

Roles

A device can have several hosts, and each host has a role. An artifact with role "nav" installs only on the host whose role is nav. An artifact with no role installs on every host. A role is 1 to 63 of a-z, 0-9, - and _, and starts with a letter or a digit. Only a bundle and an oci take a role. Any other value or kind fails with ATC-E0536. The rule is the same as for role in atc.toml. See Devices with several hosts.

bundle nav-stack {
    source "./nav"
    role "nav"
}

atc ota push --device pushes to every host of a device. A device block maps each role to the SSH target of its host. It is the [device.hosts] table of atc.toml. Each child of hosts is one host: the node name is the role, and the argument is the target. A role follows the rule above (ATC-E0536), and a blank target fails with ATC-E0537.

bundle planner {
    source "./planner"
    role "nav"
}

device {
    hosts {
        nav "ops@nav-14"
        compute "ops@compute-14"
    }
}

bundle

bundle web {
    source "./dist"
    exclude "**/.git" "**/*.log" "tmp"
    hooks {
        install {
            exit { script "@self/scripts/check.sh"; timeout-secs 60 }
        }
        activate {
            exit { script "@self/scripts/start.sh" }
        }
        deactivate {
            enter { script "@self/scripts/stop.sh"; timeout-secs 30 }
        }
    }
}

The push takes this file.

oci

oci camera {
    source "ghcr.io/acme/camera:2.1"
    platform "linux/arm64"
}

The push takes this file. Leave out source for an image you built locally with atc oci build.

module

module updater {
    source "./modules/updater"
    payload "./modules/updater-assets"
}

The push takes this file. A module cannot have the name payload. Modules today describes the package that source must contain and what the device does with it.

Dependencies with requires

requires names the other artifacts that an artifact depends on. It is the requires array of an atc.toml lifecycle table. Each dependency must already be active on the device, in the channel the push delivers to. The device refuses a push that carries an artifact and its dependency together, so push the dependency first.

bundle app {
    source "./dist"
    requires config
}

The push takes this file, and the device takes it when config is already active there. Write each name bare, as one path component. An empty name, a <kind>/<name> entry such as bundle/config, and a name such as . or ~config are errors (ATC-E0791). Give several names on one requires line: requires config certs.

Hooks

A hooks block nests a state, then a phase, then a body:

hooks {
    <state> {
        <phase> { script "<reference>"; timeout-secs <n> }
    }
}

The bundle example above shows three hooks.

• State: check, download, install, activate, deactivate, commit, rollback, failure or uninstall.
• Phase: enter or exit. The error phase parses and never runs, so the file is refused if it declares one.
• script (required): @self/<path> for a file inside this artifact, @<artifact>/<path> for a file inside another artifact, or an absolute path on the device. A repository-relative path such as "./check.sh" is refused. At check.*, download.* and install.enter the new version is not installed yet, so @self runs the ACTIVE version’s script. A missing script skips the hook there, which covers a first delivery and a hook that this release adds. From install.exit on, @self runs the new version’s script. rollback.* and failure.* run the script of the version that the unwind restores, and skip when there is none. The atc.toml reference has the table for every gate.
• timeout-secs (optional): a whole number of seconds. 0 removes the time limit. With no timeout-secs, the limit is 30 seconds.
• allow-missing #true (optional): if the script does not exist on the device, skip the hook instead of failing the delivery. This applies on every delivery.
• allow-missing-initial #true (optional): the same skip, but only when the device has no earlier version of this artifact. On a later delivery, a missing script fails the delivery as usual.

The two allow-missing nodes are the atc.toml keys allow_missing and allow_missing_initial, with the same rules. Set one or the other, not both. They need an absolute script, and lint refuses them on @self or @<artifact> (ATC-E0775). Only a missing file is skipped. A script that exists but is not executable still fails the delivery. The atc.toml reference has the full rules and an example:

bundle agent {
    source "./dist"
    hooks {
        download {
            enter {
                script "/usr/lib/agent/pre-download.sh"
                allow-missing-initial #true
            }
        }
    }
}

A cell (one state and one phase) can have one hook only (ATC-E0564). The rules for when each cell runs, the order, the environment and the ATC_HOOK_CONTEXT document are the same for both formats. Read them in the Lifecycle hooks section of the atc.toml reference.

The top-level hooks block

A hooks block at the top level of the file holds the hooks that belong to the whole delivery, not to one artifact:

bundle app {
    source "./dist"
}

hooks {
    pre { script "@app/hooks/announce.sh"; timeout-secs 10 }
    activate {
        enter { script "@app/hooks/stop-sensors.sh" }
    }
    progress {
        download { script "@app/hooks/report.sh"; interval-secs 5 }
    }
    post { script "@app/hooks/verify.sh" }
    gc-enter { script "@app/hooks/may-uninstall.sh" }
}

The push takes this file.

A stage hook takes the states that the whole delivery crosses once: activate, deactivate, commit, rollback and failure. check, download and install run once for each artifact, so put those hooks in the artifact’s own hooks block. The body of each hook is script and an optional timeout-secs, as in an artifact’s hooks block. A stage hook also takes allow-missing. It does not take allow-missing-initial, because a stage hook belongs to no one artifact, so it has no first delivery. A bookend (pre, post, gc-enter, gc-exit) takes neither. A progress body is script and interval-secs, and both are required. Only download takes a progress hook, and interval-secs must be more than 0.

None of these hooks belongs to an artifact, so a script cannot use @self. Name the artifact that carries the script: @<artifact>/<path>. A bookend also cannot use an absolute path. Lint reports these refusals as ATC-E0775. A one-label hook other than pre, post, gc-enter or gc-exit is an error (ATC-E0553). A label that names no state or no phase is ATC-E0775, and a second hook at one cell is ATC-E0564.

For a <state> { <phase> { … } } hook, the @<artifact> must be in the same push or active in the channel. If it is in neither, atc ota push refuses the push at preflight.

Hooks in a published delivery

atc channel deliver --lifecycle-config atc.kdl reads the artifact hooks blocks and the top-level hooks block of the file. The hooks go into the signed delivery manifest, as they do from an atc.toml. A path that ends in .kdl is read as atc.kdl. Any other path is read as atc.toml. A workload hooks block is refused, as it is for a push.

Modules and use

use <module> makes the node names that a module defines available in this file. Write it before or after the nodes that use it. atc looks for the module definition in this order:

1. .atc/modules/<module>/module.kdl in the repository
2. the modules built into atc: systemd, podman and nerdctl

atc does not fetch a module definition from an atc server yet. To use a module that is not built in, copy its module.kdl into .atc/modules/.

To see which definition won and why, run atc config lint --explain-modules.

use takes two optional parts:

• version="<constraint>" refuses a definition whose version does not match the constraint (ATC-E0751).
• A rename <old>=<new> child gives a kind a different node name in this file. Use it when two modules define the same kind name, because that is an error without a rename (ATC-E0580).

Workloads

A workload is a service on the device that runs from an artifact. from names that artifact, and it is required on every workload (ATC-E0582). Two workloads can run from one artifact.

use systemd

bundle stack {
    source "./deploy"
}

systemd charger {
    from stack
    unit "charger.service"
}

systemd telemetry {
    from stack
    unit "telemetry.service"
}

The push takes this file. The device must have the systemd module. Modules today, section 5 describes what the device does with a workload.

The built-in workload kinds take these children:

Every kind also accepts hooks, which the push refuses today.

A kind from a module in .atc/modules/ takes the fields that its module.kdl declares. Lint cannot check those fields, so it prints the warning ATC-E0754.

Artifact kinds from a module

A module can also define an artifact kind. The artifact takes source, platform and hooks. The module’s production recipe decides how atc makes its bytes, and which of those children the artifact needs:

• walk-directory walks the source directory, as a bundle does. platform is an error. The push takes it.
• oci-pull pulls the source image, as an oci does. platform is required. The push takes it.
• oci-pull-erofs-convert is the recipe of the built-in podman kind. platform is required. The push refuses it today. See What the push refuses today.

This file lints, and the push refuses it because of the podman artifact:

use podman version="^1"

podman web-img {
    source "ghcr.io/acme/web:1.4"
    platform "linux/arm64"
}

container web {
    from web-img
    name "acme-web"
    restart on-activate
    health {
        cmd "curl -fsS http://localhost:8080/healthz"
        timeout-secs 10
    }
}

The podman and nerdctl modules both define container, so a file that uses both renames one of them. This file lints, and the push refuses it for the same reason:

use podman
use nerdctl {
    rename container=nctr
}

podman web-img {
    source "ghcr.io/acme/web:1.4"
    platform "linux/arm64"
}

container web {
    from web-img
}

nctr sidecar {
    from web-img
}

Peripherals

A peripheral node names a hardware peripheral, gives the address that its flash hooks get as ATC_PERIPHERAL_ADDRESS, and lists the bundles it takes. Write each bundle as a bare name.

bundle radio-fw {
    source "./firmware/radio"
}

peripheral radio-front {
    address "usb:001"
    radio-fw
}

peripheral radio-rear {
    address "usb:002"
    radio-fw
}

The push takes this file. Only a bundle can back a peripheral.

A peripheral can declare its flash hooks in a hooks block. It is the [[peripheral]] lifecycle.hook.* table of an atc.toml, with the same <state> { <phase> { script …; timeout-secs … } } shape as an artifact’s hooks block.

bundle radio-fw {
    source "./firmware/radio"
}

peripheral radio-front {
    address "usb:001"
    radio-fw
    hooks {
        check { enter { script "/usr/local/sbin/probe.sh" } }
        install { enter { script "/usr/local/sbin/flash.sh"; timeout-secs 120 } }
        rollback { enter { script "/usr/local/sbin/restore.sh" } }
    }
}

The push takes this file. Each delivery carries the declarations of the peripherals that subscribe to one of its bundles: the address, the hooks, and the declared order. atc ota push sends them on the push wire, and atc channel deliver --lifecycle-config puts them in the signed manifest when one of them declares a hook. The device flashes the delivery’s peripherals with those hooks, in the order this file declares them.

A hook at a cell the peripheral ladder does not run is an error (ATC-E0775). The table in atc-toml.md (“Which states a [[peripheral]] may declare”) lists the cells. Two hooks at one cell are ATC-E0564.

An atcd older than this release ignores the declarations and flashes with the [[peripheral]] blocks of its own --config file. So atc ota push refuses to send a peripheral hook to such a daemon. Update atcd first with atc bootstrap atcd upgrade. A pulling device of an older release also ignores them, and the publish does not refuse.

daemon and bootstrap

Use these blocks when atcd on the device is not where the defaults put it.

daemon {
    socket-path "/run/robot/atcd.sock"
    remote-path "/opt/robot/bin/atcd"
}

bootstrap {
    auto #false
}

bundle app {
    source "./dist"
}

The push takes this file.

Each node needs a block. A bare daemon or bootstrap with no { … } is an error (ATC-E0553), and an empty block keeps every default. Any other child is an error (ATC-E0544). A child you leave out takes the same default as in atc.toml.

push

atc ota push reads this block for its defaults.

push {
    target "team/staging"
    autoswitch #true
}

bootstrap {
    confirm #false
}

bundle app {
    source "./dist"
}

The push takes this file.

Each part of target is one path component of letters, digits, - and _. The namespace agency is reserved for atcd’s own updates. Lint reports a target that breaks these rules as ATC-E0790. The block needs { … } (ATC-E0553), and any other child is an error (ATC-E0544).

The push block is not a channel node. A channel node lists the members of a composition, which the push refuses today.

server

atc ota publish and the atc channel commands read this block to find the atc server. atc auth login reads only ca-bundle. It takes its server from --server or from the active login profile, not from this block. It is the same as the [server] table of atc.toml, for these three keys only.

server {
    url "https://atc.example.com"
    org "7d0c7f5e-1f7e-4c55-9d43-3f0d6a1b2c3d"
    ca-bundle "/etc/atc/private-ca.pem"
}

bundle app {
    source "./dist"
}

The push takes this file.

The block needs { … } (ATC-E0553), and an empty block keeps every default. Any other child is an error (ATC-E0544). The other keys of the [server] table configure atcd on the device, not the repository, so atc.kdl has no spelling for them. The one exception is credential_store, which nothing reads, so it has no spelling either. atc bootstrap atcd install --server sets the server for a device.

ownership

atc ota publish --deliver reads this block. One manifest owns the delivery config of a server channel: its hooks, workloads, peripherals, state_dirs and member roles. Another manifest that ships artifacts into that channel defers to the owner. It is the same as the [ownership] table of atc.toml.

ownership {
    id "robot-base"
    owns "stable" "beta"
    defers {
        canary "robot-core"
    }
}

bundle app {
    source "./dist"
}

The push takes this file.

The channels are server channels, the --channel of atc ota publish. They are not the push { target } channel. An id and an owner id follow the rules of an artifact name. The load and atc config lint apply the [ownership] rules of atc.toml, with the same codes.

The block needs { … } (ATC-E0553), and any other child is an error (ATC-E0544). A file with no ownership block makes no claim. Release through atc-server says what the server does with the claim.

What the push refuses today

These constructs lint with no error. The push refuses the whole file and names the construct, so that nothing in the file is dropped without a message. Lint gives the same forecast as ATC-E0597.

A channel composition with delivery hooks:

bundle app {
    source "./app"
}

channel software {
    app
    os-config
    hooks {
        pre { script "./scripts/announce.sh" }
        post { script "./scripts/verify.sh" }
    }
}

A workload hook can attach only to activate and deactivate. Its script is an absolute path on the device, or @<artifact>/<path> for an artifact of the same file. @self is an error, because a workload has no tree of its own:

use systemd

bundle stack {
    source "./deploy"
}

systemd charger {
    from stack
    unit "charger.service"
    hooks {
        deactivate {
            enter { script "@stack/drain.sh" }
        }
    }
}

What has no KDL spelling yet

These settings exist in atc.toml and have no atc.kdl node. They are device settings: each device keeps them in its atcd store, and atcd settings set or atc bootstrap atcd install sets them (issue #6337). A push from atc.kdl uses the default value of each. atc config migrate lists the ones an atc.toml sets.

• every [daemon] key other than socket_path and remote_path
• the [storage] table
• every [server] key other than url, org and ca_bundle_path

Private CA trust

By default the control-plane HTTPS of atcd and atc trusts the public web PKI roots only. A self-hosted, air-gapped, or lab control plane usually has a certificate that a private CA signed. Name that CA, or the device cannot enroll.

The reverse tunnel of atcd reads the same field and starts from the same base trust set. It reads no system certificate store. See Which roots each half adds to.

The symptom

The device never enrolls, and the atcd unit stays green. Read the journal on the device:

journalctl -u atcd -n 50

A device that is not enrolled yet writes one line per poll cycle:

enrollment check failed: send enroll request. TLS or connect failure: check [server] ca_bundle_path

An enrolled device writes the check-in form of the same failure:

check-in errored: server: checkin send: error sending request for url (...). TLS or connect failure: check [server] ca_bundle_path

The atc CLI reports the same class of failure, and names both of its sources:

. TLS or connect failure: check [server] ca_bundle_path or SSL_CERT_FILE

The hint fires on every connect failure. A refused port and a wrong port get it too. A request that reached the server never gets it, so a 404 or a 500 is a different problem.

Install a device with the CA

atc bootstrap atcd install uploads the bundle and writes the config field:

atc bootstrap atcd install deploy@device \
  --server https://atc.example.com \
  --enrollment-token-file ./enroll.token \
  --ca-bundle ./ca.pem

--ca-bundle names a PEM file on your controller, not on the device. atc reads and parses that file before it opens the SSH connection. So a bundle atc cannot use fails before the device is touched:

error: ATC-E0419: bootstrap preflight on deploy@device failed: --ca-bundle ./ca.pem: No such file or directory (os error 2)

A file that exists but holds no certificate takes the same refusal, and the message then reads --ca-bundle the trust anchor bundle ./ca.pem holds no certificates.

The device copy lands at /etc/atc/ca.pem, owned by root, mode 0644. A certificate is public, and every account that verifies the control plane must read it. Bootstrap then writes these settings to atcd’s store. The device keeps no config file for them:

server.url = https://atc.example.com
server.control_mode = server_only
server.enrollment_token_path = /etc/atc/enroll.token
server.ca_bundle_path = /etc/atc/ca.pem

Read them on the device with atcd settings ls, as the account that runs atcd.

--ca-bundle needs --server. A trust anchor with no control plane is inert, so atc refuses the flag alone.

Add the CA to a device that is already installed

Copy the PEM to the device yourself, then set the path in atcd’s store. Run this on the device as the account that runs atcd (atc by default):

sudo -u atc atcd settings set server.ca_bundle_path=/etc/atc/ca.pem

An enrolled device refuses atc ota settings set from a laptop, because it accepts changes only from its server. atcd reads its settings at start, so restart the unit to apply the new value:

systemctl restart atcd

Give an absolute path. atcd settings set refuses a relative one and changes nothing.

A bundle atcd cannot use stops the pull cycle. Unreadable, unparseable, and holding no certificate all count. There is no fall back to the public roots. A silent fall back would turn a typo into a TLS failure much later:

certificate trust setup failed ([server] ca_bundle_path): the trust anchor bundle /etc/atc/ca.pem holds no certificates. The pull cycle stops until the bundle is fixed

atcd re-reads the file once per poll cycle, so this state recovers with no restart. Replace the bundle with write-then-rename. A partial file aborts one cycle, and the next cycle self-heals.

The atc CLI

atc composes two sources, in this order, on top of the public roots:

1. [server] ca_bundle_path, from the atc.toml the command resolved
2. $SSL_CERT_FILE

Both sources add anchors. Neither removes a public root, and neither replaces the other. A source you state but atc cannot use is an error, and the message names which of the two to fix:

error: control plane setting: [server] ca_bundle_path: cannot read the trust anchor bundle /home/you/ca.pem

Which roots each half adds to

[server] ca_bundle_path is additive everywhere. It never removes a root. Both halves of atcd add it to the same base set.

atc oci is a third case and is not in that table. Its base is the bundled webpki roots too, but the field does not reach it. The registry client reads $SSL_CERT_FILE alone. The component table below records that.

Two consequences follow.

• A CA in the device image does nothing for the tunnel. The tunnel reads no system store. Name the CA of a private rendezvous in [server] ca_bundle_path. Before August 2026 the tunnel read the system store instead. A device on an older build still behaves the old way.
• Name your own CA, and nothing else. An anchor is an issuer. Any CA in the bundle can sign a certificate for the rendezvous host name, and the device accepts it. A bundle holding a copy of a whole system store hands that power to every CA in it.

What each component reads

Four consequences follow, and each one catches operators who expect one rule everywhere:

• atcd reads no environment variable. A stray SSL_CERT_FILE in a systemd unit must not change the trust set of a whole fleet. That is a failure nobody configured. The device honors the config field alone.
• A private registry behind the same CA needs $SSL_CERT_FILE. The registry client reads that variable alone, so a configured control plane does not cover atc oci. Go’s crypto/x509 reads the same variable, so skopeo gets the same trust from the same value.
• An atc.kdl manifest names the bundle in its server block. server { ca-bundle "/abs/path.pem" } sets the same field as [server] ca_bundle_path. See the server block. An atc.kdl with no server block sets no bundle, so $SSL_CERT_FILE is then the only source atc has.
• atc-server reads no atc.toml at all. Its configuration is environment variables. Set $SSL_CERT_FILE for the process when the identity provider sits behind your CA. Without it the server cannot fetch the signing keys, and then no token validates.

Related

• Onboarding walks the first update end to end.
• Verifying the binary covers a different trust question: whether the atc binary itself is genuine.

Remote access without a VPN

atcd carries its own reverse tunnel. The robot dials out to your control plane over wss://, and you open a shell on it from wherever you are. There is no mesh VPN, no second vendor’s agent, and nothing listening for a connection on the device.

It is off until you configure it, and configuring it is one TOML table. Delete the table and the tunnel is gone. Any other access you installed, such as plain SSH on a routable address, is untouched, and nothing else about the device changes.

That matters if your customers will not accept a VPN mesh on their network. The usual objection is that nobody can audit who connected when. This path answers it. Every session through your own atc-server gets a record of the principal that opened it, the device it reached, and whether the device end answered.

Read the limits on that. The server records the PRINCIPAL, so two API keys for one principal are indistinguishable. Per-person attribution needs a human identity, or a separate principal for each person. A failed audit write leaves a gap and does not close the session. A standalone wstunnel rendezvous writes no atc-server record at all.

It does NOT make session contents inspectable. The payload is SSH inside TLS, so a packet inspector still cannot read the shell. What it gets instead is one ordinary HTTPS flow to one host you name. It can allow, block, or rate limit that flow by destination, port, and SNI. A mesh VPN gives it neither.

Read what the rendezvous side needs before you plan around this. The device half is ready. The server half runs the rendezvous on one task, and a public deployment without the CloudFront lockdown still 404s the operator route.

How it reaches a robot behind NAT

robot ---- dial out, wss:// ----> atc-server <---- wss:// ---- your laptop
  |                              (pairs the two)                    |
  +--- sshd on 127.0.0.1:22                             ssh --------+

The device opens the connection. The control plane holds it, then hands it to an operator who asks for that device. The connection is a WebSocket over TLS. A firewall that does not terminate TLS sees an ordinary HTTPS flow, and it can classify that flow by destination, port, and SNI. It cannot read the WebSocket framing inside.

An egress-only warehouse firewall can therefore allow it, and you need no port forward. It is not automatic. A proxy that terminates TLS, or a policy that refuses a WebSocket upgrade, still blocks the tunnel.

The device needs no inbound rule. A relayed connection goes to the device’s own sshd, and the operator’s own ssh speaks to it. Today one command opens the tunnel: atc fleet device shell. The other device-touching subcommands still need a route you can already reach.

Before you start

The device must already hold enrollment state. The tunnel authenticates with the device credential that enrollment persisted, and it reads that credential from the store, not from the config. A device that never enrolled logs device is not enrolled yet and retries until it does.

A [server] table is what enrolls a device automatically, and it is also where ca_bundle_path lives. It is not itself the credential. A device enrolled out of band can park with no [server] table. A device with a [server] table that is not yet enrolled cannot.

Warning: nothing enforces a distinct listen address

No part of atc, atcd, or atc-server compares one device’s listen value against another one. There is no fleet-wide register of addresses, and no check runs at any point. Read which of the two cases below you are in before you write the field.

Against atc-server, listen is ignored. That rendezvous never reads the value. It pairs an operator with a device by the UUID that the presented credential resolves to. Two devices may carry the same listen with no consequence. atcd still requires the field, and it must parse as an IP address and port such as 0.0.0.0:2222. A host name is refused.

Against a standalone wstunnel rendezvous, give every device a distinct bind address. That is a different deployment, and not a drop-in one. atcd dials the fixed path /v1/devices/tunnel/events, so the rendezvous must accept it. atc fleet device shell does not work against it either, because that command only speaks to atc-server’s operator route. An operator points ssh at the address the rendezvous published instead. Within that deployment, the collision below is live.

wstunnel keys its reverse listeners by the whole bind address. It REUSES the existing listener when a second device claims the same one, and it does not refuse that device. Two devices with one listen value then share one listener.

An operator who connects to that address reaches whichever device takes the connection first, and neither side reports an error. The operator may then get a shell prompt on the wrong robot. A host key already recorded for the intended robot can refuse the connection first. That is a warning, not a guarantee.

Assign the addresses from a table your fleet owns, and do not copy the example value below onto a second device.

Turn it on

Set the three tunnel keys in atcd’s store. Run this on the device as the account that runs atcd (atc by default):

sudo -u atc atcd settings set \
  daemon.tunnel.server=wss://tunnel.internal.example.com:443 \
  daemon.tunnel.listen=0.0.0.0:2222 \
  daemon.tunnel.forward_to=127.0.0.1:22

• daemon.tunnel.server is the rendezvous the device dials. It must be wss://. Plaintext ws:// is refused.
• daemon.tunnel.listen is what the device ASKS the rendezvous to publish. atc-server ignores it, but the key is required. Read the warning above.
• daemon.tunnel.forward_to is the device-local address a relayed connection is handed to. 127.0.0.1:22 is the default.

A device that accepts direct control takes the same keys from a laptop with atc ota settings set <target> …. An enrolled device refuses that, because it accepts changes only from its server.

Restart the daemon:

sudo systemctl restart atcd

Confirm that the daemon read the table and built a tunnel client:

journalctl -u atcd -b | grep 'atcd tunnel'

A configured device logs one line each time it builds a client:

atcd tunnel: parking on wss://tunnel.internal.example.com:443 to publish 0.0.0.0:2222 -> 127.0.0.1:22

That line comes BEFORE the first dial, so it does not prove that the device reached the rendezvous. A device that cannot reach it prints the same line and then logs atcd tunnel: dial to the rendezvous failed.

The only end-to-end proof is a successful atc fleet device shell against that device. The server-side atc_server_tunnel_park_total metric carries no device or org label, so it tells you that SOME device parked and nothing about yours.

The daemon checks the values at startup, and it names the field that is wrong. It checks that server is a URL and that its scheme is wss. It checks that listen is an IP address and port. forward_to may be a host name and port as well, so localhost:22 is accepted there and refused in listen.

atcd tunnel: disabled — [daemon.tunnel] server "https://rv.example.com" uses scheme "https"; expected wss://

A bad string VALUE leaves the daemon running. It still ships updates, and only the tunnel is off.

Warning: a table that will not DECODE is different, and it stops the daemon. The decode runs before any of the checks above. It fails on a missing server or listen, and on a wrong TOML type such as listen = 2222. A misspelled server or listen is the same failure, because the required key is then absent.

An unknown key ALONGSIDE the required ones is the third case. It warns, the key is not applied, and the daemon starts.

Open a shell

atc fleet device shell <device-uuid>

Add --user when the account on the device is not your local login name. Put a one-shot command after --:

atc fleet device shell 4f1c0e2a-... --user deploy -- uptime

Every word after -- reaches the device as one argument. atc quotes each one, so a space, a brace, a quote, a glob, or a $ inside an argument survives:

atc fleet device shell 4f1c0e2a-... -- jq '{a,b}' /tmp/x.json

--as-ssh-user follows the same rule. It sent the words bare until issue #5414, and the shell on the device re-split them. The command above then ran as jq {a,b} /tmp/x.json.

To run a pipeline, a redirect, or a ~, pass the shell program as ONE argument:

atc fleet device shell 4f1c0e2a-... -- sh -c 'journalctl -u atcd | tail -20'

atc opens the tunnel, publishes it on a loopback port, then runs the system ssh at that port. ssh does the SSH. It records and checks the host key under the device UUID rather than the loopback port. The key therefore survives from session to session instead of churning.

Warning: this is trust on first use. atc passes StrictHostKeyChecking=accept-new, so the FIRST connection to a device UUID accepts whatever key answers. Only a LATER change of key fails the check. If you must be sure of the identity on first contact, put the expected host key in your known_hosts under the device UUID before you connect.

If the device is offline, or if it has no daemon.tunnel.server setting, the command fails before ssh starts. It reports ATC-E0461.

Turn it off

A re-run of atc bootstrap atcd install keeps the tunnel settings. An install writes only the keys the operator states, and --server states only the server.* keys.

To remove the tunnel, unset the three keys and restart atcd:

sudo -u atc atcd settings unset daemon.tunnel.server daemon.tunnel.listen daemon.tunnel.forward_to
sudo systemctl restart atcd

The daemon then dials nothing, and it behaves as it did before the keys existed. There is no package to remove and no service to disable.

What the device trusts

The DEVICE authenticates the rendezvous certificate against the bundled webpki roots, plus whatever [server] ca_bundle_path names. It reads no system certificate store. See Private CA trust.

The OPERATOR half reads the same bundle. atc fleet device shell builds its TLS configuration from atc_cmd_server::trust::rustls_config_for, which carries the bundled webpki roots, plus [server] ca_bundle_path, plus $SSL_CERT_FILE. It reads no system store either.

Warning: check which atc build the operator runs. This half changed on 23 August 2026, in #3548. An atc built before that dialled through tokio-tungstenite’s built-in connector, which trusts the bundled webpki roots alone. Against such a build a rendezvous behind your own CA accepts every device and refuses every operator. Upgrade atc, or give the operator-facing name a publicly trusted certificate.

The two halves still dial different URLs. The device dials [daemon.tunnel] server. The operator dials the control-plane base URL that the atc CLI is configured with. They may be different names, and they must reach the same atc-server process, because the pairing lives in that process’s memory.

So one private CA can now serve both halves. Name it in [server] ca_bundle_path on the device, and in the [server] table the operator’s atc reads. The two files are different files on different machines, and both need the anchor.

You cannot turn verification off. wss:// is required, and atcd refuses the tls_verify_certificate escape that wstunnel offers.

This trust base changed in August 2026. An older build read the device image’s system certificate store instead. Confirm which build a device runs before you plan a trust root around it.

Who may open a shell, and what is recorded

The route always requires an authenticated admin actor. What counts as one depends on a server-side flag, so check which of these your atc-server runs.

With ATC_ORG_RBAC_ENABLED=true, the caller must belong to the org named in the path AND hold admin rank there. A platform admin passes for any org.

Warning: the flag is OFF by default. The check then falls back to a plain server-admin check, and a server admin can open a shell on any device in any org. Turn org RBAC on before you tell a customer that the tunnel is scoped to their org.

The device-membership check runs either way. A device that belongs to a different org than the path names answers 404, byte-identical to the answer for a UUID that exists nowhere.

atc-server writes an admin.device.tunnel.open audit record for a granted session AND for a refused one, once the caller is authenticated. A missing or invalid credential is refused before the audit request exists, so it leaves no tunnel record at all. A failed audit write is logged and does not close an already-open session.

Do not alert on refusals alone. One refused record covers several events, and this list is not exhaustive:

• the UUID exists nowhere
• the device belongs to another org
• the caller does not hold admin rank
• the device is offline
• the device answered, but its forward_to service sent nothing
• the device dropped between the reachability check and the handoff

Only the first three look anything like an attack. An operator who retries an unplugged robot, or a robot whose sshd is stopped, produces a run of refusals that reads exactly like probing. Connectivity alone does not separate these, so read the refusal message before you treat a run as an attack.

The server also counts every park on the atc_server_tunnel_park_total metric. It carries one label, displaced, with three values. Read them before you build an alert on it.

• none: the ordinary park, and the denominator. A fleet with no tunnel-enabled device reports zero here, which is not the same as no displacements.
• closed: this park replaced a registration whose connection had already gone away. A device on a bad network produces these constantly. Do not alert on them.
• live: this park pushed out a registration that still looked healthy. One per robot recovering from a silent connection drop is normal. A run of them for one device means two holders of that device’s credential, such as a disk image cloned after enrollment. Rotate the device credential.

A granted record proves that the device end answered. It does not prove that an sshd answered. Whoever holds the device credential controls that byte stream. Nothing the server can inspect separates a real sshd from a client that writes one frame. Only a host key you verified elsewhere and pinned first closes that gap. The default trust on first use protects later sessions and not the first one.

Dependencies

atcd and the atc CLI contain no Tailscale code and need no Tailscale daemon. On the device the tunnel needs outbound TLS to your atc-server and nothing else. Remote access is a config block you add and delete on its own schedule.

Tailscale does appear elsewhere in this repository. atc-onboard, the tool for the hardware test farm, uses it to reach lab boards. The atc-server terraform also defaults ingress_mode to tailscale, which puts the managed load balancer on a private subnet. That is a hosting choice for our own deployment, not a requirement of the tunnel.

What the rendezvous side needs

The device half is ready. The server half carries one live limit and one shape you have to deploy, both recorded in cloud/atc-server/terraform/tunnel.tf. Read them before you promise this path to anyone.

Both tunnel paths must reach the rendezvous service, and it runs one task. The pairing registry lives in process memory, and a parked device holds a live channel end. So a device parks on whichever task the load balancer picked, and an operator pairs only if their request lands on that same task. With two tasks that is a coin flip, and the operator sees a pairing timeout that looks exactly like an offline robot.

tunnel.tf closes that by ROUTING rather than by pinning: it adds an ECS service at desired_count = 1, its own target groups, and listener rules that send /v1/devices/tunnel/events and /v1/admin/orgs/*/devices/*/tunnel to it. serve keeps both routes in its binary and never receives them, so serve may scale freely. Do not pin serve for the tunnel’s sake.

Raise the tunnel’s capacity with tunnel_task_cpu, never by adding a task. A second rendezvous task returns the failure the routing removed.

The costs are real and worth stating. One task is one point of failure. Its rollout is stop-then-start, so a deploy is a tunnel gap of the target group’s 30-second drain plus task start plus two health checks. Devices redial after it. Uninterrupted tunnel availability needs inter-replica forwarding, which issue #3984 carries.

Unprotected public ingress 404s the operator route. The operator half lives under /v1/admin/*. The load balancer installs a priority-10 rule that answers 404 for that whole prefix. That rule is active for ingress_mode = "public" and for "protected_public" alike, whenever the CloudFront origin-verify lockdown is OFF. Our prod is one such deployment. The atc CLI then reports the device as not found, and atc-server logs nothing, because the request never arrived.

The tunnel rule follows that decision rather than overriding it: where the priority-10 block is active it carries the device path only, deliberately, so the ops block gains no unused hole. A deployment that wants the operator half on unprotected public ingress has to decide to open the prefix.

With the lockdown ON, the block has count = 0, and the tunnel rule carries both paths plus the origin-verify header the lockdown requires. That is our staging shape, and the operator half works there. The lockdown, not the ingress mode alone, is what decides it.

The DEVICE half is unaffected. It parks on /v1/devices/tunnel/events, which no block rule matches.

Until a shared pairing registry and that forward rule land, treat this as an internal-testing path rather than a fleet-wide one.

What this path does not do yet

• It carries ssh only. atc ota push over the tunnel is not wired up. A push still needs a route you can already reach.
• One session per tunnel. A second atc fleet device shell opens its own pairing.
• A park lasts 25 seconds, then the device redials. An operator who arrives during the gap waits up to 3 seconds.

Lifecycle hook cutover: the flip moved earlier

The current delivery pointer used to flip in the middle of the Commit state, between commit.enter and commit.exit. It now flips inside Activate, before activate.exit. Two new states, Activate and Deactivate, sit between Install and Commit.

Nothing you already wrote stops parsing and nothing stops running. This is a hard cutover with no compatibility shim and no deprecation window, because none is needed. Every declaration that was valid before is still valid and every hook still fires. What changed is when one of them fires relative to the moment the new version goes live, and how long two others wait for it.

This page answers one question. Does your hook need to move?

What the order is now

Before:

check.* → download.* → install.* → commit.enter → [FLIP] → commit.exit

Now:

check.* → download.* → install.*
  → activate.enter                    (new version asks: may I go live?)
      → deactivate.enter              (old version stands down)
      → deactivate.exit
      → [FLIP]
  → activate.exit                     (new version confirms: I am live)
  → commit.enter
  → commit.exit

Deactivate runs nested inside Activate, so every step that can fail happens before the one destructive step. A veto at activate.enter has torn nothing down and owes no compensation. The transition table that decides this lives in crates/atc-lifecycle/src/state/fsm.rs; the driver that walks it is crates/atcd-applier/src/driver/delivery_flow.rs.

commit.enter is the one gate that swapped sides. It used to run before the flip and now runs after it. activate.exit is a new post-flip gate, and commit.exit was post-flip before this change and still is.

The observable symptom

Open your hook script. You need only one fact about it.

Does the script reach the live delivery pointer? Either directly, by naming a path under the channel’s delivery/current, or indirectly, by restarting, reloading or querying something whose own configuration resolves through it. A systemctl restart, an atc ota status, a unit file whose ExecStart points into the active delivery dir: all of those reach it.

If the answer is no, stop reading. Your hook is unaffected. Everything a hook gets in ATC_HOOK_CONTEXT is stable across the flip: version.new_sha, version.old_sha, self_root.new, self_root.old and artifacts_dir hold the same values on both sides of it. A script built only from those fields cannot observe the flip at all, and so cannot have been broken by moving it.

If the answer is yes, read the section for the gate you declared.

commit.enter: check this one first

commit.enter moved from before the flip to after it. It is the only gate whose meaning changed, and it changed silently: the hook still runs, still at the same point in the printed sequence, and still exits 0.

The difference is what it reads.

# in a commit.enter hook, on the same device, same push
atcd get-path
# before this change: the OLD delivery dir
# now:                the NEW delivery dir

atcd get-path prints the active delivery directory, and atcd get-path -a <name> prints one artifact’s active payload directory. Use it rather than building the path yourself: the directory is keyed by delivery id, and the v5 layout already moved it once, which broke every hand-written copy. With no flag the verb reads the current pointer. With --delivery-id it lists the delivery directories of the default channel instead. Neither path opens the database or needs a running daemon, so both are safe to call from inside a hook.

If your hook must read one SPECIFIC delivery, name it. atcd get-path --delivery-id <ID> reports that delivery rather than whichever one current points at, and --delivery-id <ID> -a <name> reports one of its members. A delivery bookend gets the id from ATC_DELIVERY_CONTEXT. A per-artifact hook does not receive one.

A delivery gets its directory when it is activated. On a fresh push that happens at the flip, so delivery.pre, activate.enter, deactivate.enter and deactivate.exit cannot look the INCOMING delivery up: it has no directory yet. activate.exit, commit.enter, commit.exit and delivery.post run after the flip and can. On atc ota switch the target was committed by an earlier push, so every phase can look it up.

If your commit.enter hook drains connections, flushes a queue, snapshots state, or stops a service belonging to the version being replaced, it is now doing that after the replacement went live. Move it. deactivate.enter and deactivate.exit exist for exactly that work, and they run the outgoing version’s own declaration rather than the incoming one (see Whose declaration runs).

If your commit.enter hook only inspects the incoming artifact on disk, or only reads ATC_HOOK_CONTEXT, leave it alone.

commit.exit: correct, but late

A commit.exit hook was post-flip before and is post-flip now. It still sees current pointing at the new version. It is not broken and you do not have to touch it.

What changed is the gap in front of it. The window between the flip and commit.exit used to hold only the daemon’s own post-flip work. It now also holds activate.exit, the delivery-wide commit.enter stage barrier, and one member-scoped commit.enter gate per artifact. Each of those can run an operator script with its own timeout_secs, so the new version can be live for a while before your commit.exit hook gets to run.

That matters if the hook is what makes the new version usable. A device whose current points at v2 while v2’s service is still stopped is running nothing, and the longer that gap the worse it reads in the field.

Before, and still working:

[bundle.lifecycle.hook.commit.exit]
script = "@self/scripts/restart_service.sh"
timeout_secs = 60

Better now:

# activate.exit is the FIRST gate after the flip. Bring the new version into
# service here, and the live-but-stopped window is as short as it can be.
[bundle.lifecycle.hook.activate.exit]
script = "@self/scripts/restart_service.sh"
timeout_secs = 60

Leave it at commit.exit if it reports, notifies, validates or marks. A hook that posts to Slack, drops a marker file, or curls a health endpoint does not care that it runs a few seconds later.

Both cells are veto-capable, so moving the script does not weaken the gate. A non-zero exit at activate.exit unwinds the delivery and reverts the pointer exactly as one at commit.exit does.

Where each kind of hook belongs now

The last two are new slots, not relocations. See atc-toml.md for their scope rules.

Does your hook claim a name the device holds only once, a systemd unit name or a TCP port? Read A name shared across versions first. The table above puts the start at activate.exit to shorten the live-but-stopped window. For a shared name that row is also a correctness rule, and the start must be a restart.

Do you move a @self hook to check.*, download.* or install.enter? There it runs the ACTIVE version’s script, not the incoming one, and a missing script skips the hook. See Which version’s script @self runs. A @self hook that you move OUT of install.enter to install.exit starts to run the incoming version’s script.

Does the hook run an absolute script that the artifact itself installs, for example a file a .deb puts under /usr/lib? Then a move to an earlier cell can run it before the package exists. On the first delivery to a device that fails the hook with ENOENT. Set allow_missing_initial = true on that hook. Read A hook script that the artifact installs.

Whose declaration runs

deactivate.* is the one state whose hooks come from the version being torn down rather than the delivery arriving. A push reads them back from that version’s own .atc/lifecycle.pb sidecar; a pulled delivery reads the signed declaration the poller retained when it installed that version. That has a consequence worth stating plainly, on either plane: a deactivate.* hook you add today does not run on today’s delivery. It runs on the next one, when the version you are shipping now becomes the outgoing version.

Two corollaries:

• A first deploy traverses Deactivate and dispatches nothing. There is no outgoing version to tear down.
• A stage-only push (one that stages every artifact but flips nothing, so no --switch and no autoswitch) and a re-push of the already-active version also dispatch nothing there. Neither replaces a running workload, so tearing one down would leave the device running nothing and report success. Neither activates anything either, so neither runs activate.*.

uninstall.* follows the same rule for the same reason: the sweep reads it from the sidecar of the version it is about to delete.

atc ota switch and atcd ctl switch

Both now run six gates where they ran two: activate.enter, deactivate.enter, deactivate.exit, the flip, activate.exit, commit.enter, commit.exit. If you parse the streamed progress, expect three times the frames.

deactivate.* was absent from this path until #3698. The reason was real. The switch plane had none of the machinery Deactivate depends on. A vetoed switch would then leave the outgoing version torn down, with nothing able to restore it. It has that machinery now: a durable teardown record, a boot payer, and the compensation a refusal owes. So the two cells run, from the OUTGOING version’s declaration.

Read your deactivate.* declarations before you upgrade. A teardown reads the outgoing version’s declaration. So the scripts that start running are the ones already on your fleet. A refused teardown vetoes the switch and leaves the device where it was.

Devices that crashed under the old daemon

Nothing to run. deliveries.phase is free text, and the recovery reader accepts both the old Commit spelling and the new Activate one, so a device that crashed mid-flip under a pre-cutover daemon recovers correctly under a post-cutover one. There is no schema migration and no backfill step. The dual acceptance is the upgrade path (crates/atcd-core/src/crash_position.rs::classify_crash_position).

A crash between deactivate.exit and the flip is now a rollback case: the daemon reactivates the outgoing version on the next boot rather than leaving the device stranded.

A word that means two different things

atc ota ls prints a push status column whose values include activating and committing. Those are not the lifecycle states this page is about. They are a separate vocabulary in crates/atc-store/src/pushes.rs describing where a push sits in the transfer pipeline, and they predate these states. If you are matching on one, be sure which one you mean.

Checklist

1. rg -l 'commit\.(enter|exit)' path/to/your/atc.toml. No hits means nothing to do.
2. For each hit, open the script. Does it reach delivery/current, directly or through a service it restarts? No means leave it.
3. commit.enter that touches the outgoing version, moves to deactivate.*.
4. commit.exit that brings the new version into service, moves to activate.exit.
5. Everything else stays.
6. Push to one device and read the hook rows back with atc ota status --json before you roll it out. The sequence you should see is in What the order is now.

Modules today: what a repo can actually do

Every claim below is cited to a file and line on dev. Where something does not work, this says so; the gap list at the end is the point of the document, not an appendix to it.

The short version: a repo can ship one executable per module. atcd now drives four of the nine verbs against it. workload start runs after a successful delivery. workload health runs after a successful start the applier session itself ran, and after no other start (§4). The deactivation drain runs workload stop, and workload remove for a module that leaves the device (§4).

The device also reads the delivered module.kdl. It spawns only the verbs that file declares. It applies the per-verb timeout the file asks for (§2).

Workloads parse, bridge, cross the wire, and persist per delivery, and the trigger spawns one workload start per DECLARED workload (§5). A stock atc can now get one to start. The workload kinds it knows are systemd (from the systemd module) and container (from podman and nerdctl). Those are MODULE names, and podman is also an artifact kind, not a workload kind. A workload of any other kind now pushes too. Its declared fields are read against its own module’s schema and sent as a fields_json_v2 document (§5).

A workload of a known kind names a built-in module, and no built-in module is ever in a push’s changeset — so the trigger cannot see it. The commit derives a lifecycle for it instead, keyed by a version made from the declaration set, and the drain runs the start out of the image’s own copy of the module (§5). It needs the image to really carry that module; nothing here installs one. A module-defined artifact kind now pushes for walk-directory and oci-pull (§3). oci-pull-erofs-convert has a producer too, and the bridge still refuses it, because no device consumer reads what that producer stages — so the built-in podman artifact kind is still unusable. atc ota ls, ota status, ota deliveries and atc ota df name the landed kind by its own spelling (§3). The other five verbs are still unreachable code.

A worked example that stays inside those limits lives at examples/heartbeat-module/, with an e2e test that pushes it to a real container: crates/atc-cli/tests/module_example_e2e_tc.rs.

1. Getting a module into a config at all

A module can only be declared in atc.kdl. No TOML syntax declares one, by design — docs/internal-site/src/design/2026-08-02-modules-production-readiness.md §S1b declined to add a [module] table to the TOML schema (crates/atc-config/src/module_artifact.rs:1-16).

A repo whose nearest manifest is atc.kdl is read as KDL with no setup at all. That is the shape a module repo has, so a module push is the plain command:

atc ota push atc@<device>:22 --version 0.1.0 --switch

Upward discovery stops at the nearest directory that holds atc.kdl or atc.toml. When that directory holds both, atc.kdl wins (#6337). Both files parse, so the losing file is ignored with no error. atc prints a note on stderr that names the ignored file, the file that won, and ATC_KDL_CONFIG (manifest_format::shadowed_manifest_notice). atc config lint says the same thing (ATC-E0598).

ATC_KDL_CONFIG=0 (crates/atc-config/src/parse.rs, kdl_config_enabled) restores the rule from before #6337: an atc.toml at any depth wins, and an atc.kdl is read only when no atc.toml is on the walk. It is the one-line way back for a repo that holds both files and depends on its atc.toml.

An explicit -c <file>.kdl needs no variable. The operator names the file, so nothing is shadowed, and the format follows the name. A .kdl-named file that holds TOML text no longer loads. It stops at a KDL error that names the file.

The full table lives on KDL_CONFIG_ENV in crates/atc-config/src/parse.rs; the ambient half of the decision lives in crates/atc-config/src/manifest_format.rs.

The declaration takes three fields: name on the top line, source, and an optional payload (ModuleArtifactConfig, crates/atc-config/src/module_artifact.rs:46-63). The smallest declaration is three lines:

module heartbeat {
    source "./modules/heartbeat"
}

source is required for module (kdl_schema.rs:184-186) and platform is forbidden (kdl_schema.rs:219-224). The name must be a single path component, because it becomes a directory on the device (kdl_schema.rs:160-162). payload names a directory this push walks into the package’s payload/ subtree (§2).

What the same file may NOT contain

kdl_bridge::bridge translates the rest of atc.kdl into a Config, and it refuses the whole file rather than silently dropping anything it cannot carry. Adding any of these to a module repo’s atc.kdl makes the push fail:

A hooks { … } block on an artifact is not on that list. kdl_lifecycle converts it, and PushManifest::artifact_lifecycle carries it to the push (kdl_bridge.rs:214-220). A hook that cannot convert is refused per hook as ATC-E0775, by atc config lint and by the push (push_manifest.rs:155).

A plain workload node is not on that list any more: it bridges, rides the push, and lands in delivery_workloads (§5). Only the two workload shapes above still refuse, and each refuses per declaration, so a file with three bad workloads reports all three.

A bare use <module> line is allowed but buys nothing on its own: it only makes a module’s node names available. What it buys once you use those names depends on which one — a systemd workload now pushes, an artifact of a walk-directory or oci-pull kind now pushes (§3), and a podman artifact is still caught by the last row above, because its recipe is oci-pull-erofs-convert.

Device coordinates a KDL repo can say, as of the daemon and bootstrap top-level nodes:

daemon {
    socket-path "/run/robot/atcd.sock"
    remote-path "/opt/robot/bin/atcd"
}
bootstrap {
    auto #false
}

socket-path and remote-path are the TOML [daemon] fields of the same name, and auto is [bootstrap] auto; each is derived by the bridge, and omitting a node means what omitting the TOML table means rather than a second default (crates/atc-config/src/kdl_bridge.rs, the daemon:/bootstrap: fields). remote-path is not decoration next to the socket: it also feeds the push connection target, so a device with atcd installed off /usr/local/bin/atcd needs it. Both blocks reject an unknown child, so the rest of each namespace stays reserved for later fields. Working example: crates/atc-config/tests/fixtures/kdl/valid/pushable/device-coordinates.kdl.

The rest of [daemon] — telemetry endpoints, hook-retry budgets — is still defaulted: it is daemon-side configuration, not something a producer file describes.

2. What the module package must contain

Read this section against prepare_module_package (crates/atc-config/src/module_package.rs:745). That is the production entry point. resolve_push_set reaches it through ModulePackageCache::get_or_prepare (crates/atc-app/src/ota_push_resolve.rs:102-111). build_module_manifest is not the entry point: it has no production caller, and its own comment says so (module_manifest.rs:245). It also refuses any module that declares a payload, so it cannot describe the package shape below.

prepare_module_package runs ONCE per push, before the retry loop. A producer re-run inside that loop could hand attempt two different bytes for one push.

The package’s own directory holds exactly two flat files (stage_core_files, module_package.rs:646):

modules/<name>/module.kdl
modules/<name>/<name>          # the executable, same name as the module

• A directory entry is refused (module_package.rs:669) and a third file is refused (:683). A symlink is refused by O_NOFOLLOW, which reports ELOOP (:381-385), and a socket or an unbacked device node reports ENXIO (:386-391).
• The executable must be a regular file (:675) carrying the owner execute bit (:691). Both checks are sender-side only. A wire Manifest carries no file type, so the receiver has nothing to re-derive them from (module_manifest.rs:29-35).
• A module may not be named module.kdl or payload (module_package.rs:749-772). The receiver refuses those two and exec-digest.json as well (module_protocol.rs:233-268).

The payload node

A package may carry a THIRD thing: a payload subtree. Declare it on the module node.

module heartbeat {
    source "./modules/heartbeat"
    payload "./assets"
}

payload names a repository-relative directory. prepare_module_package walks it and stages every regular file beneath it under the package’s payload/ prefix (module_package.rs:799-802). The node is valid on a module artifact only. On any other kind it is ATC-E0770 (kdl_parse_core_nodes.rs:117-133).

payload is sugar for the walk-directory producer, so it refuses what that walk refuses. The walk ships REGULAR FILES only, and it descends a directory to find them. It refuses every other entry type by that property, not by a list (walk_payload_entry, module_package.rs:516-532). The refused types are a symlink, a FIFO, a socket, a device node, and any type a future kernel adds. A symlink is ATC-E0759 and the rest are ATC-E0757. The walk also refuses a non-UTF-8 entry name (:562-566) and a root that escapes the repository.

The receiver accepts those entries. validate_module_manifest counts only the NON-payload entries against the two-entry rule, and admits any number of payload/** entries at any depth (module_protocol.rs:154-165, with payload_subpath at :130-133). It refuses an entry that names the payload directory itself (:170-179), and any entry with an empty path component (:206-214). The entries land under payload/ in the delivered version directory (§7 gap 10).

The definition

Both sides parse module.kdl. On the push side stage_core_files reads the definition through the shared bounded loader (crates/atc-config/src/module_definition.rs, MODULE_KDL_MAX_BYTES). It refuses the push locally when the definition does not parse, exceeds the cap, or declares a module other than the one pushed. The error is ConfigError::InvalidModuleDefinition (module_package.rs:699-722).

So a definition declares its verbs and a per-verb timeout (verb-timeout "workload start" 120), and it is validated before a byte moves. That property did not change when production moved off build_module_manifest. Only the site did.

The device re-reads the delivered file before every verb it spawns. read_verified_definition checks the file against the digest the delivery’s validated manifest declared, parses it, and refuses a definition that names a different module (crates/atcd-applier/src/agent_push_flow/module_verb_trigger.rs:223-267). Two declarations then decide what happens. The declared verb set gates every spawn, and an undeclared verb is skipped rather than run (may_spawn, :605-627, called at :763). The declared verb-timeout becomes that verb’s deadline (:798-805). A drained verb has no retained manifest to compare against, so it parses the delivered file unverified: DefinitionEvidence::OnDiskOnly (:292-299).

The rest of a definition still reaches no device code. A definition also declares kinds, their field schemas, their production recipes, its rollback contracts, a run-as uid, and managed paths. The device parses and validates all of them, and then reads none of them.

3. Artifact kinds and production recipes

ProductionRecipe has three variants — walk-directory, oci-pull, oci-pull-erofs-convert (crates/atc-config/src/module_definition.rs:128-133).

All three produce. Only two reach a consumer on the device.

• A recipe is a field of ArtifactKind::FromModule only (kdl_schema.rs:61-65). ModulePackage is a core kind and has no recipe field. Its payload comes from the payload node instead — sugar for the same walk-directory producer, run by prepare_module_package.

• walk-directory is performed by prepare_recipe_artifact (crates/atc-config/src/module_package.rs). It is the SAME walk the payload node runs. Same openat ladder, same staging root. One producer, two callers, not a second directory walker.

• oci-pull is performed by prepare_oci_layout_artifact (crates/atc-config/src/module_package_oci_pull.rs). It is the SAME walk again — over a materialized OCI image layout instead of a repository directory. atc-config cannot resolve an image ref (it depends on neither atc-oci nor atc-store), so the two halves are split: atc_app::ota_push_resolve_recipes resolves the artifact’s source ref against the laptop cache and builds the artifact’s index.json, and this crate walks the cached blobs and writes the two layout files beside them. The resolution is the SAME one a core oci artifact gets — resolve_oci_source_digest, then the cached-platform check — so there is no second registry client.

• oci-pull-erofs-convert is oci-pull plus ONE GATE, in the same atc_app::ota_push_resolve_recipes: every layer of the resolved manifest must already carry the EROFS media type, the all-or-nothing rule the device’s own desired-set computation applies. It converts nothing; a tar-layered image is refused with atc oci import … --layer-format erofs. The privileged half it was once blocked on is not needed and not rebuilt: mounting is done by crates/atc-mount-helper, a root service with an identifier-only protocol that shipped separately.

  The bridge still refuses it, and that is deliberate. The producer stages an image the device cannot yet find: a module-defined artifact materializes into the channel artifact tree, while the mount reconciler and the digest resolver read oci_tags, the by-digest paths, and mount leases. Until that registration lands, accepting the recipe would report a successful push over an image nothing mounts.

So “which recipes can a module use” still has two answers: walk-directory and oci-pull. The built-in podman kind declares oci-pull-erofs-convert, so podman web-img is still refused.

What an oci-pull artifact does end to end

use registry
image web-img {
    source "ghcr.io/acme/web:1.4"
    platform "linux/amd64"
}

with registry resolved from .atc/modules/registry/module.kdl declaring an artifact "image" whose recipe is "oci-pull".

The field rules are oci’s, not walk-directory’s, and the asymmetry is the recipe’s rather than a policy choice per kind. source is required on both, and it DENOTES different things: a repository-relative directory for one, a registry image reference for the other. platform is REQUIRED here and FORBIDDEN there — one ref can hold an image per architecture and a device has exactly one, so omitting it would push whichever architecture happened to be imported; a directory walk has no registry to select from at all (ArtifactKind::platform_rule, enforced in both directions by kdl_parse_core_nodes).

The image must already be in the LAPTOP CACHE, exactly as for a core oci artifact: atc oci import <ref> is the pull, and a miss is CliError::OciNoCachedImage carrying that command as its hint. atc ota push does not reach a registry.

What lands on the device is the image LAYOUT — oci-layout, index.json, and blobs/sha256/<digest> — at the artifact root, so an oci: transport reference resolves against the delivered directory. The index.json names exactly the ONE ref the artifact declares. It is NOT atc_oci::layout_view’s, which lists every laptop tag pointing at the digest so that concurrent atc oci commands do not clobber each other’s view: that is right for a shared cache view and wrong for an artifact, where it would make target_sha a function of the laptop’s tag bookkeeping. Nothing is written into the cache, so --dry-run resolves one of these without touching store state. Not an unpacked rootfs: a rootfs is made of symlinks, device nodes and uid/gid-owned entries, the walk refuses the first two by design (ATC-E0756/ATC-E0757), and the manifest can carry neither ownership nor whiteouts. Unpacking belongs to whatever consumer has the privilege for it, and no such consumer exists on the device yet (§1).

What a walk-directory artifact does end to end

use assets
files web-assets {
    source "web"
}

with assets resolved from .atc/modules/assets/module.kdl (the repo-vendored rung, crates/atc-config/src/module_resolution.rs) declaring an artifact "files" whose recipe is "walk-directory".

1. resolve_module_defined_artifacts (module_artifact.rs) resolves it to a ModuleDefinedArtifactConfig, parsing the <module>.<kind> spelling (assets.files) against the kind grammar at that one boundary.
2. PushManifest::module_defined carries it. It does NOT go into Config. That is §S1b’s ruling for the module core kind, not a second one. Config is atc.toml’s schema, [[bundle]] and [[oci]] are its only sections, and no TOML section name can spell <module>.<kind>.
3. resolve_push_set runs the recipe ONCE per push, through the same ModulePackageCache a module package uses, so a retry does not re-run a producer.
4. The produced tree is staged at the ARTIFACT ROOT, with no payload/ prefix. That prefix is module-package layout. The wire prefix is now the caller’s parameter, not the producer’s constant.
5. build_from_module_artifact_plan builds ArtifactPushPlan::FromModule. build_artifact_decl puts the spelling on ArtifactDecl.module_defined_kind and leaves the enum kind UNSPECIFIED. Those are the two halves resolve_decl_kind reads.

source is REQUIRED for a walk-directory kind (ArtifactKind::source_required). It is derived from the recipe, not from provenance, for the reason name_shape is. A ProductionRecipe carries no path of its own, so the artifact’s own source is the recipe’s only parameter.

What an operator SEES. ArtifactVersions, ArtifactStatus, DeliveryMember and PerArtifactResult each carry a module_defined_kind beside their ArtifactKind, exactly as ArtifactDecl does on the push path, and artifact_kind_from_wire reads the pair back. So atc ota ls, ota status and ota deliveries name a landed artifact assets.files.

The enum half alone still means bundle when the spelling is empty, which is the v1.0-client case and is deliberate.

atc ota df, push rollback messages and orphan warnings read the pair too (#5127). pb::DfArtifact and pb::ActiveArtifact each gained a module_defined_kind field, and the enum-only converter artifact_kind_to_pb is deleted, so no site can take the first half of the pair by accident.

4. Which verbs actually fire

CommitVerb has six variants and DeferrableVerb three (crates/atc-module/src/verb.rs:87-93, :24-31) — nine verbs in the vocabulary. Production invokes four, from four call sites, all of which go through spawn_commit_verb (module_verb_trigger.rs):

The one module package left in the tree does declare another of the verbs: probe lists workload status (crates/atc-module/fixtures/probe-module/module.kdl). The podman and systemd manifests, which also declared verbs production never spawns, were deleted as unreachable (issue #3665). A declaration is not a trigger, and no production code path spawns those verbs, so declaring one changes nothing. Every one of the four that does fire is gated on the module’s own declaration: an undeclared verb is skipped rather than spawned. atc_module::verb_declaration::default_action states what the skip means (may_spawn, module_verb_trigger.rs). A skip is counted, under atc.agent.delivery.module_verb_skips.

Every committed delivery also books one delivery-level verdict, under atc.agent.delivery.workload_health_verdict. It is tagged verdict (healthy|unhealthy|unasserted). It answers a different question from the per-verb counters above. Did anything report that this delivery’s workload came up?

A delivery that activates no module books unasserted. So does one whose module declares no workload health. Per verb an undeclared health check is a benign skip. Per delivery it is nobody looking, which is issue #3106. The pass emits on every path, so an absent series means the daemon did not report. It never means a healthy fleet (spawn_health_verbs, module_verb_health.rs).

Scope, stated rather than implied. spawn_health_verbs has ONE caller, and that caller is the applier session (crates/atcd-applier/src/session.rs:878). So the health pass covers the starts that session ran, and FOUR production paths start a module with no health pass at all (module_verb_health.rs:122-142):

A recovered first activation, a replacement start and a pulled delivery’s start each get an exit status and nothing else. The baked case does reach this pass and folds unasserted into the delivery verdict, so that skip is a number. The other three never reach the pass, so they book no verdict at all.

The PUSH plane is therefore this counter’s whole subject. A pulled delivery books nothing here. Read atcd.pull.delivery_commit_total beside it to see that half as a number.

Its subject is every COMMITTED delivery, which includes a stage-only push. Such a push terminalizes Committed and activates nothing, so it books unasserted and no workload was ever due to come up. That is true, and it does dilute the ratio on a fleet that stages before it switches. The question the counter answers survives it: does this device ever produce a healthy verdict at all?

atc_module::commit_coordinator, which did map a CommitStep to stop/start/health, had no production caller and is gone. It was deleted in PLAN-D slice 5 along with sequence.rs’s delivery-ordering half (ArtifactName, DeliveryMember, DeliveryPlan), because it sequenced verbs around the store flip and production does an atomic flip followed by best-effort post-commit verbs. Recover it from git history at commit f8f193746, the last tree that still contained it. (It is unrelated to the atcd-applier commit coordinator, which batched whole-blob CAS commits, never touched a module verb, and was deleted in #7439.)