Onboarding: install atc and ship your first update

Who this is for: you have a device (a robot, a Linux box, a container) and you want to deliver software to it from your own machine using the atc CLI. You do not need this repository, and you do not build anything from source — you install a released, signed atc binary and go. This is the fast path from zero to one real, verified update. It assumes you can use a terminal and SSH, and nothing else about atc.

Working in this codebase (contributing)? Start at building-from-source.md instead. That path builds the binaries from source.

Mental model

• atc — the controller. Runs on your machine (laptop / CI). It delivers artifacts to a device over SSH. This is what you install below.
• atcd — the daemon. Runs on the device. It owns a content-addressed store at /var/atc and applies updates atomically. On a systemd device, atc installs it for you, or the device installs it with atcd install (Path A below). On a device without systemd as PID 1, atcd install --no-service seeds it with no unit, or you stage it by hand (Path B).
• Every push is versioned and atomic: bytes are staged first, then the active version is flipped only when you ask with --switch. Rollback is just a flip back to a prior version — nothing is rebuilt or re-downloaded.
• You declare what to ship in an atc.toml manifest and run atc from that directory.

The steps below are sequential — run them in order.

Two pages cover this journey. This one is the complete path: both device types, lifecycle hooks, container images, rollback, and the traps each step hides. deploy-your-first-update.md is the short version of the same journey, with a troubleshooting table. Read one of them, not both.

1. Install atc (on your machine)

curl -fsSL https://atc.dev/install.sh | sh

This drops the atc binary at ~/.local/bin/atc. If that directory isn’t on your PATH, add it before anything else — otherwise the next command fails with atc: command not found:

export PATH="$HOME/.local/bin:$PATH"   # this shell only

If each step runs in a fresh shell (CI, a script, a new terminal), this export won’t carry over. In an interactive shell, add the line to ~/.bashrc/~/.profile now so every later atc invocation finds the binary. But a non-interactive runner (CI, a headless script — bash invoked as sh -c …) sources neither file, so there the reliable fix is to set PATH explicitly in each step, or just call the binary by its full path ($HOME/.local/bin/atc). Inside one script a single export at the top holds for the rest of that script. It is per-step runners, such as CI jobs or separate sh -c invocations, that need it repeated.

Then sanity-check:

atc --version

The one-liner does two things: it fetches the atc binary over TLS (verifying its sha256 digest against the index — transport + integrity), then runs atc upgrade as its final step. That self-update re-checks the signed index — now verifying an ed25519 signature over it against a key pinned inside the binary (cryptographic authenticity) — and provisions a bundled skopeo into the atc-store, the tool atc shells out to for OCI artifacts (atc oci import/push; used only if you ship a container image, see the aside in step 3). So the provisioned skopeo … line you see scroll past during install is expected, not a stray side effect, and the ed25519 check has already run by the time the one-liner exits — it is not a separate step you trigger later. (It’s best-effort: a network or cross-arch hiccup, or a custom ATC_BASE_URL, skips it with a note — re-run atc upgrade yourself to provision skopeo.) Full trust story: verifying-the-binary.md.

The install script honors a few env vars — ATC_BIN_DIR (where to install), ATC_TRACK (which release track to fetch), ATC_BASE_URL (which index origin to fetch from). These configure this install only; they don’t change where a later atc upgrade self-update pulls from.

Put each variable after the pipe, on sh. The script reads it there. To install from the nightly track:

curl -fsSL https://atc.dev/install.sh | ATC_TRACK=nightly sh

ATC_TRACK=nightly curl … | sh sets the variable for curl only. The script does not see it and installs stable, with no error.

Which release track should you use? stable is the default and the right answer for a first install. The nightly and internal dev tracks, version pinning, and the token the dev track needs are all in Appendix: release channels, at the end of this page. Skip it for now.

2. Bootstrap the daemon onto the device

Every command from here on uses deploy@device as the SSH target. Replace it with your own login user and the device’s address, for example pi@192.168.1.5, or with a Host alias from ~/.ssh/config.

Preflight — which path applies to you. atc bootstrap installs atcd as a systemd service, so systemd must be PID 1 on the device. Check that first — it decides which of the two paths below you take:

# accept-new: this is the doc's FIRST contact with the device. Without it the
# host-key prompt hangs a non-interactive shell (details in the note below).
ssh -o StrictHostKeyChecking=accept-new deploy@device 'ps -p 1 -o comm='   # systemd → Path A ; anything else → Path B

This asks what runs as PID 1 (the init process), not whether systemd is installed. A box can carry the systemd tools yet boot a different init — e.g. sshd as PID 1 in a container — and that’s Path B.

• Prints systemd → Path A (the atc bootstrap command just below).
• Prints anything else — a bare container, a minimal image → Path B: don’t run the bootstrap command (it aborts with ATC-E0419, installing only the systemd unit); use the foreground recipe in Path B instead.

Don’t infer the path from the OS or distro — a perfectly ordinary-looking Debian/Ubuntu host can still be Path B if it runs without systemd as PID 1 (e.g. inside a container-based sandbox). Always run the ps check; never assume from how the box looks.

First SSH to a fresh device? The initial connection triggers a host-key trust prompt that hangs a non-interactive run — it fires before any sudo or password prompt. The robust fix is to set StrictHostKeyChecking=accept-new for the host in ~/.ssh/config — it seeds the key on first connect no matter how the device is reached. You can instead pre-seed with ssh-keyscan device >> ~/.ssh/known_hosts (bare host, no deploy@), but ssh-keyscan ignores ~/.ssh/config aliases and talks straight to DNS on port 22 — so for a device reached through a config alias, a ProxyJump, or a forwarded port it fails with getaddrinfo … name resolution (or silently returns nothing). Ask ssh what an alias resolves to — ssh -G <alias> | grep -E '^(hostname|port) ' — then ssh-keyscan -p <port> <real-host>. That covers a renamed host or a forwarded port, but not ProxyJump: ssh-keyscan takes no proxy option and reads no config, so it dials the target directly and either hangs or grabs the wrong key. For a device behind a jump host, use accept-new — the one approach that works whatever the config does.

Concretely: with Host myrobot / HostName 192.168.1.1 / ProxyJump jumphost in your config, ssh myrobot reaches the device through the jump host but ssh-keyscan myrobot tries to resolve myrobot in DNS and fails, because it reads neither the alias nor the ProxyJump. ssh -o StrictHostKeyChecking=accept-new deploy@myrobot … takes the same route ssh already takes, so it works.

Path A — systemd device

atc installs and starts atcd on the device over SSH:

atc bootstrap atcd install --yes --non-interactive deploy@device

Because you installed atc with the one-liner (no repo, no locally-built daemon), atc auto-downloads the arch-matched atcd from the same signed release origin and verifies it (ed25519 signature + sha256) before it is ever copied to the device. This works automatically for linux-x86_64-musl and linux-aarch64-musl devices; for any other device arch (32-bit ARM, RISC-V, …) supply a binary yourself with --from <path/to/atcd>.

• --enroll: enroll the device with a control plane in the same command. The install mints a single-use enrollment token and stages it on the device. No file on your machine holds the token. The server and org come from --server and --org, else the repo’s atc.toml [server] table, else your atc auth login. To use a token minted ahead with atc org enroll-token (one token for many devices, or a token for another person), pass --enrollment-token-file <path> instead.
• --yes (-y) skips the confirmation prompt; --non-interactive is key-only SSH with no prompts. Use both in automation — a sudo or SSH password prompt hangs in a non-TTY shell. Passwordless sudo + key auth on the device is strongly preferred.
• --setup-rootless: delivered podman in one command. Without it, the install leaves delivered podman not set up. atcd ctl podman env then refuses with no_recorded_identity, and atc shell has no podman on PATH. With the flag, bootstrap runs three steps after the install, in this order: atcd upgrade as the store owner, atcd ctl setup-rootless --provision, and atcd ctl setup-rootless --apply. The owner comes from User= in the unit, so a --root install updates as root. The setup-rootless steps install setuid newuidmap and newgidmap helpers, so the flag is opt-in and nothing else runs them for you. It sets up the account that the unit runs hooks as (atc-hooks on a new install). Name another account with --rootless-user <name>. A lifecycle hook gets the delivered engine only when it runs as that account. With --root there is no hook account, so you must pass --rootless-user. A failed step stops the run and names the step. The install stays in place, and you run the same command again. Each step is safe to repeat.
• A slow or flaky link. When the SSH connection fails, bootstrap tries the step again, up to 5 attempts per step and 120 s of waiting per run. It prints one bootstrap: ssh to … retrying line for each attempt. A step that changes the device runs again only when ssh never reached the device. If the connection drops while such a step runs, bootstrap stops with ATC-E0303 and says that the step possibly ran. Run the same command again. A dropped link never removes an install that already started.
• The boot-confirm guard. The install adds a guard that systemd runs before each atcd start. It counts the starts. When a new atcd version fails to reach a healthy ready state on more starts than the guard’s limit (ATCD_BOOT_CONFIRM_LIMIT), the guard rolls back to the previous version. Later sections use this name for it. The guard writes to the atcd journal, so read it with ssh deploy@device 'sudo journalctl -u atcd'. A rollback shows as atcd-boot-confirm: <N> attempts > <limit> — rolling back to prior atcd, then atcd-boot-confirm: reverted to prior atcd.

You do not name the device. A hostname, an IP address and localhost are all valid targets. atcd names itself, so the name does not depend on how you reached the box. It uses a word pair and a 6-character hex suffix (fluffy-kitty-3fa9c1), made from a hash of the hardware identity that bootstrap reads during install. The same hardware gets the same name back after a reflash. A box with no usable hardware identity gets a word pair made from its device id instead. A box that already has a name keeps it.

ATC-E0419 with no systemd means the device has no systemd as PID 1. The message prints a foreground recipe, which is Path B below.

Path A on the device: atcd install

Use this when the device installs atcd itself, for example from a provisioning script, a first-boot job or Ansible. It needs no atc anywhere. The device needs curl, OpenSSL 3, awk, and sha256sum or shasum. Without systemd as PID 1, the script installs with --no-service (Path B). Run on the device:

curl -fsSL https://atc.dev/install-atcd.sh | sudo sh -s -- --socket-owner deploy

The script downloads the atcd for the device’s CPU. It checks the ed25519 signature on the release index and the sha256 of the binary. Then it runs atcd install with your arguments. If a check fails, it stops before it runs atcd.

atcd install does the same install as atc bootstrap atcd install, and it takes these flags:

More than one store on a device. An install that changes the device points /usr/local/bin/atcd at its own store, so the install that ran last owns the link. A link to the atcd of another store is not a difference. --check reports the symlink action only for no link, a file in place of the link, or a link that names no file. --link also reports it when the link names another store.

To add a second store and keep the link where it is, pass --no-link. That install does not make the link, change it, check it, or remove it on a rollback. Its rollback also keeps the boot-confirm guard, because the first install uses it too. It removes the atc shell sudoers file only when that run made the file.

A --no-link install with hook isolation writes the atc shell sudoers file only when the file is absent or holds the grant of its own accounts alone. Any other file stays as it is. When that file does not grant the accounts of the new store, atc shell of it exits 4. atcd install, --check included, prints a kept /etc/sudoers.d/atc-shell line for it. --check reports no action for it, because no run changes it. A path there that is not a regular file stops the install.

The store saves --no-link as an empty no-link directory in the store root. It stays across a reboot and an atcd update. A later atcd install of that store keeps the choice with no flag. atc bootstrap atcd upgrade of that store does not point the link at it, and atc bootstrap atcd uninstall of it keeps the link and the guard.

The uninstall removes the sudoers file only when two things are true. The file holds the grant of that store, and no other atcd store on the device uses the same accounts. A store that saved --no-link keeps the file when the uninstall keeps the accounts, as --keep-store does. When the uninstall deletes the accounts, it removes their grant too. A store with no hook isolation never removes it.

The prompt lists the file as removed only when the run removes it. When the prompt cannot read a fact, its line names every condition.

--check reports the no_link action when the saved choice differs from the run’s: --no-link on a store with no saved choice, or --link on a store with one. --link removes the saved choice and takes the link. atc bootstrap atcd install --no-link and --link pass the flag to atcd install on the device:

sudo atcd install --no-link --store-root /var/atc2 \
  --set daemon.service_name=atcd2 --socket-path /run/atcd2/atcd.sock

For atc to reach that store, give it a link of its own, and set [daemon] remote_path in the repo’s atc.toml to that link. Do not set remote_path to the store’s binary itself: atc bootstrap atcd upgrade points remote_path at that binary, and would replace the binary with a link.

sudo ln -s /var/atc2/channel/agency/atcd/delivery/current/atcd/atcd /usr/local/bin/atcd2

atcd install is safe to run again. On a device that already has the correct install, it changes nothing. On a damaged install, it repairs only the difference, so the repair is the same command. The exit code tells you the result:

atcd install refuses, with exit 1, on a device without systemd unless you pass --no-service (see Path B). It also refuses when a systemd drop-in sets ExecStart for the unit, because the install cannot own a command that a drop-in replaces. The message names the drop-in file. Remove it, then run the install again.

install-atcd.sh exits 1 if a download or a check fails. Otherwise it exits with the atcd install code. These variables change what it fetches. Put them after sudo, as sudo env ATC_TRACK=nightly sh -s -- …, because sudo drops your environment.

Each track also has its own copy of the script, which defaults to that track: https://atc.dev/agency/nightly/install-atcd.sh. The copy at https://atc.dev/install-atcd.sh changes only on an official release.

With Ansible. One task covers a normal run and --check mode. In check mode it passes --check, so atcd install changes nothing and exits 3 on a difference:

- name: Install atcd
  become: true
  ansible.builtin.shell: >-
    set -o pipefail &&
    curl -fsSL https://atc.dev/install-atcd.sh |
    sh -s -- --json --socket-owner deploy {{ '--check' if ansible_check_mode else '' }}
  args:
    executable: /bin/bash
  check_mode: false
  register: atcd_install
  changed_when: >-
    atcd_install.rc == 3 or
    (atcd_install.rc == 0 and (atcd_install.stdout | from_json).changed)
  failed_when: atcd_install.rc not in [0, 3]

check_mode: false lets the task run in check mode, which is safe because it then passes --check. Keep pipefail. Without it, a failed download feeds sh an empty script, which exits 0. To install a binary that is already on the device, run atcd install --json … in place of the curl pipe.

Name the device

atcd keeps the device name in <store_root>/device_name. On a default (root) install that is /var/atc/device_name. A rootless install uses its own store root. The file holds one name as plain text. atcd trims surrounding whitespace. atcd ignores an empty value, an IP address and a loopback name such as localhost.

atcd resolves the name at every start and again when it enrolls. The enrol request carries the resolved name. atcd picks the first match in this list:

1. The name the server returned after a --replaces reinstall or a rename.
2. An atc.device seed from a unit an older bootstrap wrote.
3. The file <store_root>/device_name.
4. A name made from the hardware digest, such as adjective-animal-xxxxxx.
5. A word pair made from the device id.

atcd writes the winning name back to the file when the file is missing or differs. On a first start with no file, atcd therefore writes the hardware-derived name. To choose the name, write the file before the atcd unit first starts. Do this in the factory image or in a first-boot step. The store directory must exist.

sudo install -d /var/atc
printf '%s\n' robot-07 | sudo tee /var/atc/device_name

This works for the default store root. The install refuses a store root other than /var/atc that already holds files and is not an atcd store.

The hardware digest comes from the DMI UUID or the device-tree serial. Containers on one host share the host UUID. Cloned VM images keep the UUID of the source. Devices like these derive the same name, so set the file on each one.

The server keeps the name a device enrolled with. If you edit the file later, only the name in telemetry changes. After a --replaces reinstall or a rename, the name from the server comes first, so an edit changes nothing.

To rename a device after it enrolled, use the console or the CLI. In the console, open the device and select Rename. On the command line, run:

atc fleet device rename <device-id> robot-07

The server refuses a name that another device in the org has. It also refuses an empty name, an IP address and a loopback name such as localhost. The device picks up the new name at its next check-in. Its diagnostic reports use the new name at once. Its telemetry uses it from the next start of atcd.

If another device in the org already holds the requested name, the server enrolls the new device as <name>-<last 6 hex digits of its device id>. It returns a warning in the enrol reply. atcd logs that it was renamed and saves the new name. Telemetry reports the new name from the next atcd start. The run that enrolled keeps the name it started with. A retry of the same enrolment is not renamed, and a --replaces reinstall is not renamed. To keep the identity of a reflashed box, use atc bootstrap atcd install --replaces <device>.

Path B — not systemd (containers, minimal images)

On the device, with atcd install --no-service. It seeds the store and installs the boot-confirm guard, writes no unit, and calls no systemctl. It has the same --check, --json and exit codes as Path A. install-atcd.sh adds the flag itself when PID 1 is not systemd:

curl -fsSL https://atc.dev/install-atcd.sh | sudo sh -s -- --socket-owner deploy

This line runs on the device, so the device needs curl, OpenSSL 3, awk, and sha256sum or shasum. Without curl the installer does not start, and only the device shell reports that curl is not found. When the installer starts, it stops and names the first other tool that is missing. A device without these tools takes the by-hand flow below.

Nothing supervises atcd here. On success, the install prints the one command that starts it in the foreground, and --json carries it as start_command. Run that command under your own supervisor (the container entrypoint, for example).

From your machine, by hand. atc bootstrap won’t stage atcd here (it aborts at the systemd preflight), so you stage it by hand. Two steps: fetch the arch-matched atcd onto your machine, then start it on the device in the foreground.

1. Fetch atcd from the release origin (the same signed index the installer used — this is exactly what bootstrap does for you on a systemd device). Run on your machine; needs jq. Run it with bash, not sh. The block uses a bash array (auth=()), which dash rejects as a syntax error.

# Pick target to match the DEVICE's CPU (not your machine's) — a wrong-arch
# push looks identical in `ota status` later, so get it right here:
case "$(ssh deploy@device 'uname -m')" in
  x86_64)  target=linux-x86_64-musl ;;
  aarch64|arm64) target=linux-aarch64-musl ;;
  *) echo "unsupported device arch — copy your own atcd to deploy@device:/tmp/atcd" >&2; exit 1 ;;
esac
# Read the track your atc came from. Every release prints it in `atc --version`.
# `atc version --json` first shipped in 0.1.5. On 0.1.4 and older it fails with
# "unrecognized subcommand 'version'", so this line parses `--version` instead.
track=$(atc --version | sed -n 's/.*track \([a-z][a-z]*\)).*/\1/p')
: "${track:?could not read track from 'atc --version'. Set it by hand, e.g. track=nightly}"
# The internal `dev` track is token-gated; without the header a dev-stamped atc
# gets a bare 403 here. Send it ONLY for that track and only to atc.dev, mirroring
# what `atc` itself does — a credential should never ride a request that does not
# need it.
auth=()
if [ "$track" = dev ] && [ -n "${ATC_CHANNEL_TOKEN:-}" ]; then
  auth=(-H "X-Atc-Channel-Token: $ATC_CHANNEL_TOKEN")
fi
idx=$(curl -fsSL "${auth[@]}" "https://atc.dev/agency/$track/index.json")
url=$(printf '%s' "$idx" | jq -r --arg t "$target" '.artifacts."agency/atcd"[$t].url')
dig=$(printf '%s' "$idx" | jq -r --arg t "$target" '.artifacts."agency/atcd"[$t].digest' | sed 's/^sha256://')
# `jq -r` prints the string "null" for a missing key rather than failing, and the
# next line would then try to fetch a host named `null` ("curl: (6) Could not
# resolve host: null") — a confusing error for a track that simply has no atcd
# for this arch. Check before fetching:
case "$url" in
  ''|null) echo "no atcd published for $target on track $track" >&2; exit 1 ;;
esac
# Keep these three chained. `sha256sum -c` reports a mismatch only by EXIT
# STATUS, so as separate lines a tampered download still reaches the scp.
curl -fsSL "$url" -o atcd && \
echo "$dig  atcd" | sha256sum -c && \
scp atcd deploy@device:/tmp/atcd

The device has curl and OpenSSL 3? Fetch on the device instead. install-atcd.sh also checks the signature on the index, which the block above does not. It picks the device’s CPU itself, and ATC_DOWNLOAD_ONLY writes the verified binary and installs nothing, so do not pass it arguments here. To install as well, run the script with no ATC_DOWNLOAD_ONLY: it adds --no-service itself (see the start of Path B).

track=$(atc --version | sed -n 's/.*track \([a-z][a-z]*\)).*/\1/p')
ssh deploy@device "curl -fsSL https://atc.dev/install-atcd.sh | ATC_TRACK=$track ATC_DOWNLOAD_ONLY=/tmp/atcd sh"

For the dev track, also set ATC_CHANNEL_TOKEN in the remote command.

Need a specific atcd version, not the track’s current one? The block above always resolves whatever the track publishes today, which is what you want for a normal bootstrap. To pin an older build — testing an upgrade or reproducing a customer’s version — take the binary from that release instead of the track index. Releases are tagged by date (nightly-2026-08-12), and each one attaches index.json alongside the binaries:

tag=nightly-2026-08-12          # the release you want
# $target is the DEVICE's target from the case statement above, e.g.
# linux-aarch64-musl — the same wrong-arch trap applies here.
gh release download "$tag" -R Agency-Tool-Company/agency-code \
  -p "atcd-$target" -p 'index.json' -p 'index.json.sig'
mv "atcd-$target" atcd

Verify the index signature before you trust its digest. The digest below only proves that atcd matches what index.json claims, so an index an attacker also replaced would still say OK. Check the ed25519 signature first with the pinned pubkey, exactly as verifying-the-binary.md describes (it carries the current key and the OpenSSL alternative):

atc sign verify index.json index.json.sig <pinned-pubkey-hex> && \
dig=$(jq -r --arg t "$target" '.artifacts."agency/atcd"[$t].digest' index.json \
  | sed 's/^sha256://') && \
echo "$dig  atcd" | sha256sum -c && \
scp atcd deploy@device:/tmp/atcd

The && chain is load-bearing, not style. atc sign verify and sha256sum -c both report failure through an exit status (VERIFY FAILED: …, exit 1), not by stopping your shell. set -e does not cover this: it has no effect on commands pasted into an interactive shell, which is how an operator runs this. Run them unchained and a rejected signature falls through to the digest check against the attacker’s index, then copies the substituted binary to the device.

Then continue with step 2 below — nothing else about Path B changes. Note that Path A’s --from <path> flag is a bootstrap option, and Path B runs no bootstrap command at all, so a pinned version here is purely a matter of which bytes you scp.

scp reads ~/.ssh/config exactly as ssh does, so aliases, Port, and ProxyJump all resolve — but a -o flag you passed on an earlier ssh command line does not carry over. A successful PID-1 check with inline -o StrictHostKeyChecking=accept-new has already written the device key to known_hosts, so this later scp finds it and does not prompt. If that check never completed — you skipped it, or it failed before connecting — this scp is the first contact and stops on the host-key prompt. Pass the option again (scp -o StrictHostKeyChecking=accept-new …), or set it in ~/.ssh/config once and stop repeating it.

The index URL derives the track from atc --version (which prints … track <name>)), so atcd always comes from the exact track your atc was installed from — controller and daemon can’t drift apart, whatever the installer’s current default track is.

2. Install and start it on the device. Two rules the controller depends on: the binary must live at /usr/local/bin/atcd (the fixed path atc probes), and atcd must run as the same user you SSH in as (start it without sudo so it owns the socket — a root-owned socket makes every later atc command fail Permission denied).

For the simplest copy-and-paste flow, run the whole block as one SSH command. The final setsid … & line backgrounds and detaches atcd; it also works if you already ran the setup lines separately, so don’t start a second copy; two daemons contend for the same socket.

On a container device, plan for restarts before you run it: /run is tmpfs — it does not survive a container restart, so the mkdir + atcd run lines must run on every container start. Bake them into your entrypoint, not a one-time setup script.

ssh deploy@device '
  sudo install -m 0755 /tmp/atcd /usr/local/bin/atcd
  sudo mkdir -p /var/atc /run/atcd
  sudo chown -R "$(id -un):$(id -gn)" /var/atc /run/atcd
  # no sudo on the next line — atcd must own the socket (the second rule above)
  setsid /usr/local/bin/atcd run \
    --store-root /var/atc --socket /run/atcd/atcd.sock \
    </dev/null >/tmp/atcd.log 2>&1 &
'

Release builds log nothing, so an empty /tmp/atcd.log is normal. It is not a sign that the daemon failed to start. Confirm it came up before you continue:

ssh deploy@device 'pgrep -x atcd >/dev/null || { echo "atcd is NOT running" >&2; exit 1; }'

From here the device behaves like any bootstrapped one — skip the bootstrap command and continue below. Nothing here needs jq or python3 on the device. The later steps that do say so where they need it. The hook script in step 3 needs only awk, which every base image carries. The OCI arch check in step 5 needs jq, which a minimal image omits, so expect to install it there.

Started it with sudo by mistake? Then root owns the socket and the very next atc command fails with Permission denied — the error names the socket, not the cause, so it’s easy to misread as a config problem. Stopping it takes sudo too: a plain pkill atcd as the SSH user cannot signal a root-owned process and exits non-zero having done nothing, so the stray daemon keeps the socket and the retry fails identically.

ssh deploy@device 'sudo pkill -x atcd; sleep 1; pgrep -x atcd || echo "stopped"'

Confirm it printed stopped before restarting — then re-check the setsid line for a stray sudo and start it again without one. Never start a second copy while the first is alive: two daemons on one /var/atc contend for the same socket and store.

Restarting atcd (Path B). There is no unit to systemctl restart, so a crashed or wedged daemon is a manual stop-then-start. pgrep -x atcd is the liveness check; if it is already dead, skip straight to the setsid line in step 2 above. To bounce a running one:

ssh deploy@device '
  sudo pkill -x atcd || true      # sudo: it may be root-owned; non-zero = none running
  sleep 1
  pgrep -x atcd && { echo "still running — not starting a second" >&2; exit 1; }
  sudo mkdir -p /run/atcd && sudo chown -R "$(id -un):$(id -gn)" /run/atcd
  setsid /usr/local/bin/atcd run \
    --store-root /var/atc --socket /run/atcd/atcd.sock \
    </dev/null >/tmp/atcd.log 2>&1 &
'
atc ota status deploy@device

The /run/atcd re-create matters after a container restart, not a plain daemon crash. /run is tmpfs, so the socket dir is gone on a fresh container even though /var/atc survives. Never start a second copy while the first is alive. Two daemons contend for the same socket and store.

Upgrade atc first — both paths. atc and atcd exchange a protocol generation at handshake, and accept only an exact match. A stale atc therefore cannot push to a daemon you just upgraded, and the reverse is refused too. So update your own binary before you touch the device:

atc upgrade
atc --version          # confirm the new version and track

Only a push checks that generation. atc ota status and atc ota ls use a separate handshake, so they still answer across a push-wire skew. A clean status therefore proves the daemon is up, not that a push will pass. A push wire mismatch warning means the pair cannot push until you upgrade the named binary. Rollback carries the full skew rule, the warning, and the error a mismatch prints.

Upgrading atcd later (Path A). On a systemd device the controller does the whole swap for you:

atc bootstrap atcd upgrade --yes --non-interactive deploy@device
atc ota status deploy@device   # confirm the new daemon answers

It ships a verified atcd, restarts the unit so the daemon boots the new version (boot-confirm-guarded), and waits for it to come back. --track <name> aligns the device to a track other than your atc binary’s own. Downgrades are always permitted, so this also puts a device back on an older revision.

The new binary comes from ATCD_LOCAL_BIN or a local build when one is present, otherwise the signed release origin. The command therefore needs either a local binary or network access. It is deliberately not taken from the device’s own store. The daemon can write there, and the privileged step must not execute bytes the daemon controls. The store at /var/atc is untouched.

Upgrading atcd later (Path B). atc bootstrap atcd upgrade is Path A only — it restarts a systemd unit and chowns the store to the daemon’s own user, so on a Path B device it fails (no unit, and no such user to chown to). Replace the binary by hand instead. Re-run 1. Fetch atcd from the release origin above to fetch the new binary, then:

ssh deploy@device '
  set -e
  # sudo: the running daemon may be root-owned (see the sudo note above), and an
  # unprivileged pkill would fail silently and leave it holding the socket.
  sudo pkill -x atcd || true          # non-zero simply means none was running
  sleep 1
  # Explicit exit, not `! pgrep`: under `set -e` a negated command is a tested
  # condition, so a surviving daemon would NOT abort the script.
  if pgrep -x atcd; then echo "atcd still running — not starting a second" >&2; exit 1; fi
  sudo install -m 0755 /tmp/atcd /usr/local/bin/atcd
  # same rules as first start: /run is tmpfs, and no sudo on atcd itself
  sudo mkdir -p /run/atcd && sudo chown -R "$(id -un):$(id -gn)" /run/atcd
  setsid /usr/local/bin/atcd run \
    --store-root /var/atc --socket /run/atcd/atcd.sock \
    </dev/null >/tmp/atcd.log 2>&1 &
'
atc ota status deploy@device   # confirm the new daemon answers

The store at /var/atc is left alone: the upgrade is a binary swap, not a store migration. One exception — if the old daemon had been running under sudo, its store writes are root-owned and the unprivileged daemon can’t read them. Hand the store back before starting: ssh deploy@device 'sudo chown -R "$(id -un):$(id -gn)" /var/atc'.

Verify the daemon is reachable — both paths. After Path A’s bootstrap or Path B’s foreground start, confirm the controller↔daemon handshake (the authoritative proof it’s up):

atc ota status deploy@device

An (empty) store listing means the controller can reach the daemon. A just-started daemon can miss the first 5 s ping and return ATC-E0308: … ping … timed out — that’s a cold-start race, not a failure. Re-run the same command once and it answers instantly.

3. Declare what to ship — atc.toml

Work in a directory you own (not this repo). Scaffold a minimal manifest:

mkdir -p ~/first-deploy && cd ~/first-deploy
atc ota init        # writes a minimal atc.toml (one [[bundle]])

atc ota init writes a scaffold: one [[bundle]] named after the current directory with source = "dist", and the rule about where writable state goes. It also carries the lifecycle-hook and long-running-service blocks as comments. Pass --full to also write the compiled-binary block, or read it in docs/operating/atc-toml.md. Pass --format kdl to write an atc.kdl instead. It won’t write beside an existing atc.toml or atc.kdl. This walkthrough doesn’t use the scaffold’s values — its bundle name (app), source directory (app_bundle), and the live commit hook that step 5 verifies all differ. Don’t reconcile the two: replace the generated atc.toml wholesale with this complete example, which ships a tiny bundle (a directory of files) and drops a marker file when the update commits:

[[bundle]]
name = "app"
source = "app_bundle"             # a directory you stage: payload + scripts/

[bundle.lifecycle.hook.commit.exit]   # fires after "app" commits active
script = "@self/scripts/mark.sh"      # @self = this artifact's root on the device
timeout_secs = 60

This marker hook reports that the delivery is durable, which is what commit.exit is for. A hook that brings the new version into service, such as starting a unit or swapping containers, belongs at activate.exit. That is the first gate after the flip, so the gap where the device points at the new version without running it stays short.

If you stage without activating (--no-switch, or no --switch while autoswitch is off), no activate.* hook runs. Nothing is activated, so a restart at activate.exit does not restart the running version. See lifecycle-hook-cutover.md.

source resolves against the directory holding atc.toml, not against your current directory. That still holds under -c/--config, where the manifest can live anywhere. An absolute path is not rejected: it is taken as written, so the push works on your machine and breaks on anyone else’s. Keep source relative. A path that resolves to nothing fails with ATC-E0504: bundle source not found: <path>.

Check the shared libraries of each binary you stage. A binary that links libraries from the build machine fails on the device. It reports error while loading shared libraries. Run file <binary> first, then ldd <binary>, on a machine that does NOT hold the build tree.

file comes first because the artifact may not be a binary. An autotools or libtool build writes a wrapper shell script under the binary’s own name, and leaves the real ELF under .libs/. ldd reports nothing useful about that wrapper, so a check that starts at ldd reads as clean and stages the wrapper.

The glibc risk runs one way. glibc keeps older binaries working, so a binary built against an older glibc than the robot’s runs there. Only the reverse needs a static link: a newer build machine, an older robot. Run ldd --version on both machines to see which case you have.

atc ota push answers that question for you when it can. It reads the robot’s glibc over the ssh connection it already holds, and it compares that against the release each staged binary binds. A robot that meets the release says nothing at all. A robot below it REFUSES the push (ATC-E0469), and names both numbers. Pass --allow-glibc-skew when the workload runs under a glibc the probe cannot see, inside a container or a chroot on the robot.

A push that cannot read the robot’s glibc says nothing, because it measured nothing. Run ldd --version on the robot yourself then.

A static link is the reliable answer. The flag depends on the toolchain, and none of them is a guarantee on its own:

• autotools: configure with --disable-shared --enable-static. That governs the project’s OWN libraries. It does not make libc static, so the binary can still link glibc dynamically. Add LDFLAGS=-static (or libtool’s -all-static) when you want libc in the binary too.
• Go: build with CGO_ENABLED=0 go build. A plain go build looks static and often is not. The standard library pulls cgo in for DNS lookups and for os/user, which links the build machine’s glibc. go build prints no warning about it. Two consequences to know. A program that genuinely needs cgo will not build this way. And the pure-Go resolver reads /etc/hosts and /etc/resolv.conf directly, so an exotic nsswitch.conf behaves differently.
• Rust: build for a musl target, such as x86_64-unknown-linux-musl.

Then check the result rather than trusting the flag. file should report statically linked, and ldd should report not a dynamic executable. If ldd still lists libraries, the flag did not do what you wanted.

Copying the libraries ldd names into source is NOT sufficient on its own. The loader must also be told to look beside the binary, and a RUNPATH covers only the direct dependencies of the object that carries it. Read shared-library-resolution.md before you take that route.

Delivered files are read-only at runtime. The device materializes every bundle file 0o440, or 0o550 when the source file is executable. No delivered file keeps a write bit. A program that writes beside its own binary works in local development and fails after the deployment. Keep writable state under $HOME, or under a path you pass in an environment variable.

The directories that hold those files are not read-only. They keep the daemon user’s own mode, so a manual write over ssh as that user can still succeed.

Do not predict which way a write goes. Two inputs decide it and neither is a guarantee. [daemon] hook_run_as_user picks the account a hook runs as, defaulting to atcd’s own uid. The directory’s mode comes from atcd’s umask when the store was built, so a permissive umask leaves it group-writable, and atc-hooks is in the store group. The same program can take EACCES on one device and succeed on the next. Only the file modes are a guarantee.

A write beside the binary does not follow the pointer. Each version lands in a directory of its own, so a file written there stays with the version that was current when it was written. Retention prunes it later, and a rollback to that version brings it back. Do not read “the next push moves on” as “the data is gone”.

There is no version field. The version is assigned at push time (step 4). A manifest still carrying version = is a hard error.

The device decides what stays on disk. Two device settings control it:

• storage.delivery_keep_n (default 5) keeps the newest labeled deliveries on each channel. A delivery is one committed push. It keeps the versions it installed, so each of these deliveries is a rollback target.
• storage.version_keep_n (default 0) sets the total number of versions to keep for each artifact. Versions that a kept delivery installed count toward this total, and they stay even when they exceed it. The device removes the least recently used of the other versions first.

The active version is always kept. The device prunes in a housekeeping sweep, so you never prune by hand. To change a setting, run atc ota settings set deploy@device storage.delivery_keep_n=10. The change takes effect when atcd restarts.

The hook script below parses its context with awk. Every base image carries it, busybox and debian:stable-slim included. Do not use python3 in a hook: a minimal image such as debian:stable-slim omits it, and so does the meta-atc image. A hook that calls a missing python3 exits nonzero. atcd reads that as a veto: it rolls the whole delivery back and reports the push as failed.

Do not use jq in a hook. Many device images omit it. atcd also pins the hook PATH to the standard system directories, so a hook sees only what the image already installed. A hook that calls a missing jq gets an EMPTY value rather than an error. It writes a broken file and still exits 0, and the push then reports the hook as successful.

Now create the bundle and its hook script:

mkdir -p app_bundle/scripts
echo "hello from atc" > app_bundle/payload.txt
cat > app_bundle/scripts/mark.sh <<'SH'
#!/usr/bin/env bash
set -euo pipefail
# Lifecycle hooks run with a CLEARED environment. The delivery context arrives
# as ONE JSON variable, ATC_HOOK_CONTEXT. There is no ATC_VERSION or any other
# scalar to read a delivery field from. Parse the JSON with awk.
#
# `ctx` reads one dotted field. A second argument gives the value to use when
# the field, or an object above it, is null. With no second argument, a field
# under a null object prints "". An unknown key or a null value exits nonzero,
# and `set -e` then stops the hook, so no step acts on an empty value.
ctx() {
  awk -v path="$1" -v nargs="$#" -v dflt="${2-}" '
    function fail(why) { print "ATC_HOOK_CONTEXT." path ": " why > "/dev/stderr"; exit 1 }
    function find(t, k,   j, c, d, q) {
      k = "\"" k "\":"
      for (j = 1; j <= length(t); j++) {
        c = substr(t, j, 1)
        if (q) { if (c == "\\") j++; else if (c == "\"") q = 0 }
        else if (c == "{") d++
        else if (c == "}") { if (--d == 0) return 0 }
        else if (c == "\"") { if (d == 1 && substr(t, j, length(k)) == k) return j + length(k); q = 1 }
      }
      return 0
    }
    BEGIN {
      s = ENVIRON["ATC_HOOK_CONTEXT"]; n = split(path, key, ".")
      for (i = 1; i <= n; i++) {
        if (substr(s, 1, 4) == "null") { print dflt; exit 0 }
        if (substr(s, 1, 1) != "{") fail("not an object")
        p = find(s, key[i])
        if (p == 0) fail("no such field")
        s = substr(s, p)
      }
      if (substr(s, 1, 4) == "null") { if (nargs < 2) fail("null"); print dflt; exit 0 }
      if (substr(s, 1, 1) != "\"") fail("not a string")
      for (j = 2; j <= length(s); j++) {
        c = substr(s, j, 1)
        if (c == "\"") { print v; exit 0 }
        if (c == "\\") { j++; c = substr(s, j, 1); if (c != "\"" && c != "\\" && c != "/") fail("holds a control escape") }
        v = v c
      }
      fail("not a string")
    }'
}
name=$(ctx artifact.name)
sha=$(ctx version.new_sha)
echo "atc update $name @ $sha ran" > /tmp/atc-app-ran
SH
chmod +x app_bundle/scripts/mark.sh

If a commit hook exits nonzero (a missing reader, a script error), atcd treats it as a veto and automatically rolls the whole delivery back to the previously-active version, reporting the push as failed — a bad hook can’t strand the device half-updated. That is why the hook above fails loudly instead of writing an empty marker.

What the hook receives. $ATC_HOOK_CONTEXT is one JSON document with stable, always-present keys (an absent value is JSON null, never a missing key):

Read a field with the ctx helper above, e.g. ctx version.old_sha. The full ATC_HOOK_CONTEXT schema and hook reference is in atc-toml.md — ATC_HOOK_CONTEXT.

PATH is pinned, not inherited. “Cleared environment” includes PATH: your shell’s PATH never reaches the hook. atcd exports a fixed, absolute one —

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

— so a hook shelling out by bare name resolves only standard system locations. Anything your hook calls (curl, jq, a tool in /opt) must already be installed on the device and reachable from that PATH; otherwise call it by absolute path. So the hook above reads its context with awk, which every base image carries. It does not use jq or python3, which an image may omit. This is deliberate. An inherited PATH would let a writable directory on the daemon’s environment decide what a hook executes. HOME, USER, LOGNAME, and SHELL are set from the hook user’s passwd entry, as in a login session.

The full set a hook gets. ATC_HOOK_CONTEXT carries the delivery context, and it is the only place to read delivery data. The rest of the environment is fixed:

Nothing from your shell reaches the hook. atc ota push --hook-env KEY=VALUE injects one extra variable for one push, and atc ota switch --hook-env does the same for one switch.

Which user is that? It depends on which path you took in step 2:

• Path A (atc bootstrap, systemd): bootstrap creates an atc-hooks user and bakes --hook-run-as-user into the unit. Hooks run as atc-hooks. That user is in the daemon’s group, so it can read the 0750 store but cannot corrupt it. You do not create this user by hand.
• Path B (manual foreground atcd): no hook user is configured, and no atc-hooks user exists. Hooks run as the user that launched the daemon, which is your SSH login user. Do not pass --hook-run-as-user here. A hook uid switch needs a root or CAP_SETUID daemon, which the un-sudo’d foreground recipe deliberately avoids.

The hook runs on both paths. The paths differ in which uid it gets, not in whether it fires. Full reference: atc-toml.md.

Two limits on what a hook can do with systemd. Both come from the device or from the sandbox, not from your hook.

• systemctl --user needs two packages the device may not ship. On Debian or Ubuntu, install libpam-systemd and dbus-user-session. Without libpam-systemd, loginctl enable-linger starts no user manager and each systemctl --user call answers Failed to connect to bus. Without dbus-user-session there is no /run/user/<uid>/bus, so the DBUS_SESSION_BUS_ADDRESS in the table above names nothing, and a hook that speaks D-Bus fails. atc bootstrap atcd install reads the device and warns when either one is absent. atcd installs and runs without both.
• A hook writes its units under its own home, not under /etc. atcd’s unit sets ProtectSystem=full, which each hook inherits. So /usr, /boot, and /etc are read-only to it. The hook user’s home is in ReadWritePaths, so put unit files in ~/.config/systemd/user/ and drive them with systemctl --user. That is the pattern the isolated hook user exists for. It holds no sudo, so a root-owned system unit is out of reach from a hook either way.

Your program prints no version of its own

Some programs carry one version and no more. An autotools C project is the common case. jq --version prints the jq release, and the project has no second version to bump per push. You do not need one. atc owns the version label, and you set it with --version at push time (step 4). atc ota status reports the live label, so “which version is on the device” never depends on the program.

Add an on-device version marker only when the DEVICE itself must report the answer, such as from a log line or a health endpoint. Two patterns work, and neither one patches the program:

• Stage a VERSION file beside the binary. Write the same string you pass to --version into the source directory before each push. The file ships with the payload, so it lands in the version’s own directory on the device. A hook runs in that versioned directory, so it can read the file beside itself. Give the file one writer, your build script, so the file and the push label cannot drift.

  Do not treat the hook’s directory and atc shell -a <name> as one place. atc shell -a <name> enters delivery/current/<name>, which is the ACTIVE version. A hook runs in the versioned tree of the version the push installs. After a stage-only push the two differ. The hook reads the new VERSION file, and the shell still reads the old one. ATC_HOOK_CONTEXT.current_root hands a hook the active pointer when it wants that one instead.

• Ship a wrapper script and point the service at it. The wrapper prints a labelled line, then runs the real binary. Use this when something already starts the program and you cannot add a file read.

Either way, print the label with the value. Bare version output makes a poor marker, for the reason step 5 gives.

Shipping a container image instead?

A [[bundle]] ships a directory. To ship a container image, import it into atc’s local store first, then declare an [[oci]] artifact next to (or instead of) the bundle:

atc oci import alpine:latest              # pull from a registry into atc's store
# or import what your local engine already holds:
#   atc oci import --docker my-app:latest   (--podman for podman)

The registry pull goes through skopeo, which reads your machine’s /etc/containers/registries.conf. On a host whose config is still in the v1 format, that pull fails with a registries.conf format error before atc sees the image. You don’t need to fix the host config — pull the image with your local engine and import from there instead:

docker pull alpine:latest && atc oci import --docker alpine:latest

[[oci]]
name     = "app"
source   = "alpine:latest"   # the ref you imported
platform = "linux/amd64"     # REQUIRED; match the DEVICE's CPU, not your machine's

Shipping an image instead of a bundle? Then that block is the whole file. There is no [[bundle]] to keep:

[[oci]]
name     = "app"
source   = "alpine:latest"
platform = "linux/amd64"

platform is required because an image ref can carry several architectures, and atc must know which one belongs on the device. It is the same wrong-arch trap as the target note in step 2: a wrong value looks fine at push time. Take the value from the device, not from your machine. ssh deploy@device 'uname -m' prints x86_64 for linux/amd64 and aarch64 for linux/arm64. OCI names the arch differently from the kernel, so linux/x86_64 and linux/aarch64 are not the values to use. Pushing and activating then work exactly like the bundle flow in steps 4-5, and [[oci]] artifacts take lifecycle hooks the same way a bundle does (under [oci.lifecycle.*]), so a commit.exit hook can verify an image just as mark.sh verifies the bundle.

Verifying which platform landed is the one step that differs: the ELF-byte check in step 5 reads a single binary, and an image has no equivalent. The arch lives in the image’s config blob, two hops from the version dir’s index.json (index → the single manifest blob → its config.digest). Walk it on the device:

ssh deploy@device '
  d=/var/atc/channel/local/default/delivery/current/app
  m=$(jq -r ".manifests[0].digest" "$d/index.json" | cut -d: -f2)
  c=$(jq -r ".config.digest" "$d/blobs/sha256/$m" | cut -d: -f2)
  jq -r ".architecture, .os" "$d/blobs/sha256/$c"
'

That prints the arch and OS baked into the image that actually landed — expect amd64 on an x86-64 device, arm64 on aarch64, and your laptop’s arch here is the wrong-platform bug.

This check needs jq on the device, and nothing else in this guide does. The hook script reads its context with awk for that reason. So install jq before you run the block above, and only for this check:

ssh deploy@device 'sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y jq'  # Debian/Ubuntu
# Alpine: sudo apk add jq   ·   Fedora/RHEL: sudo dnf install -y jq

Don’t hand-parse these blobs with grep/sed instead: the config digest sits in a "config" object that also holds a "digest" key for every layer, so a line-oriented match picks up a layer digest and the walk dies on a missing blob.

atc ota status and atc ota ls --artifacts deploy@device confirm the version activated. For an OCI-only delivery, their Arch column reads unknown: the column reports the first ELF arch found among the active delivery’s members, and OCI members are not ELF files on disk. Check an OCI payload’s arch from its image config instead.

Full [[oci]] schema and OCI lifecycle hooks: atc-toml.md.

Shipping a module instead?

A bundle and an OCI artifact are payloads the device unpacks. A module is an executable the device also runs, each time a delivery ships it — including a re-push at an unchanged SHA. A delivery that carries a module forward untouched does not re-run it. Write start to be safe to run again. Modules are declared in atc.kdl rather than atc.toml, and much of the module vocabulary is still refused at push time, so read modules-today.md before you design around one. A repo you can copy, with an e2e test that pushes it to a real container, is examples/heartbeat-module/.

Optional: run the whole push on your own machine first

atc ota push --local starts a throwaway atcd on this machine and pushes to it. It needs no device, no SSH and no root, and it runs the real daemon. So a manifest and a hook that pass here ran against the real executor. Use it to check atc.toml and mark.sh before you involve the device.

Get an atcd for this machine first. The installer places only atc, so a fresh machine has no atcd and the push stops on ATC-E0463. One command provisions one, and it needs no device:

atc bootstrap atcd fetch

It reads the same signed release origin atc upgrade reads. It checks the ed25519 index signature and the sha256 artifact digest. Then it writes the binary where --local looks for one: beside your atc when that atc is a plain binary, and ~/.local/bin otherwise. It also says whether --local can reach the result, and names the fix when it cannot.

Add --track <track> to follow another channel, or --source <name> to fetch from a private release mirror registered in sources.toml. Both resolve the way atc upgrade resolves them. Then:

atc ota push --local --switch --local-root /tmp/atc-local-test

Give --local-root an absolute path, and keep it short. A relative path makes every hook fail to spawn (spawn failed: No such file or directory), and the failure vetoes the push. The root also holds atcd’s Unix socket, and a socket path over 103 bytes is refused with ATC-E0463.

• Linux only. The released atcd assets are Linux x86_64 and aarch64 binaries. On any other host atc bootstrap atcd fetch refuses and says so.
• --local needs an atcd for this machine. atc bootstrap atcd fetch is the short way. atc also accepts one you supply: put it next to atc, put it on PATH, or name it with ATC_LOCAL_ATCD. The Path B fetch block in step 2 downloads one too, but pick your OWN target there, not the device’s.
• Pass --switch or --no-switch. The command refuses to choose. --switch runs the activate and commit hooks. --no-switch stages, which runs the download and install hooks and then stops.
• --local-root <DIR> keeps the run: the store, atcd.log, and each hook’s stdout and stderr under hooks/. Without the flag --local uses a temporary directory and removes it. A directory that already holds something else is refused.
• This is not a sandbox. Hooks run under your uid, in your home, against your services. --local relocates the store and nothing else.
• Push the same content twice into one --local-root and the daemon treats it as already present. No hooks run at all. Change something between runs.

Your hook writes /tmp/atc-app-ran on this machine, so read it here:

cat /tmp/atc-app-ran

4. Push and activate

Every artifact by default — a bare atc ota push pushes every artifact declared in atc.toml. (Omit --version and the push gets an auto-generated <host>-<timestamp>-<entropy> label.) Give the push a version and add --switch to activate it:

atc ota push --version v0.1.0 --switch deploy@device

• You don’t have to be in the manifest directory. atc discovers atc.toml by walking up from the current directory, so any subdirectory of ~/first-deploy works. Pass -c/--config <FILE> to point at a manifest elsewhere and skip that search entirely.

• --only <name> (repeatable) narrows the push to a subset of artifacts; a bare push (no --only) pushes all of them.

• -t app@v0.1.0 is shorthand for --only app --version v0.1.0.

• Without --switch the version is staged on the device but it keeps running the old one.

• The push log shows the commit.exit hook firing. The log prints one line per hook, when it finishes. For a hook that prints nothing on success, like the mark.sh above, which only writes a marker file, the log shows:

    hook logs: <cache-path>
    ✓ commit.exit  app  scripts/mark.sh  8ms
    i commit.exit produced no output
  

  The log prints hook logs: once per push, under the push-started line. Under --non-interactive, which prints no push-started line, it prints above the first hook that records output to disk. A failed hook reads ✗ and its exit state, for example exit 3 or timed out.

  The interactive view draws the same hook as └ ✓ commit.exit app scripts/mark.sh under the Commit row, and names hook logs: <cache-path> once, under the first line of the push.

  <cache-path> is $XDG_CACHE_HOME/atc/hooks/<timestamp>-<target>/, or ~/.cache/atc/hooks/<timestamp>-<target>/ when XDG_CACHE_HOME is unset. It holds one <state>.<phase>__<hook-run-id>.stdout/.stderr pair per hook run. These are captured on your machine by atc, not on the device. Recording is on by default for every push; --capture-hooks <DIR> pins a different directory, and --json is the one mode that records nothing.

  The ✓ commit.exit line is your confirmation the commit.exit hook ran and succeeded — it’s the one stable signal to grep for. A failed hook prints ✗ and the reason instead: exit 3, signal 9, timed out or spawn error. Each hook run also carries a unique opaque id, which the recorded filenames use. Don’t string-match against that. The hook logs: and i lines are normal, not a warning: atc captures every hook’s stdout/stderr to disk by default, and notes when a hook was silent. The hook runs after the atomic version flip, exactly as the [bundle.lifecycle.hook.commit.exit] comment says.

• atc ota status’s lifecycle log names the same hook by its journal label, commit/pre_hook_exit, which does not match the commit.exit you wrote in atc.toml. That’s the daemon’s internal journal name for the gate, and pre_ means “about to run the exit hook”, not “before the commit”. commit.exit, commit.post, and commit/pre_hook_exit all name the same gate; commit.enter / commit.pre map to commit/pre_hook. The push log uses commit.exit. Match on ✓ commit.exit, never on the journal label.

• Did the push stop with ATC-E0460: atc and atcd are incompatible or ATC-E0016: unsupported protocol version before any payload moved? An older atc can show ATC-E0443: store: push failed first, with the ATC-E0016 nested below it. Then atc and the device’s atcd speak different push wires. This happens on a device that an older release bootstrapped. Upgrade the binary that the error names. Rollback carries the full rule.

5. Verify it landed

Prove it with commands — don’t assume:

atc ota status deploy@device                        # active version should now be v0.1.0
ssh deploy@device 'grep "^atc update app @" /tmp/atc-app-ran'  # hook marker + content
ssh deploy@device 'od -An -tx1 -j18 -N2 /usr/local/bin/atcd'  # ARCH that landed (see below)

If ota status shows v0.1.0 current and the marker line is present, the update is live. Grep the marker’s content (not just cat it) — a hook that ran with a broken parse could leave an empty file that a bare cat would still show.

Label every version string you check. A bare version marker is a common cause of a verify step that reads as a failure. A program’s own --version often prints 1.0.0 and nothing else. That string holds no stable word to anchor a grep, so one grep cannot separate three cases:

• the new version ran
• an older version ran
• the command printed nothing at all

Print a label beside the value instead. The label proves the command ran, and the value proves which version ran:

ssh deploy@device 'echo "myapp-version=$(myapp --version)"'

Then check the label and the value together:

ssh deploy@device 'echo "myapp-version=$(myapp --version)"' \
  | grep -q 'myapp-version=v0.1.0'

A command that fails prints the label with an empty value. The paired grep rejects that line, so a broken run cannot read as a pass. Use the same pattern for a hook marker file: write label=value, never the value alone.

Confirm the architecture that shipped. ota status reports version and sha, atcd --version reports atcd <semver> with its existing prefix, and atcd version (the subcommand, which also takes --json) reports semver, git sha, and build date. None of them report target arch, so a wrong-arch binary (e.g. a linux-x86_64-musl build on an aarch64 device) looks identical there. Read the deployed binary’s ELF header directly — byte 18 is e_machine, the CPU the binary was built for:

ssh deploy@device 'od -An -tx1 -j18 -N2 /usr/local/bin/atcd'
#   3e 00  → x86-64      ·   b7 00  → aarch64      (ELF e_machine, little-endian)

e_machine is a little-endian 16-bit field, so the two bytes read low-byte-first: 3e 00 is 0x003e (62, x86-64) and b7 00 is 0x00b7 (183, aarch64). Bytes in the other order would mean a big-endian ELF, which no device here targets.

od is coreutils, so this works even on the stripped, file-less container images Path B targets. If file is installed it’s friendlier — ssh deploy@device 'file /usr/local/bin/atcd' prints … x86-64 … statically linked — but it’s absent on minimal images, so don’t lead with it. Don’t fall back to uname -m: it reports the device’s CPU, not the binary’s arch, so it can’t catch a wrong-arch push onto a matching-CPU host — the ELF-byte check above is the real proof. (Default daemon path is /usr/local/bin/atcd; adjust if you set [daemon] remote_path.)

Where your files land on the device. In the default (delivery-pinned) mode, point on-device services at the authoritative delivery pointer — atc flips it atomically on every switch and rollback:

/var/atc/channel/local/default/delivery/current/<name>

6. Rollback

Rollback is a flip back to an earlier version, so it needs at least two on the device — if you’ve only pushed v0.1.0 so far, repeat step 4 with --version v0.2.0 first. List what’s on the device, then switch:

atc ota ls deploy@device                      # list on-device versions
atc ota switch deploy@device <older-version>  # re-activate an earlier one

switch needs a current atcd. Whole-delivery switch is its own wire operation, so an atcd that predates it doesn’t recognize the request and the switch fails with ATC-E0455: atcd is too old for the whole-delivery switch. Nothing was activated — the old daemon refuses before touching the store — so the device is still on the version it was already running. The fix is forward only: upgrade atcd on the target (atc bootstrap atcd upgrade deploy@device on a systemd device, or the Path B binary swap in step 2), then re-run the switch. Downgrading atc to match an old daemon is not supported.

An old atc against a newer atcd is refused too. The push wire is a single generation per build. atc and atcd exchange a protocol version at handshake and accept only an exact match. A push across a skew in either direction is therefore rejected before anything transfers. There is no “old-client-still-works” window for a push. status and ota ls still answer, as the paragraphs below show.

Keep the pair in step: upgrade atc with atc upgrade, and the device’s atcd by Path A or Path B above.

The refusal happens at the push’s first stage, before any payload transfers, and it names the binary to upgrade:

• A current atc against an older atcd fails with ATC-E0460: atc and atcd are incompatible, followed by upgrade this daemon (atcd) and the fix command atc bootstrap atcd upgrade <target>.
• An older atc against a newer atcd fails with ATC-E0016: unsupported protocol version, followed by upgrade the pushing client (atc). A release that predates ATC-E0460 wraps it in ATC-E0443: store: push failed.

atc ota status does not refuse a skewed pair. Status and ota ls use the attach handshake, which has its own version and does not move with the push wire. Measured: an atc 0.1.3 (atc/5) read status and listed deliveries from an atcd 0.1.5 (atc/6) with no error, then failed its first push with ATC-E0016.

A newer atcd reports its push wire in the status reply, and atc ota status then flags a skew itself. The output ends with warning: push wire mismatch: this atc speaks atc/N, the daemon speaks atc/M. and a fix: line that names the binary to upgrade. --json carries the same answer as "push_wire": {"atc", "atcd", "match"}. An atcd that predates the field reports "atcd": null and "match": null, and prints no warning. For that daemon, compare the Daemon column of atc ota status deploy@device with atc --version. The same release always speaks the same wire. Different releases may still match, so a push is the final answer.

A switch re-runs the target version’s activate and commit lifecycle hooks (with trigger: "user_activate" in $ATC_HOOK_CONTEXT). They are not “fire once at first install.” The order is activate.enter, deactivate.enter, deactivate.exit, the pointer flip, activate.exit, commit.enter, commit.exit. The two deactivate cells run the declaration of the OUTGOING version, which is the one the switch leaves. Your marker is rewritten on rollback, carrying that version’s sha. A hook that restarts a service or runs a health check fires on the rollback too. The store itself is only a pointer flip — nothing is rebuilt or re-downloaded — but the hooks do run.

Prove the rollback the same way you proved the push (step 5) — one verification per delivery, whether it arrived by push or by switch:

atc ota status deploy@device                    # active version is the older one again
ssh deploy@device 'grep "^atc update app @" /tmp/atc-app-ran'  # marker rewritten with THAT version's sha

.version.new_sha is the artifact’s content sha, not the delivery version string. It tracks the payload, so it changes only when the payload changes. Roll back to a version whose payload is byte-identical and the marker gets rewritten with the same sha it already held. An unchanged marker is therefore not evidence that the hook was skipped. To prove a re-run for any payload, make the hook append instead of overwrite, or record a timestamp with the sha:

echo "atc update $name @ $sha ran at $(date -Is)" >> /tmp/atc-app-ran

The ✓ <state>.<phase> hook line in the push output is the authoritative signal that the hook ran; the marker proves it did its work.

Pass a bare delivery version (v0.1.0) — in the default delivery-pinned mode that re-activates the whole delivery. The name@version form (or -a <name>) is a per-artifact escape hatch, valid only when assignment_mode = per_artifact; under the default it is rejected with ATC-E0454: switch refused: … device is delivery-pinned.

Open the hatch with atc ota assignment-mode:

atc ota assignment-mode deploy@device per_artifact --for 1h   # lease the opt-in
atc ota status deploy@device                                  # MODE reads per-artifact
atc ota assignment-mode deploy@device delivery_pinned         # back to the default

--for is the safer form. The daemon re-reads the mode at the instant it activates. An expired lease therefore refuses the next lone switch, and the device returns to the delivery-pinned posture on its own. Omit --for and the override has no expiry.

The ota switch argument order is <TARGET> <VERSION> — host first, version second. A reversed form like atc ota switch v0.1.0 deploy@device parses v0.1.0 as the hostname and fails with ATC-E0303: could not resolve hostname.

Appendix: release channels

dev gets you the tip of dev within about an hour of a merge, instead of waiting for the nightly.

Installing a specific version

A track always serves its current release: https://atc.dev/agency/<track>/index.json is overwritten in place on every publish, and the install script takes ATC_TRACK but no version pin. So the curl | sh one-liner cannot give you v0.1.2, and neither can the Path B fetch script below.

Pinned stable and nightly versions come from the GitHub release page instead. Those releases keep their own binaries, so prior versions stay available:

gh release list -R Agency-Tool-Company/agency-code --exclude-pre-releases   # stable releases
gh release download v0.1.2 -R Agency-Tool-Company/agency-code \
  -p 'atc-linux-x86_64-musl' -p 'SHA256SUMS'
grep 'atc-linux-x86_64-musl$' SHA256SUMS | sha256sum -c   # verify before running
chmod +x atc-linux-x86_64-musl
./atc-linux-x86_64-musl --version                         # atc 0.1.2 (6b17235ca …, track stable)

A pinned release prints the wire generation it speaks as a trailing [wire atc/N]. Releases older than that banner print no such segment. v0.1.2 above is one of them.

The push wire accepts only an exact match, so a pinned atc older than the device’s atcd cannot push to it. See An old atc against a newer atcd is refused too in step 6.

A bannerless binary does not tell you its generation, and the release notes do not state one. For a stable release, use this list. It was read from each tag’s source and checked against the published binaries:

A bannerless nightly may speak either generation. The banner arrived after the atc/6 bump, so its absence proves nothing. Do not use atc ota status deploy@device as the probe: status uses a separate handshake and answers across a push-wire skew. A push is the probe that answers. A mismatch is refused at the push’s first stage, before any payload transfers. If the versions match, the push uploads and stages a real delivery. With autoswitch = true in atc.toml, it also activates it. Use this probe only when you intend to push that delivery.

Assets are named <binary>-<target> for the same targets the index carries (atc/atcd × linux-x86_64-musl, linux-aarch64-musl; atc also darwin-arm64), alongside SHA256SUMS and the signed index.json. Pick the target matching the machine that will run the binary. For atcd that is the device, not your laptop.

Nightly builds are published as pre-releases (dropped by --exclude-pre-releases above); list them with a bare gh release list.

The internal dev channel is the exception. Its per-merge publishes record only a lightweight tag, with no release object and no assets, to keep the Releases page clear. So gh release download has nothing to fetch for a dev build. Pin a nightly release instead, or build the binary yourself from the tag’s commit.

This is the path for testing CLI/daemon compatibility against an older release. Note that a pinned binary is outside the self-update flow: a later atc upgrade moves it back to the track’s current release.

Opting in

The dev track is internal and token-gated: the index is served only to requests carrying X-Atc-Channel-Token, which atc reads from ATC_CHANNEL_TOKEN.

# 1Password → Software vault → item "atc-dev-channel" → field "token"
export ATC_CHANNEL_TOKEN=…        # put this in your shell profile
atc bootstrap track dev           # persists in sources.toml
atc upgrade

On a machine with no atc yet, the installer takes the same variable:

export ATC_CHANNEL_TOKEN=…
curl -fsSL https://atc.dev/install.sh | ATC_TRACK=dev sh   # or ATC_TRACK=pr-<number>

The installer reads the token only from the environment, and it sends the token only to https://atc.dev/agency/dev/ and /agency/pr-*/. With no token, it stops before the first download and names ATC_CHANNEL_TOKEN.

The value must carry no trailing newline or surrounding whitespace. The edge compares the header byte-for-byte and atc sends what you exported verbatim — it rejects a blank token but never trims one — so a stray newline from a copy or a $(cat file) produces a 403 while the 1Password value looks perfectly correct. The current value is 64 hex characters — nothing enforces that length, it is just how this one was minted, so treat the count below as a fingerprint of today’s token rather than a rule:

printf '%s' "$ATC_CHANNEL_TOKEN" | wc -c   # 64 for the current token

What is actually required is that the value match the deployed one exactly, with no leading or trailing whitespace.

atc --version should now print track dev.

To go back: atc bootstrap track stable && atc upgrade. Use atc bootstrap track, not a one-shot --track — the flag does not clear a track recorded in sources.toml, so the next bare update would return to dev.

Five things that will confuse you otherwise

No token means 403 on every update. atc names ATC_CHANNEL_TOKEN in the error rather than surfacing a bare HTTP status, so it should be obvious. A 403 with the token set is one of two things: a stale value (re-pull from 1Password) or stray whitespace in it (check the byte count above). The second is the likelier one and the harder to see, since the 1Password UI shows nothing wrong.

Version numbers barely move. Dev builds are 0.1.3-rc.<minute>.<sha>, all sharing one base until the next official release. Self-update compares the content DIGEST, not just semver (the channel is published rolling), so you do get new builds. The version string is just not how you tell them apart. Use the sha.

No GitHub releases. Each publish records a bare git tag (dev-<date>-<sha>) and nothing on the Releases page. Deliberate: per-merge prereleases nobody consumed are what got this channel retired the first time. Release notes live at the URL the index carries.

Not every merge publishes. A docs-only or terraform-only merge does not trigger a release, and a burst of merges collapses into one. Expect roughly one release every couple of hours on a busy day, not one per merge.

The macOS atc arrives after the Linux binaries. Each dev release publishes the index twice. The first publish carries only the Linux targets, so a device gets its binaries without a wait for the macOS runner. When the native macOS build finishes, the release republishes the index with the darwin-arm64 entry. Between the two publishes, the index has no darwin-arm64 atc. On a Mac, an update in that window fails with release index has no artifact for package agency/atc on target darwin-arm64.

The gap is usually one or two minutes. It is longer when the macOS runner is queued or the macOS build is slow. The timeout-minutes of the macOS build (build-atc-macos-release.yml) and of the release-macos job (release.yml) set how long each job can run.

If you see that error on the dev track, wait a few minutes and run atc upgrade again. If the error stays, open the newest dev-release run in GitHub Actions and read the status of two jobs. These are build atc (macOS / Apple Silicon, after the Linux publish) and republish index with macOS atc. If either job is queued or in progress, wait for it. If either job failed, the index has no darwin-arm64 until the next dev release. Use nightly meanwhile, or build atc locally with AGENCY_BACKEND=local ./agency build -c opt //crates/atc-cli:atc.

Before you rely on it

The gate covers the index, not the binaries: those live in the shared content-addressed blob prefix and stay publicly readable by digest. It keeps customers off the channel rather than keeping the bytes secret.

Rotating the token is a tofu apply plus an env update everywhere, with no overlap window — it is baked into the CloudFront function, which cannot read a secrets store. It is a long random value precisely so it does not need a schedule.

Going deeper

• building-from-source.md — the same journey but building atc/atcd from source via ./agency (the contributor / CI path).
• deploy-your-first-update.md — end-to-end quickstart with the no-systemd / container recipe and a troubleshooting table.
• atc-toml.md — the atc.toml manifest schema and the full lifecycle-hook (ATC_HOOK_CONTEXT, @self/@bundle, state/phase) reference.
• lifecycle-hook-cutover.md — which gate each hook belongs at, and what to check if you already have commit.* hooks in the field.
• verifying-the-binary.md — independently verify the atc/atcd binary you’re running is genuine (sha256 + ed25519-signed index).
• console-config-publish-readiness.md: run this checklist before you promise a customer config publish from the web console. Until it is green, use the CLI path above.

For an ATC-Exxxx error, see docs/development/error-codes.md.