Modules today: what a repo can actually do

Every claim below is cited to a file and line on dev. Where something does not work, this says so; the gap list at the end is the point of the document, not an appendix to it.

The short version: a repo can ship one executable per module. atcd now drives four of the nine verbs against it. workload start runs after a successful delivery. workload health runs after a successful start the applier session itself ran, and after no other start (§4). The deactivation drain runs workload stop, and workload remove for a module that leaves the device (§4).

The device also reads the delivered module.kdl. It spawns only the verbs that file declares. It applies the per-verb timeout the file asks for (§2).

Workloads parse, bridge, cross the wire, and persist per delivery, and the trigger spawns one workload start per DECLARED workload (§5). A stock atc can now get one to start. The workload kinds it knows are systemd (from the systemd module) and container (from podman and nerdctl). Those are MODULE names, and podman is also an artifact kind, not a workload kind. A workload of any other kind now pushes too. Its declared fields are read against its own module’s schema and sent as a fields_json_v2 document (§5).

A workload of a known kind names a built-in module, and no built-in module is ever in a push’s changeset — so the trigger cannot see it. The commit derives a lifecycle for it instead, keyed by a version made from the declaration set, and the drain runs the start out of the image’s own copy of the module (§5). It needs the image to really carry that module; nothing here installs one. A module-defined artifact kind now pushes for walk-directory and oci-pull (§3). oci-pull-erofs-convert has a producer too, and the bridge still refuses it, because no device consumer reads what that producer stages — so the built-in podman artifact kind is still unusable. atc ota ls, ota status, ota deliveries and atc ota df name the landed kind by its own spelling (§3). The other five verbs are still unreachable code.

A worked example that stays inside those limits lives at examples/heartbeat-module/, with an e2e test that pushes it to a real container: crates/atc-cli/tests/module_example_e2e_tc.rs.

1. Getting a module into a config at all

A module can only be declared in atc.kdl. No TOML syntax declares one, by design — docs/internal-site/src/design/2026-08-02-modules-production-readiness.md §S1b declined to add a [module] table to the TOML schema (crates/atc-config/src/module_artifact.rs:1-16).

A repo whose nearest manifest is atc.kdl is read as KDL with no setup at all. That is the shape a module repo has, so a module push is the plain command:

atc ota push atc@<device>:22 --version 0.1.0 --switch

Upward discovery stops at the nearest directory that holds atc.kdl or atc.toml. When that directory holds both, atc.kdl wins (#6337). Both files parse, so the losing file is ignored with no error. atc prints a note on stderr that names the ignored file, the file that won, and ATC_KDL_CONFIG (manifest_format::shadowed_manifest_notice). atc config lint says the same thing (ATC-E0598).

ATC_KDL_CONFIG=0 (crates/atc-config/src/parse.rs, kdl_config_enabled) restores the rule from before #6337: an atc.toml at any depth wins, and an atc.kdl is read only when no atc.toml is on the walk. It is the one-line way back for a repo that holds both files and depends on its atc.toml.

An explicit -c <file>.kdl needs no variable. The operator names the file, so nothing is shadowed, and the format follows the name. A .kdl-named file that holds TOML text no longer loads. It stops at a KDL error that names the file.

The full table lives on KDL_CONFIG_ENV in crates/atc-config/src/parse.rs; the ambient half of the decision lives in crates/atc-config/src/manifest_format.rs.

The declaration takes three fields: name on the top line, source, and an optional payload (ModuleArtifactConfig, crates/atc-config/src/module_artifact.rs:46-63). The smallest declaration is three lines:

module heartbeat {
    source "./modules/heartbeat"
}

source is required for module (kdl_schema.rs:184-186) and platform is forbidden (kdl_schema.rs:219-224). The name must be a single path component, because it becomes a directory on the device (kdl_schema.rs:160-162). payload names a directory this push walks into the package’s payload/ subtree (§2).

What the same file may NOT contain

kdl_bridge::bridge translates the rest of atc.kdl into a Config, and it refuses the whole file rather than silently dropping anything it cannot carry. Adding any of these to a module repo’s atc.kdl makes the push fail:

A hooks { … } block on an artifact is not on that list. kdl_lifecycle converts it, and PushManifest::artifact_lifecycle carries it to the push (kdl_bridge.rs:214-220). A hook that cannot convert is refused per hook as ATC-E0775, by atc config lint and by the push (push_manifest.rs:155).

A plain workload node is not on that list any more: it bridges, rides the push, and lands in delivery_workloads (§5). Only the two workload shapes above still refuse, and each refuses per declaration, so a file with three bad workloads reports all three.

A bare use <module> line is allowed but buys nothing on its own: it only makes a module’s node names available. What it buys once you use those names depends on which one — a systemd workload now pushes, an artifact of a walk-directory or oci-pull kind now pushes (§3), and a podman artifact is still caught by the last row above, because its recipe is oci-pull-erofs-convert.

Device coordinates a KDL repo can say, as of the daemon and bootstrap top-level nodes:

daemon {
    socket-path "/run/robot/atcd.sock"
    remote-path "/opt/robot/bin/atcd"
}
bootstrap {
    auto #false
}

socket-path and remote-path are the TOML [daemon] fields of the same name, and auto is [bootstrap] auto; each is derived by the bridge, and omitting a node means what omitting the TOML table means rather than a second default (crates/atc-config/src/kdl_bridge.rs, the daemon:/bootstrap: fields). remote-path is not decoration next to the socket: it also feeds the push connection target, so a device with atcd installed off /usr/local/bin/atcd needs it. Both blocks reject an unknown child, so the rest of each namespace stays reserved for later fields. Working example: crates/atc-config/tests/fixtures/kdl/valid/pushable/device-coordinates.kdl.

The rest of [daemon] — telemetry endpoints, hook-retry budgets — is still defaulted: it is daemon-side configuration, not something a producer file describes.

2. What the module package must contain

Read this section against prepare_module_package (crates/atc-config/src/module_package.rs:745). That is the production entry point. resolve_push_set reaches it through ModulePackageCache::get_or_prepare (crates/atc-app/src/ota_push_resolve.rs:102-111). build_module_manifest is not the entry point: it has no production caller, and its own comment says so (module_manifest.rs:245). It also refuses any module that declares a payload, so it cannot describe the package shape below.

prepare_module_package runs ONCE per push, before the retry loop. A producer re-run inside that loop could hand attempt two different bytes for one push.

The package’s own directory holds exactly two flat files (stage_core_files, module_package.rs:646):

modules/<name>/module.kdl
modules/<name>/<name>          # the executable, same name as the module

• A directory entry is refused (module_package.rs:669) and a third file is refused (:683). A symlink is refused by O_NOFOLLOW, which reports ELOOP (:381-385), and a socket or an unbacked device node reports ENXIO (:386-391).
• The executable must be a regular file (:675) carrying the owner execute bit (:691). Both checks are sender-side only. A wire Manifest carries no file type, so the receiver has nothing to re-derive them from (module_manifest.rs:29-35).
• A module may not be named module.kdl or payload (module_package.rs:749-772). The receiver refuses those two and exec-digest.json as well (module_protocol.rs:233-268).

The payload node

A package may carry a THIRD thing: a payload subtree. Declare it on the module node.

module heartbeat {
    source "./modules/heartbeat"
    payload "./assets"
}

payload names a repository-relative directory. prepare_module_package walks it and stages every regular file beneath it under the package’s payload/ prefix (module_package.rs:799-802). The node is valid on a module artifact only. On any other kind it is ATC-E0770 (kdl_parse_core_nodes.rs:117-133).

payload is sugar for the walk-directory producer, so it refuses what that walk refuses. The walk ships REGULAR FILES only, and it descends a directory to find them. It refuses every other entry type by that property, not by a list (walk_payload_entry, module_package.rs:516-532). The refused types are a symlink, a FIFO, a socket, a device node, and any type a future kernel adds. A symlink is ATC-E0759 and the rest are ATC-E0757. The walk also refuses a non-UTF-8 entry name (:562-566) and a root that escapes the repository.

The receiver accepts those entries. validate_module_manifest counts only the NON-payload entries against the two-entry rule, and admits any number of payload/** entries at any depth (module_protocol.rs:154-165, with payload_subpath at :130-133). It refuses an entry that names the payload directory itself (:170-179), and any entry with an empty path component (:206-214). The entries land under payload/ in the delivered version directory (§7 gap 10).

The definition

Both sides parse module.kdl. On the push side stage_core_files reads the definition through the shared bounded loader (crates/atc-config/src/module_definition.rs, MODULE_KDL_MAX_BYTES). It refuses the push locally when the definition does not parse, exceeds the cap, or declares a module other than the one pushed. The error is ConfigError::InvalidModuleDefinition (module_package.rs:699-722).

So a definition declares its verbs and a per-verb timeout (verb-timeout "workload start" 120), and it is validated before a byte moves. That property did not change when production moved off build_module_manifest. Only the site did.

The device re-reads the delivered file before every verb it spawns. read_verified_definition checks the file against the digest the delivery’s validated manifest declared, parses it, and refuses a definition that names a different module (crates/atcd-applier/src/agent_push_flow/module_verb_trigger.rs:223-267). Two declarations then decide what happens. The declared verb set gates every spawn, and an undeclared verb is skipped rather than run (may_spawn, :605-627, called at :763). The declared verb-timeout becomes that verb’s deadline (:798-805). A drained verb has no retained manifest to compare against, so it parses the delivered file unverified: DefinitionEvidence::OnDiskOnly (:292-299).

The rest of a definition still reaches no device code. A definition also declares kinds, their field schemas, their production recipes, its rollback contracts, a run-as uid, and managed paths. The device parses and validates all of them, and then reads none of them.

3. Artifact kinds and production recipes

ProductionRecipe has three variants — walk-directory, oci-pull, oci-pull-erofs-convert (crates/atc-config/src/module_definition.rs:128-133).

All three produce. Only two reach a consumer on the device.

• A recipe is a field of ArtifactKind::FromModule only (kdl_schema.rs:61-65). ModulePackage is a core kind and has no recipe field. Its payload comes from the payload node instead — sugar for the same walk-directory producer, run by prepare_module_package.

• walk-directory is performed by prepare_recipe_artifact (crates/atc-config/src/module_package.rs). It is the SAME walk the payload node runs. Same openat ladder, same staging root. One producer, two callers, not a second directory walker.

• oci-pull is performed by prepare_oci_layout_artifact (crates/atc-config/src/module_package_oci_pull.rs). It is the SAME walk again — over a materialized OCI image layout instead of a repository directory. atc-config cannot resolve an image ref (it depends on neither atc-oci nor atc-store), so the two halves are split: atc_app::ota_push_resolve_recipes resolves the artifact’s source ref against the laptop cache and builds the artifact’s index.json, and this crate walks the cached blobs and writes the two layout files beside them. The resolution is the SAME one a core oci artifact gets — resolve_oci_source_digest, then the cached-platform check — so there is no second registry client.

• oci-pull-erofs-convert is oci-pull plus ONE GATE, in the same atc_app::ota_push_resolve_recipes: every layer of the resolved manifest must already carry the EROFS media type, the all-or-nothing rule the device’s own desired-set computation applies. It converts nothing; a tar-layered image is refused with atc oci import … --layer-format erofs. The privileged half it was once blocked on is not needed and not rebuilt: mounting is done by crates/atc-mount-helper, a root service with an identifier-only protocol that shipped separately.

  The bridge still refuses it, and that is deliberate. The producer stages an image the device cannot yet find: a module-defined artifact materializes into the channel artifact tree, while the mount reconciler and the digest resolver read oci_tags, the by-digest paths, and mount leases. Until that registration lands, accepting the recipe would report a successful push over an image nothing mounts.

So “which recipes can a module use” still has two answers: walk-directory and oci-pull. The built-in podman kind declares oci-pull-erofs-convert, so podman web-img is still refused.

What an oci-pull artifact does end to end

use registry
image web-img {
    source "ghcr.io/acme/web:1.4"
    platform "linux/amd64"
}

with registry resolved from .atc/modules/registry/module.kdl declaring an artifact "image" whose recipe is "oci-pull".

The field rules are oci’s, not walk-directory’s, and the asymmetry is the recipe’s rather than a policy choice per kind. source is required on both, and it DENOTES different things: a repository-relative directory for one, a registry image reference for the other. platform is REQUIRED here and FORBIDDEN there — one ref can hold an image per architecture and a device has exactly one, so omitting it would push whichever architecture happened to be imported; a directory walk has no registry to select from at all (ArtifactKind::platform_rule, enforced in both directions by kdl_parse_core_nodes).

The image must already be in the LAPTOP CACHE, exactly as for a core oci artifact: atc oci import <ref> is the pull, and a miss is CliError::OciNoCachedImage carrying that command as its hint. atc ota push does not reach a registry.

What lands on the device is the image LAYOUT — oci-layout, index.json, and blobs/sha256/<digest> — at the artifact root, so an oci: transport reference resolves against the delivered directory. The index.json names exactly the ONE ref the artifact declares. It is NOT atc_oci::layout_view’s, which lists every laptop tag pointing at the digest so that concurrent atc oci commands do not clobber each other’s view: that is right for a shared cache view and wrong for an artifact, where it would make target_sha a function of the laptop’s tag bookkeeping. Nothing is written into the cache, so --dry-run resolves one of these without touching store state. Not an unpacked rootfs: a rootfs is made of symlinks, device nodes and uid/gid-owned entries, the walk refuses the first two by design (ATC-E0756/ATC-E0757), and the manifest can carry neither ownership nor whiteouts. Unpacking belongs to whatever consumer has the privilege for it, and no such consumer exists on the device yet (§1).

What a walk-directory artifact does end to end

use assets
files web-assets {
    source "web"
}

with assets resolved from .atc/modules/assets/module.kdl (the repo-vendored rung, crates/atc-config/src/module_resolution.rs) declaring an artifact "files" whose recipe is "walk-directory".

1. resolve_module_defined_artifacts (module_artifact.rs) resolves it to a ModuleDefinedArtifactConfig, parsing the <module>.<kind> spelling (assets.files) against the kind grammar at that one boundary.
2. PushManifest::module_defined carries it. It does NOT go into Config. That is §S1b’s ruling for the module core kind, not a second one. Config is atc.toml’s schema, [[bundle]] and [[oci]] are its only sections, and no TOML section name can spell <module>.<kind>.
3. resolve_push_set runs the recipe ONCE per push, through the same ModulePackageCache a module package uses, so a retry does not re-run a producer.
4. The produced tree is staged at the ARTIFACT ROOT, with no payload/ prefix. That prefix is module-package layout. The wire prefix is now the caller’s parameter, not the producer’s constant.
5. build_from_module_artifact_plan builds ArtifactPushPlan::FromModule. build_artifact_decl puts the spelling on ArtifactDecl.module_defined_kind and leaves the enum kind UNSPECIFIED. Those are the two halves resolve_decl_kind reads.

source is REQUIRED for a walk-directory kind (ArtifactKind::source_required). It is derived from the recipe, not from provenance, for the reason name_shape is. A ProductionRecipe carries no path of its own, so the artifact’s own source is the recipe’s only parameter.

What an operator SEES. ArtifactVersions, ArtifactStatus, DeliveryMember and PerArtifactResult each carry a module_defined_kind beside their ArtifactKind, exactly as ArtifactDecl does on the push path, and artifact_kind_from_wire reads the pair back. So atc ota ls, ota status and ota deliveries name a landed artifact assets.files.

The enum half alone still means bundle when the spelling is empty, which is the v1.0-client case and is deliberate.

atc ota df, push rollback messages and orphan warnings read the pair too (#5127). pb::DfArtifact and pb::ActiveArtifact each gained a module_defined_kind field, and the enum-only converter artifact_kind_to_pb is deleted, so no site can take the first half of the pair by accident.

4. Which verbs actually fire

CommitVerb has six variants and DeferrableVerb three (crates/atc-module/src/verb.rs:87-93, :24-31) — nine verbs in the vocabulary. Production invokes four, from four call sites, all of which go through spawn_commit_verb (module_verb_trigger.rs):

The one module package left in the tree does declare another of the verbs: probe lists workload status (crates/atc-module/fixtures/probe-module/module.kdl). The podman and systemd manifests, which also declared verbs production never spawns, were deleted as unreachable (issue #3665). A declaration is not a trigger, and no production code path spawns those verbs, so declaring one changes nothing. Every one of the four that does fire is gated on the module’s own declaration: an undeclared verb is skipped rather than spawned. atc_module::verb_declaration::default_action states what the skip means (may_spawn, module_verb_trigger.rs). A skip is counted, under atc.agent.delivery.module_verb_skips.

Every committed delivery also books one delivery-level verdict, under atc.agent.delivery.workload_health_verdict. It is tagged verdict (healthy|unhealthy|unasserted). It answers a different question from the per-verb counters above. Did anything report that this delivery’s workload came up?

A delivery that activates no module books unasserted. So does one whose module declares no workload health. Per verb an undeclared health check is a benign skip. Per delivery it is nobody looking, which is issue #3106. The pass emits on every path, so an absent series means the daemon did not report. It never means a healthy fleet (spawn_health_verbs, module_verb_health.rs).

Scope, stated rather than implied. spawn_health_verbs has ONE caller, and that caller is the applier session (crates/atcd-applier/src/session.rs:878). So the health pass covers the starts that session ran, and FOUR production paths start a module with no health pass at all (module_verb_health.rs:122-142):

A recovered first activation, a replacement start and a pulled delivery’s start each get an exit status and nothing else. The baked case does reach this pass and folds unasserted into the delivery verdict, so that skip is a number. The other three never reach the pass, so they book no verdict at all.

The PUSH plane is therefore this counter’s whole subject. A pulled delivery books nothing here. Read atcd.pull.delivery_commit_total beside it to see that half as a number.

Its subject is every COMMITTED delivery, which includes a stage-only push. Such a push terminalizes Committed and activates nothing, so it books unasserted and no workload was ever due to come up. That is true, and it does dilute the ratio on a fleet that stages before it switches. The question the counter answers survives it: does this device ever produce a healthy verdict at all?

atc_module::commit_coordinator, which did map a CommitStep to stop/start/health, had no production caller and is gone. It was deleted in PLAN-D slice 5 along with sequence.rs’s delivery-ordering half (ArtifactName, DeliveryMember, DeliveryPlan), because it sequenced verbs around the store flip and production does an atomic flip followed by best-effort post-commit verbs. Recover it from git history at commit f8f193746, the last tree that still contained it. (It is unrelated to the atcd-applier commit coordinator, which batched whole-blob CAS commits, never touched a module verb, and was deleted in #7439.)

How a verb is invoked, which is the whole ABI a module author has:

• argv is <exe> "<verb>", for example <exe> "workload start". It is one token, space included (crates/atc-module/src/spawn.rs:443).
• the environment is env_clear()ed down to exactly PATH (pinned to atc_lifecycle::hook_env::HOOK_PATH) and ATC_MODULE_CONTEXT (crates/atc-module/src/spawn.rs:447-448, crates/atc-module/src/invocation.rs:98-107).
• stdin is /dev/null; stdout and stderr are inherited from atcd (crates/atc-module/src/spawn.rs:449).
• the child leads its own process group, which is SIGTERMed then SIGKILLed if the verb outruns its timeout (crates/atc-module/src/spawn.rs:453, :232-283). A start that blocks for the workload’s lifetime gets killed along with everything it spawned.
• the timeout is what this module’s module.kdl declared for this verb, clamped by a hardcoded 600s device cap (DEVICE_TIMEOUT_CAP_SECS and DeviceTimeoutCap, module_verb_trigger.rs, DEVICE_TIMEOUT_CAP_SECS). A definition that declares no verb-timeout for the verb gets 60s (DEFAULT_VERB_TIMEOUT_SECS). The cap is not configurable: nothing reads the root-owned device configuration that would raise it.
• exit 0 is success; every nonzero code is a plain failure. 21 means “defer” only for the three payload-plane verbs, none of which ever runs (verb.rs:59-65 vs verb.rs:124-129).

When start fires, and when it does not: after the delivery commits and survives every veto window. crate::agent::finish_applier_session_inner calls session::start_activated_modules_after_commit, which re-reads the delivery row and starts nothing unless that row is durably Committed (crates/atcd-applier/src/session.rs:783, :825-845). A delivery that merely carries a module forward spawns nothing. A delivery that re-lists the same sha does re-run start (module_verb_trigger.rs’s module doc states it, and DeliveryActivation’s changed flag is what distinguishes carried-forward from re-listed-unchanged). A failure is logged and counted, never returned: the push still reports success and the module simply never started (module_verb_trigger_start.rs:81-89). The push does not retry it. A later housekeeping sweep does, and gap 7 states which shape gets how many attempts. Do not intervene by hand before reading that gap.

5. Workloads

Declare them in atc.kdl and push. The KDL bridge used to refuse any file with a workload node; it no longer does, so the chain below is reachable from a producer file:

• WorkloadDeclaration rides PushPreflight behind the workload-declarations-v1 capability, with identity fields typed and kind-specific fields as one canonical fields_json_v1 JSON document the daemon never interprets (crates/atc-protocol/proto/atc.proto). The client gate fails closed against a daemon that does not advertise the token.
• The daemon validates the declarations at preflight and persists them against the delivery inside the commit transaction (delivery_workloads), as an idempotent full-set replace.
• Workload declarations cross the wire, are persisted per delivery, and drive the trigger: it spawns one workload start per DECLARED workload, with the workload’s own name as the context’s subject.name and its declared fields as config (spawn_start_verbs_for_module in module_verb_trigger.rs). The module-name subject survives only as the fallback for a module that declares no workloads at all.

The practical consequence for a module author: subject.name is the workload your invocation acts on — unless the delivery declared none for your module, in which case it is still your module’s own name. A module that wants to address several units of behavior individually declares one workload per unit.

Two workload shapes are refused. Each is named in the push error, and each is emitted per declaration.

• A workload with a hooks block. A workload hook runs at deactivate.* and activate.*, in the passes that stop and start the workload. Nothing carries a workload hook to a device yet, so the push refuses the block. If the behaviour fits the payload, put a hook on the artifact that from names instead.
• A workload of a module-defined kind, where a child does not match that kind’s schema. A child fails to match for four reasons. The schema does not declare its name. Its value has a different type than the schema gives it. It repeats a name. It is not a single-argument leaf node.

Those are the only two workload scans in unbridgeable_findings. A kind that declares no fields at all accepts a block carrying from alone, and refuses any other child by name.

A module-defined kind whose block matches its schema is not refused. Its declared values ride the push as a fields_json_v2 document. The field names are the keys, in alphabetical order. A built-in kind stays on fields_json_v1 and keeps its declaration order. The two arms split the kinds by who defines them, so neither replaces the other.

The daemon must advertise the workload-fields-json-v2 token before atc sends a v2 declaration. The push refuses before any bytes move if it does not.

A third wall is often met alongside them but is not a workload rule: an artifact of a module-defined kind, such as podman web-img, is refused as an artifact. A workload is never refused for what it consumes, so a container workload whose from names a plain bundle bridges and pushes — it is use-podman.kdl’s podman ARTIFACT that stops that file, not its container workload.

Editing only your declarations does not deploy them. A push whose content the device already holds short-circuits, and a stage-only push does not activate, so neither writes new declarations. To deploy a changed workload set, change an artifact in the same push, or switch to a delivery that already carries the declarations you want.

atc ota push --dry-run predicts which case you are in. Below the workload lines it prints a declarations: line: whether this push would apply them, and why not when it would not. --json carries the same answer in push_preview.workload_declarations_disposition.

Removing the last workload from atc.kdl is a destructive push. The file stays authoritative about its own workload set, so an empty one CLEARS the device’s declarations rather than leaving them alone — unlike an atc.toml push, which cannot declare workloads and carries them forward. The dry-run names that case outright, and --json reports it as workload_declarations_present: true with an empty workloads array.

A pull now carries workload declarations too (#4610). They ride INSIDE the signed delivery manifest, on DeliveryManifest.workloads, so the ed25519 signature the publisher already makes covers them. There is no second signature, no new server column, and no unsigned sibling carrier.

atc channel deliver --workloads <atc.kdl> is what authors them. The producer file is authoritative about its own workload set, exactly as it is on the push plane, and fields_json comes from the encoder the push plane sends (to_workload_wire_v1), so both planes ship the same document. The server validates the set after it verifies the signature and refuses a publish that cannot work — a from naming an artifact the delivery does not carry, a name colliding with one, a fields document that is not a JSON object.

The device extracts the declarations from the VERIFIED bytes and threads them into the same commit transaction the push plane uses. So the rules are now one set of rules on both planes:

• Some, the empty set included, REPLACES the delivery’s rows. Emptying the workload set in atc.kdl and publishing CLEARS them, as an empty push does.
• None — a publish that omitted the flag, or any UNSIGNED delivery — carries the prior active delivery’s rows forward. An unsigned delivery attested nothing, so it can never clear a set.

The parity claim is checked rather than asserted: delivery_pointer::tests::workload_parity_tests drives the same declaration policy through BOTH commit entry points and compares the resulting rows, including the authoritative clear.

An atcd that predates #4610 cannot apply a declaration, so the server refuses to publish one to it. DeliveryManifest does not deny unknown fields, so such a device verifies the signature, ignores the new field, commits “said nothing”, inherits its prior rows, and reports a successful convergence. A publisher who CLEARS a set would get silent false success on it: the workload keeps running, nothing errors, and nothing logs.

So the device reports declares_workload_declarations at check-in, and publish_delivery refuses a workload-bearing release to a channel holding a device that has never reported it. The refusal names the devices to upgrade. It is the same shape, and the same argument, as the [lifecycle.progress.*] gate.

Two properties of that gate are deliberate:

• An authoritative CLEAR arms it. The empty set is the case that breaks on an old device, so a gate keyed on a non-empty set would wave through the one publish that matters.
• A channel with no devices is REFUSED. An empty candidate set is the state every channel is in before its first device checks in, so “nobody objects, allow” would admit precisely the publishes made during bring-up, and the first device to enrol is the one that pulls them. Over-refusing costs a retry and says so; under-refusing costs a workload that keeps running after it was deleted.

A delivery that declares no workloads is untouched by the gate, which is every delivery published before #4610.

A workload on a built-in module STARTS. The trigger only walks the push’s changeset, and a built-in module — systemd, podman, nerdctl — is never in it. So the commit derives a lifecycle for it instead, from the declaration set itself: the version is baked-<digest of the declared workloads>, and the ordinary journal diff turns a first declaration into a first activation, a changed set into a retarget, and a removed set into a departure. The drain then runs the verbs out of the baked rung.

The version is the DECLARATION SET, not the module’s code. The baked executable changes only when the image changes, which no transaction observes and no journal records — a version derived from it would change with nothing to write the transition, and the previous version’s workload stop would be owed by a row nobody ever writes.

A lifecycle is not installed code. A module the image does not bake still gets one, and atc.agent.delivery.baked_module_resolutions is what says so. The counter atc.agent.delivery.workload_declared_untriggered now reports the narrower case: a declaration that NO lifecycle covers, which is near-always zero and is the regression detector for the derivation.

6. What a module can do at the floor

There are no grants. The privileged half of the module system — grants, the device executor, UserProvisioningHost — was deleted on an explicit ruling that it will not be built (PR #2303; recovery SHA d01caf5c6e682e0cd861022e6770cdf316c3fc25). ProcessVerbSpawn authorizes nothing.

A verb runs as the device’s hook user (hook_run_as_user, default atc-hooks), the same identity and the same resolution as a lifecycle hook. Before execve, the verb’s child drops to that user and clears its ambient, permitted, effective and inheritable capability sets. The hook spawn uses the same function, atc_process::drop_privileges_in_child. This matters because under hook isolation the atcd unit grants AmbientCapabilities=CAP_SETUID CAP_SETGID CAP_KILL, and ambient capabilities survive execve. If the drop fails, the verb does not run, and the error names the failed step. atcd installs the identity at start (atcd_applier::install_module_verb_identity). A process that never installed one refuses every verb.

Without hook isolation (hook_run_as_user unset, or an explicit --root install) a verb runs as atcd’s own uid with no switch, as a hook does then.

What a verb is given:

No mounts, no chowns, no created users, no managed paths outside that subtree, no capabilities. A module that needs privilege must reach something that already has it on the device (a systemd unit, a rootless podman socket) using nothing but its own uid.

The context document at ATC_MODULE_CONTEXT carries module, subject, config, payload_root, payload_manifest, version, delivery_id, state_dir, scratch_dir, records_path. Two of those are not what their names suggest, per the fields builder’s own doc (module_verb_trigger_fields.rs:27-40): config is null unless the invocation’s workload declared fields, in which case it is that workload’s declared fields_json_v1 document verbatim — the daemon never interprets it. payload_root names <version-dir>/payload/ for a module that declared no workloads, which materialize creates for every module package whether or not one shipped, so a module that walks it finds an empty directory rather than a missing one; for a declared workload it instead names the version directory of the artifact that workload consumes (from_artifact), which is a different artifact entirely. payload_manifest lists the delivered payload files when there are any, and stays Opaque for a payload-less module — deliberately, so an already- delivered executable that predates payload support can still deserialize its context.

7. The gap list

Ordered by how likely a user is to hit it.

1. A repo that holds atc.kdl beside atc.toml reads atc.kdl (#6337). atc prints a note that names the ignored atc.toml and ATC_KDL_CONFIG (manifest_format::shadowed_manifest_notice), and atc config lint says the same (ATC-E0598). ATC_KDL_CONFIG=0 reads the atc.toml instead (§2).

2. A workload declared for a built-in module now STARTS, provided the image really bakes that module. The trigger still walks the push’s changeset and no built-in module is ever in it, so the start does not come from there: the commit derives a baked lifecycle from the declaration set, and the drain runs the first-activation start out of the baked rung (§5). What remains of this gap is the module the image does NOT bake. The lifecycle is derived from the declarations alone, so such a module gets one anyway, its code root is missing, and the start stays owed rather than failing loudly. atc.agent.delivery.baked_module_resolutions is the signal: the device resolves every distinct module named by a declaration outside the changeset down atc_module::resolution’s ladder and tags the result baked_verified, baked_unverifiable, delivered, absent, or resolve_failed. atc.agent.delivery.workload_declared_untriggered counts the narrower residue: a declaration no lifecycle covers at all. Two narrower shapes are still refused outright at the bridge: a workload with hooks, and a workload of a kind with no compiled-in field schema. A module-defined artifact kind is refused too, but as an ARTIFACT (gap 6). No workload rule looks at what a workload consumes.

3. Five of the nine verbs never fire. That is one entry and two different problems. The four payload-plane verbs are blocked behind other gaps. workload status is not blocked, but it has nowhere to put an answer.

   The payload plane has its subject and its spawn paths. It has no trigger point. Two walls stood here and both are down.

   The first was the subject. payload install and the three deferrable verbs act on an artifact, never on a workload, and check_subject_plane refuses a cross-plane subject (crates/atc-module/src/context.rs). No production code could build an artifact subject: every VerbSubject variant built a Subject::workload. VerbSubject::Artifact now carries one (module_verb_trigger.rs), built from a module-defined member by ModuleDefinedMember::from_committed_member (module_verb_trigger_payload_plane.rs), which reads the DEFINING module off the <module>.<kind> spelling.

   The second was the spawn API. VerbRequest admitted only a CommitVerb, and spawn_commit_verb always called ModuleContext::for_verb, which REFUSES payload install — install must carry a writable version root. spawn_payload_install_verb supplies that root, and spawn_deferrable_verb reads exit 21 as a defer rather than as a failure.

   What is left is the four TRIGGER POINTS. Each one changes the delivery state machine, so they land in their own lane: payload install at the Install gate, before the commit stage barrier; payload stage and payload remove on their sweeps; payload reconcile on its pass. Until then the seam has no production caller and no payload-plane verb runs.

   workload status has a live subject and no answer channel. Workloads reach the device (§5), so the subject exists. The verb reports on stdout, which atcd inherits and no code captures (crates/atc-module/src/spawn.rs, the Command builder at :437-452). The records file is the design’s channel for a structured result, and parse_records_file has no production caller (crates/atc-module/src/records.rs:11-15). Both module executables in the tree print English to stdout and exit 0 whatever they find (examples/heartbeat-module/modules/heartbeat/heartbeat, crates/atc-module/fixtures/probe-module/probe-module). The exit code therefore reports nothing about the workload.

   The reader is absent too. The design names atc status and check-in reporting as the consumer (docs/internal-site/src/design/2026-07-30-update-modules-design.md:561). StatusReply carries no module field and no workload field (crates/atc-protocol/proto/atc.proto:1895). Check-in telemetry is per channel and per delivery (ChannelTelemetry, crates/atc-server/src/http/checkin_wire.rs:83). So this verb needs an answer channel, a request path, and a reader before a spawn means anything. Spawn it without them and it writes one journal line, then books an exit code that is 0 by construction.

   The other four verbs do fire (§4).

4. module.kdl declares more than the device reads. The device honours the declared verb set and the declared per-verb timeouts. The kinds a definition defines, their field schemas and recipes, its rollback contracts, its run-as uid, and its managed paths reach no device code (§2).

5. All three recipes produce. prepare_recipe_artifact performs walk-directory and prepare_oci_layout_artifact performs oci-pull, both reusing the payload node’s own walk; oci-pull-erofs-convert is that second producer behind an all-layers-are-EROFS gate (§3).

   What oci-pull-erofs-convert still lacks is the DEVICE half, and it is not the privileged one. Mounting already ships, in crates/atc-mount-helper and the atc-oci-reconcile desired-state pass, so PR #2303’s ruling on the privileged module framework does not have to be reopened. The gap is unprivileged glue: the delivered version must be published into the cache view the reconciler and atcd ctl podman run read. Until then the bridge refuses the recipe by name (§1).

6. A module-defined artifact kind pushes, lands, and reports for both bridgeable recipes. The podman ARTIFACT kind is still unusable, and now for one reason rather than five: its recipe is oci-pull-erofs-convert, whose device half gap 5 names.

   All five walls this gap used to list are down. The bridge refusal is narrowed to the recipe (§1). PushManifest::module_defined resolves such an artifact and ArtifactPushPlan::FromModule plans it. The daemon reads the kind off ArtifactDecl.module_defined_kind on both paths (atcd::preflight::resolve_decl_kind, atcd_applier::artifact_kind::pb_artifact_kind_to_store). The device stores the spelling verbatim. artifacts.kind has no CHECK, and the version directory carries no kind segment (crates/atc-store/src/layout/channel.rs), so the . in <module>.<kind> never reaches a path.

   Observation was the fifth wall. pb_kind_to_label read the wire enum alone and mapped UNSPECIFIED to Bundle. So every atc listing called a correctly-stored assets.files artifact a bundle. That was the fail-open shape of the receive side, on the report side.

   ArtifactVersions, ArtifactStatus, DeliveryMember and PerArtifactResult now each carry module_defined_kind beside the enum. The daemon fills both halves through atc_protocol::artifact_ident::artifact_kind_to_wire, and artifact_kind_from_wire reads the pair back. KindLabel is deleted, so the views hold the real ArtifactKind. UNSPECIFIED with an EMPTY spelling still reads as bundle. That is the v1.0 client, not an unknown kind.

   Three more readers took the enum alone, and all three read the pair now (#5127). §3 names them, the two wire fields they read, and the enum-only converter deleted with them. Nothing is left of this residue.

7. A failed start is RETRIED. The bound depends on which start failed. A start that runs and fails leaves the module down at that instant. A later housekeeping sweep re-runs it, at most once per sweep.

   A REPLACEMENT start, which a retarget journals a row for, gets at most five attempts (MAX_START_ATTEMPTS, module_verb_drain/retry.rs:48, with the gate at :112). Nothing on a device can tell a transient failure from a permanent one, so the drain does not classify. It bounds. What is left of this half is the module that fails all five: its row is retired and atc.agent.delivery.module_drain_unrecoverable fires once.

   That series is what an operator alerts on. Non-zero means one module will not come back without a new activation. atc.agent.delivery.module_drain_stranded still fires per failed attempt, so it reads as an outage RATE rather than as a count of distinct modules.

   A FIRST activation gets no such bound, because it journals no row to count attempts on. Migration 0044 records the obligation, the sweep leases it, and a start that fails leaves it unsettled. The next sweep takes it again (module_verb_drain/first_activations.rs:489-499). The module ABI defines workload start as convergent, which is what makes the unbounded retry safe. The lease re-stamps every obligation it reads. So one module that always fails moves to the back of the queue and starves nothing (crates/atc-store/src/module_journal_first_activation.rs:99-174).

   The backlog is atc.agent.delivery.module_drain_first_activation_owed, a gauge. A value that rises and falls is ordinary. A value that does not fall is the one to look at.

8. A stranded retarget RECOVERS by itself. A drain that ran a module’s workload stop and then failed to start the replacement used to leave the journal row claimed and unexecuted, which every later sweep skipped; the only way back was to activate the module again by hand. Migration 0055_module_deactivation_stop_stamp records stopped_at when the stop verb SUCCEEDS, which is the one fact that tells that state apart from a drain that died before its stop. A stamped row has its start re-run under gap 7’s bound. An unstamped one is still skipped forever, and there are three: the crash, a stop that FAILED, and a version directory that is gone. The failed-stop case is the one that looks like this gap and is not — the OLD version is still up and serving, so a retry would add the replacement beside it rather than recover anything. Verified by the drain’s unit tests (module_verb_drain/mod.rs, “A stranded retarget is RETRIED”). A PUSH no longer reaches this path for a module with workloads: its workload transition fails the push and the rollback starts the old workloads again, proved on a device by crates/atc-cli/tests/baked_module_start_recovery_e2e_tc.rs.

9. One producer does not drain its own teardown. The admin switch, the post-flip veto rollback, the per-artifact user activate, and the pull commit each run their stops at once. The pull commit reaches that point through PendingModuleDrain::spawn, from pull_poller::reconcile. The standalone delivery rollback journals its rows and leaves them to the housekeeping sweep. That sweep’s grace is one hour plus up to one sweep interval (AGED_DEACTIVATION_GRACE in module_verb_drain/drain.rs, and module_verb_drain/mod.rs’s producer table). The sweep reaches the first-activation starts too: migration 0044_module_first_activation_obligation records each one, and the sweep leases the marker. So the cost is latency, not loss, for both halves. That is stated once, on drain_producer_receipt in module_verb_drain/drain.rs.

10. A module package carries module.kdl, one executable, and whatever a payload "./dir" node walks in. Still no second binary: the package’s own directory holds exactly the two files, and payload entries land under payload/, which is read-only on the device.