Remote access without a VPN

atcd carries its own reverse tunnel. The robot dials out to your control plane over wss://, and you open a shell on it from wherever you are. There is no mesh VPN, no second vendor’s agent, and nothing listening for a connection on the device.

It is off until you configure it, and configuring it is one TOML table. Delete the table and the tunnel is gone. Any other access you installed, such as plain SSH on a routable address, is untouched, and nothing else about the device changes.

That matters if your customers will not accept a VPN mesh on their network. The usual objection is that nobody can audit who connected when. This path answers it. Every session through your own atc-server gets a record of the principal that opened it, the device it reached, and whether the device end answered.

Read the limits on that. The server records the PRINCIPAL, so two API keys for one principal are indistinguishable. Per-person attribution needs a human identity, or a separate principal for each person. A failed audit write leaves a gap and does not close the session. A standalone wstunnel rendezvous writes no atc-server record at all.

It does NOT make session contents inspectable. The payload is SSH inside TLS, so a packet inspector still cannot read the shell. What it gets instead is one ordinary HTTPS flow to one host you name. It can allow, block, or rate limit that flow by destination, port, and SNI. A mesh VPN gives it neither.

Read what the rendezvous side needs before you plan around this. The device half is ready. The server half runs the rendezvous on one task, and a public deployment without the CloudFront lockdown still 404s the operator route.

How it reaches a robot behind NAT

robot ---- dial out, wss:// ----> atc-server <---- wss:// ---- your laptop
  |                              (pairs the two)                    |
  +--- sshd on 127.0.0.1:22                             ssh --------+

The device opens the connection. The control plane holds it, then hands it to an operator who asks for that device. The connection is a WebSocket over TLS. A firewall that does not terminate TLS sees an ordinary HTTPS flow, and it can classify that flow by destination, port, and SNI. It cannot read the WebSocket framing inside.

An egress-only warehouse firewall can therefore allow it, and you need no port forward. It is not automatic. A proxy that terminates TLS, or a policy that refuses a WebSocket upgrade, still blocks the tunnel.

The device needs no inbound rule. A relayed connection goes to the device’s own sshd, and the operator’s own ssh speaks to it. Today one command opens the tunnel: atc fleet device shell. The other device-touching subcommands still need a route you can already reach.

Before you start

The device must already hold enrollment state. The tunnel authenticates with the device credential that enrollment persisted, and it reads that credential from the store, not from the config. A device that never enrolled logs device is not enrolled yet and retries until it does.

A [server] table is what enrolls a device automatically, and it is also where ca_bundle_path lives. It is not itself the credential. A device enrolled out of band can park with no [server] table. A device with a [server] table that is not yet enrolled cannot.

Warning: nothing enforces a distinct listen address

No part of atc, atcd, or atc-server compares one device’s listen value against another one. There is no fleet-wide register of addresses, and no check runs at any point. Read which of the two cases below you are in before you write the field.

Against atc-server, listen is ignored. That rendezvous never reads the value. It pairs an operator with a device by the UUID that the presented credential resolves to. Two devices may carry the same listen with no consequence. atcd still requires the field, and it must parse as an IP address and port such as 0.0.0.0:2222. A host name is refused.

Against a standalone wstunnel rendezvous, give every device a distinct bind address. That is a different deployment, and not a drop-in one. atcd dials the fixed path /v1/devices/tunnel/events, so the rendezvous must accept it. atc fleet device shell does not work against it either, because that command only speaks to atc-server’s operator route. An operator points ssh at the address the rendezvous published instead. Within that deployment, the collision below is live.

wstunnel keys its reverse listeners by the whole bind address. It REUSES the existing listener when a second device claims the same one, and it does not refuse that device. Two devices with one listen value then share one listener.

An operator who connects to that address reaches whichever device takes the connection first, and neither side reports an error. The operator may then get a shell prompt on the wrong robot. A host key already recorded for the intended robot can refuse the connection first. That is a warning, not a guarantee.

Assign the addresses from a table your fleet owns, and do not copy the example value below onto a second device.

Turn it on

Set the three tunnel keys in atcd’s store. Run this on the device as the account that runs atcd (atc by default):

sudo -u atc atcd settings set \
  daemon.tunnel.server=wss://tunnel.internal.example.com:443 \
  daemon.tunnel.listen=0.0.0.0:2222 \
  daemon.tunnel.forward_to=127.0.0.1:22

• daemon.tunnel.server is the rendezvous the device dials. It must be wss://. Plaintext ws:// is refused.
• daemon.tunnel.listen is what the device ASKS the rendezvous to publish. atc-server ignores it, but the key is required. Read the warning above.
• daemon.tunnel.forward_to is the device-local address a relayed connection is handed to. 127.0.0.1:22 is the default.

A device that accepts direct control takes the same keys from a laptop with atc ota settings set <target> …. An enrolled device refuses that, because it accepts changes only from its server.

Restart the daemon:

sudo systemctl restart atcd

Confirm that the daemon read the table and built a tunnel client:

journalctl -u atcd -b | grep 'atcd tunnel'

A configured device logs one line each time it builds a client:

atcd tunnel: parking on wss://tunnel.internal.example.com:443 to publish 0.0.0.0:2222 -> 127.0.0.1:22

That line comes BEFORE the first dial, so it does not prove that the device reached the rendezvous. A device that cannot reach it prints the same line and then logs atcd tunnel: dial to the rendezvous failed.

The only end-to-end proof is a successful atc fleet device shell against that device. The server-side atc_server_tunnel_park_total metric carries no device or org label, so it tells you that SOME device parked and nothing about yours.

The daemon checks the values at startup, and it names the field that is wrong. It checks that server is a URL and that its scheme is wss. It checks that listen is an IP address and port. forward_to may be a host name and port as well, so localhost:22 is accepted there and refused in listen.

atcd tunnel: disabled — [daemon.tunnel] server "https://rv.example.com" uses scheme "https"; expected wss://

A bad string VALUE leaves the daemon running. It still ships updates, and only the tunnel is off.

Warning: a table that will not DECODE is different, and it stops the daemon. The decode runs before any of the checks above. It fails on a missing server or listen, and on a wrong TOML type such as listen = 2222. A misspelled server or listen is the same failure, because the required key is then absent.

An unknown key ALONGSIDE the required ones is the third case. It warns, the key is not applied, and the daemon starts.

Open a shell

atc fleet device shell <device-uuid>

Add --user when the account on the device is not your local login name. Put a one-shot command after --:

atc fleet device shell 4f1c0e2a-... --user deploy -- uptime

Every word after -- reaches the device as one argument. atc quotes each one, so a space, a brace, a quote, a glob, or a $ inside an argument survives:

atc fleet device shell 4f1c0e2a-... -- jq '{a,b}' /tmp/x.json

--as-ssh-user follows the same rule. It sent the words bare until issue #5414, and the shell on the device re-split them. The command above then ran as jq {a,b} /tmp/x.json.

To run a pipeline, a redirect, or a ~, pass the shell program as ONE argument:

atc fleet device shell 4f1c0e2a-... -- sh -c 'journalctl -u atcd | tail -20'

atc opens the tunnel, publishes it on a loopback port, then runs the system ssh at that port. ssh does the SSH. It records and checks the host key under the device UUID rather than the loopback port. The key therefore survives from session to session instead of churning.

Warning: this is trust on first use. atc passes StrictHostKeyChecking=accept-new, so the FIRST connection to a device UUID accepts whatever key answers. Only a LATER change of key fails the check. If you must be sure of the identity on first contact, put the expected host key in your known_hosts under the device UUID before you connect.

If the device is offline, or if it has no daemon.tunnel.server setting, the command fails before ssh starts. It reports ATC-E0461.

Turn it off

A re-run of atc bootstrap atcd install keeps the tunnel settings. An install writes only the keys the operator states, and --server states only the server.* keys.

To remove the tunnel, unset the three keys and restart atcd:

sudo -u atc atcd settings unset daemon.tunnel.server daemon.tunnel.listen daemon.tunnel.forward_to
sudo systemctl restart atcd

The daemon then dials nothing, and it behaves as it did before the keys existed. There is no package to remove and no service to disable.

What the device trusts

The DEVICE authenticates the rendezvous certificate against the bundled webpki roots, plus whatever [server] ca_bundle_path names. It reads no system certificate store. See Private CA trust.

The OPERATOR half reads the same bundle. atc fleet device shell builds its TLS configuration from atc_cmd_server::trust::rustls_config_for, which carries the bundled webpki roots, plus [server] ca_bundle_path, plus $SSL_CERT_FILE. It reads no system store either.

Warning: check which atc build the operator runs. This half changed on 23 August 2026, in #3548. An atc built before that dialled through tokio-tungstenite’s built-in connector, which trusts the bundled webpki roots alone. Against such a build a rendezvous behind your own CA accepts every device and refuses every operator. Upgrade atc, or give the operator-facing name a publicly trusted certificate.

The two halves still dial different URLs. The device dials [daemon.tunnel] server. The operator dials the control-plane base URL that the atc CLI is configured with. They may be different names, and they must reach the same atc-server process, because the pairing lives in that process’s memory.

So one private CA can now serve both halves. Name it in [server] ca_bundle_path on the device, and in the [server] table the operator’s atc reads. The two files are different files on different machines, and both need the anchor.

You cannot turn verification off. wss:// is required, and atcd refuses the tls_verify_certificate escape that wstunnel offers.

This trust base changed in August 2026. An older build read the device image’s system certificate store instead. Confirm which build a device runs before you plan a trust root around it.

Who may open a shell, and what is recorded

The route always requires an authenticated admin actor. What counts as one depends on a server-side flag, so check which of these your atc-server runs.

With ATC_ORG_RBAC_ENABLED=true, the caller must belong to the org named in the path AND hold admin rank there. A platform admin passes for any org.

Warning: the flag is OFF by default. The check then falls back to a plain server-admin check, and a server admin can open a shell on any device in any org. Turn org RBAC on before you tell a customer that the tunnel is scoped to their org.

The device-membership check runs either way. A device that belongs to a different org than the path names answers 404, byte-identical to the answer for a UUID that exists nowhere.

atc-server writes an admin.device.tunnel.open audit record for a granted session AND for a refused one, once the caller is authenticated. A missing or invalid credential is refused before the audit request exists, so it leaves no tunnel record at all. A failed audit write is logged and does not close an already-open session.

Do not alert on refusals alone. One refused record covers several events, and this list is not exhaustive:

• the UUID exists nowhere
• the device belongs to another org
• the caller does not hold admin rank
• the device is offline
• the device answered, but its forward_to service sent nothing
• the device dropped between the reachability check and the handoff

Only the first three look anything like an attack. An operator who retries an unplugged robot, or a robot whose sshd is stopped, produces a run of refusals that reads exactly like probing. Connectivity alone does not separate these, so read the refusal message before you treat a run as an attack.

The server also counts every park on the atc_server_tunnel_park_total metric. It carries one label, displaced, with three values. Read them before you build an alert on it.

• none: the ordinary park, and the denominator. A fleet with no tunnel-enabled device reports zero here, which is not the same as no displacements.
• closed: this park replaced a registration whose connection had already gone away. A device on a bad network produces these constantly. Do not alert on them.
• live: this park pushed out a registration that still looked healthy. One per robot recovering from a silent connection drop is normal. A run of them for one device means two holders of that device’s credential, such as a disk image cloned after enrollment. Rotate the device credential.

A granted record proves that the device end answered. It does not prove that an sshd answered. Whoever holds the device credential controls that byte stream. Nothing the server can inspect separates a real sshd from a client that writes one frame. Only a host key you verified elsewhere and pinned first closes that gap. The default trust on first use protects later sessions and not the first one.

Dependencies

atcd and the atc CLI contain no Tailscale code and need no Tailscale daemon. On the device the tunnel needs outbound TLS to your atc-server and nothing else. Remote access is a config block you add and delete on its own schedule.

Tailscale does appear elsewhere in this repository. atc-onboard, the tool for the hardware test farm, uses it to reach lab boards. The atc-server terraform also defaults ingress_mode to tailscale, which puts the managed load balancer on a private subnet. That is a hosting choice for our own deployment, not a requirement of the tunnel.

What the rendezvous side needs

The device half is ready. The server half carries one live limit and one shape you have to deploy, both recorded in cloud/atc-server/terraform/tunnel.tf. Read them before you promise this path to anyone.

Both tunnel paths must reach the rendezvous service, and it runs one task. The pairing registry lives in process memory, and a parked device holds a live channel end. So a device parks on whichever task the load balancer picked, and an operator pairs only if their request lands on that same task. With two tasks that is a coin flip, and the operator sees a pairing timeout that looks exactly like an offline robot.

tunnel.tf closes that by ROUTING rather than by pinning: it adds an ECS service at desired_count = 1, its own target groups, and listener rules that send /v1/devices/tunnel/events and /v1/admin/orgs/*/devices/*/tunnel to it. serve keeps both routes in its binary and never receives them, so serve may scale freely. Do not pin serve for the tunnel’s sake.

Raise the tunnel’s capacity with tunnel_task_cpu, never by adding a task. A second rendezvous task returns the failure the routing removed.

The costs are real and worth stating. One task is one point of failure. Its rollout is stop-then-start, so a deploy is a tunnel gap of the target group’s 30-second drain plus task start plus two health checks. Devices redial after it. Uninterrupted tunnel availability needs inter-replica forwarding, which issue #3984 carries.

Unprotected public ingress 404s the operator route. The operator half lives under /v1/admin/*. The load balancer installs a priority-10 rule that answers 404 for that whole prefix. That rule is active for ingress_mode = "public" and for "protected_public" alike, whenever the CloudFront origin-verify lockdown is OFF. Our prod is one such deployment. The atc CLI then reports the device as not found, and atc-server logs nothing, because the request never arrived.

The tunnel rule follows that decision rather than overriding it: where the priority-10 block is active it carries the device path only, deliberately, so the ops block gains no unused hole. A deployment that wants the operator half on unprotected public ingress has to decide to open the prefix.

With the lockdown ON, the block has count = 0, and the tunnel rule carries both paths plus the origin-verify header the lockdown requires. That is our staging shape, and the operator half works there. The lockdown, not the ingress mode alone, is what decides it.

The DEVICE half is unaffected. It parks on /v1/devices/tunnel/events, which no block rule matches.

Until a shared pairing registry and that forward rule land, treat this as an internal-testing path rather than a fleet-wide one.

What this path does not do yet

• It carries ssh only. atc ota push over the tunnel is not wired up. A push still needs a route you can already reach.
• One session per tunnel. A second atc fleet device shell opens its own pairing.
• A park lasts 25 seconds, then the device redials. An operator who arrives during the gap waits up to 3 seconds.