Configuration Management

Overview

Cloud-init configures a machine once, at provisioning time. Configuration management keeps it configured for the rest of its life: you publish an immutable config release, and every machine you have targeted converges to it on its own and stays converged.

Three properties define the model, and everything else on this page follows from them:

• Releases are immutable and content-addressed. A release is a tarball with a sha256 recorded by the control plane. The same bytes that converged your first tier converge your last one — there is no per-environment rebuild, so what you tested is what ships.
• Convergence is a pull. The agent already on the machine asks the control plane what it should be running, fetches the release itself direct from the OCI registry with a short-lived, repository-scoped credential the control plane mints per response, verifies the hash, and runs the engine locally. The control plane authorizes the pull but never carries a byte of it. Nothing pushes state at your machines: no master, no SSH, no inbound connection, no orchestrator process holding the fleet's state in memory.
• The node keeps converging without us. The release, the machine's composed values, and its facts are all on local disk, so drift correction continues through a control plane outage or an agent restart.

Config releases are per-account — one account is one fleet — and are rolled out along two independent axes: Environments (ordered promotion tiers) and Rings (blast-radius partitions). This page covers both, the value composition underneath them, and what to do when a rollout goes wrong.

What a Release Contains

A release is a tar.gz of your engine state tree — for the default saltstack engine, a top.sls at the archive root plus whatever states, templates, and modules it pulls in. The agent points Salt's file_roots at the unpacked release directory and runs the highstate, so top.sls at the root is the entry point.

What is not in the tarball is just as important: no secrets and no per-machine data. Those are composed per machine by the control plane and delivered separately over the agent's mTLS channel (see Config Values). That is what makes the same artifact safe to promote from your first tier to your last.

Each release carries:

Field	Meaning
version	SemVer (v1.0.0) or CalVer (2026.08.18.104500), unique per account. It must be orderable: an unpadded form such as 2026.8.25 is refused, because nothing could ever compare it against another release
engine	saltstack (the default). ansible is a reserved value — see Requirements Today
tarball_sha256	Computed server-side, never client-supplied; the agent verifies it after download. The tarball is stored in an OCI registry as a blob, and the digest in the reference the agent is handed is this sha256 — the locator and the integrity check cannot drift apart without the pull itself failing
source	git, always — with git_ref and the resolved git_sha for provenance. There is one way in, on purpose (see Creating a Release)
is_latest / is_canary	Per-account markers used by automatic rollout policy
fixes	Set on a hotfix — the release this one repairs, which is what front-loads it
yanked	A withdrawn release. Never served, never rolled out, cannot be re-marked latest

Where a Machine Sits: Environments and Rings

Every machine that takes config has exactly two placements, and it needs both: with no environment or no ring assigned, the control plane serves it no release at all.

Environment is the promotion tier and the rollout's cursor. Each has a promotion_order, and a rollout walks them low → high, one at a time: a tier must converge healthily before the next one is served anything.

Ring is the blast-radius partition — quorum groups, roles, regional shards. Rings have no order. Within the active environment, every ring converges in parallel, each on its own time-based ramp and its own failure threshold. Rings are shared across environments; the same db ring exists in staging and in prod.

A worked example. Say your fleet is two tiers and three rings:

Environment	promotion_order	Rings converging in it
staging	10	web, db, edge
prod	20	web, db, edge

You start one rollout. It anchors on staging, and web, db, and edge each begin ramping at the same moment, independently. When every machine in staging has applied the release and none is failing, the cursor promotes to prod and the three rings ramp again from zero — re-anchored, so a slow ring in staging does not shorten prod's ramp. If the db ring in staging accumulates failures past its threshold, db halts while web and edge keep going — and because a halted ring blocks promotion, prod is never served the release.

Create them over the API. Slugs are unique per account and immutable after creation:

curl -X POST https://app.durantic.dev/api/config/environments/ \
  -H "Authorization: Bearer $DURANTIC_TOKEN" -H 'Content-Type: application/json' \
  -d '{"name": "Staging", "slug": "staging", "promotion_order": 10}'

curl -X POST https://app.durantic.dev/api/config/rings/ \
  -H "Authorization: Bearer $DURANTIC_TOKEN" -H 'Content-Type: application/json' \
  -d '{"name": "Database", "slug": "db", "failure_count_threshold": 3}'

Assign a machine by slug ("" unassigns), and read its placement back from the same resource:

curl -X PATCH https://app.durantic.dev/api/provisioning/machines/$MACHINE_UUID \
  -H "Authorization: Bearer $DURANTIC_TOKEN" -H 'Content-Type: application/json' \
  -d '{"environment_slug": "staging", "ring_slug": "db"}'

The machine list and detail responses expose environment_slug, ring_slug, applied_config_release_uuid, and config_paused.

The Ring Ramp and the Failure Threshold

A ring's wave_ladder is a time-based schedule of {after_seconds, pct} steps, anchored at the moment the current environment became active. The default ladder is:

[{"after_seconds": 0,    "pct": 1},
 {"after_seconds": 300,  "pct": 10},
 {"after_seconds": 1800, "pct": 50},
 {"after_seconds": 7200, "pct": 100}]

Each machine self-elects. It hashes (machine_uuid, release_uuid) into a stable bucket in [0, 100) and is eligible once bucket < pct. Two consequences worth internalising:

• Cohorts are nested and monotone. pct only grows, so the 1% set is a subset of the 10% set. A machine never falls back out of a rollout as it advances.
• Every release reshuffles. The hash mixes in the release UUID, so each release canaries a different 1% of the ring. You cannot pin which machine goes first, and the same unlucky nodes do not always take the first hit.

failure_count_threshold (default 10) is the number of distinct failing machines in that ring, in the active environment, that trips its auto-halt. Failures are counted by latest-result-per-machine, so one node flapping counts once and a node that recovers drops out of the count by itself. Setting it to 0 disables auto-halt for that ring.

Both fields are PATCHable on a ring, alongside name and config_values (slug is immutable):

curl -X PATCH https://app.durantic.dev/api/config/rings/$RING_UUID \
  -H "Authorization: Bearer $DURANTIC_TOKEN" -H 'Content-Type: application/json' \
  -d '{"wave_ladder": [{"after_seconds": 0, "pct": 10},
                       {"after_seconds": 600, "pct": 100}]}'

A ring also accepts max_in_flight, min_settle_seconds, and stall_timeout_seconds on the same PATCH. They cap how many of its machines converge at once instead of letting the whole wave go in parallel, and they are the subject of Kubernetes Cluster Upgrades; left at their default 0 they change nothing about the behaviour described here.

Config Values: The Five Layers

Your states should not hard-code environment-specific data. Config values are the data half of a release: a JSON object the control plane deep-merges from five layers per machine and delivers with the assignment. For the Salt engine they arrive as pillar, so a state reads them as {{ pillar['nginx_workers'] }}.

Lowest precedence first:

#	Layer	Where you set it
1	Account defaults	Account configuration (fleet-wide baseline)
2	Environment	config_values on the environment — this is how you vary staging from prod
3	Ring	config_values on the ring
4	Machine role	config_values on each role, applied in merge_priority order
5	Machine	config_values on the machine (highest precedence)

Dicts merge recursively; a higher layer wins on scalars and on type mismatches. Lists follow the account's config_list_merge setting — replace (the default, higher layer wins) or concat.

Secrets and Variables

Config-value strings may reference account secrets and variables with the same syntax as a cloud-init role: {{ secrets.NAME }} and {{ vars.NAME }} (see Secrets & Variables).

They are resolved server-side, per machine, while the assignment is built, and travel to that one machine over its mTLS channel. They are never in the release tarball, so the artifact stays environment-identical and shareable across tiers. Rotating a secret re-renders and re-converges the affected machines on its own — no manual force needed.

A missing secret takes the machine out of the rollout

A reference to an undefined secret or variable does not crash the machine's agent config and is not silently ignored: the machine is recorded with a render error, its row in the rollout status shows errored, and it is excluded until you fix the reference. A machine stuck in errored blocks its environment's promotion, which is the point — it makes a broken release obvious instead of quietly partial.

Secret values that the engine echoes into its own output are redacted server-side (***REDACTED***) before the apply result is stored. Treat that as a useful defence, not a guarantee: it does not sanitise the node's own local logs.

Facts

Alongside values, the assignment carries facts — read-only, control-plane-authoritative data about the machine's placement that the node cannot know about itself. The agent materializes them as Salt grains under the durantic namespace, so your top.sls can target on them:

{% if grains['durantic']['ring'] == 'db' %}
  - states.postgres
{% endif %}

The facts are machine_uuid, is_gateway, account, ring, roles, region, datacenter, mesh_network, and wg_ip (empty values are omitted). They never contain secrets, because they get logged.

There is no environment fact

Facts carry the machine's ring, not its environment. To vary behaviour by tier, put the difference in the environment's config_values layer and read it from pillar — that is what layer 2 is for.

Creating a Release

Building from git is the only way a release comes into existence. There is no upload endpoint, no upload intent, and no presigned PUT: a release is always git archive of a ref in your account's config repository. If you are following an older runbook that mints an upload intent and PUTs a tarball, those calls no longer exist and will 404.

This is a product line, not a gap

A configuration story whose unit is a git repo per account has one ingest path on purpose — every release has a commit behind it, so git_sha provenance is a property of the model rather than a field someone remembered to fill in. The practical consequence is the honest one: you cannot create a release without a git repository the control plane can clone. The repo URL is repo wiring rather than a fleet operation, so whoever operates your control plane sets it for you (see Tag Tracking).

Build From Git

The control plane fetches its cached clone of your config repository, git archives the ref, and creates the release:

curl -X POST https://app.durantic.dev/api/config/releases/build-from-git \
  -H "Authorization: Bearer $DURANTIC_TOKEN" -H 'Content-Type: application/json' \
  -d '{"git_ref": "v1.29.0", "mark_latest": true, "notes": "quarterly baseline"}'

Two properties make this the better path when it is available: git archive of a tree is byte-stable, so the same tag rebuilds to the same sha256; and the build is idempotent on (account, git_sha), so requesting the same commit twice reuses the existing release rather than duplicating it. The resolved git_sha is stored on the release as provenance.

The release's version is the ref itself when the ref is a version and is orderable — v1.29.0, 2026.08.18.104500 — and the short commit sha otherwise. A branch name gives you a sha-named release, and so does a tag that looks like a version but cannot be ordered (2026.8.25), since adopting one would create a release nothing could ever compare.

A failed git build is visible only as a release that never appears

build-from-git returns 202 with no task handle. Bad input is rejected synchronously (unknown git_ref charset, a fixes_version that does not exist in your account, an account with no config repository), but a failure after the enqueue — clone, fetch, or archive errors — is recorded only in the control plane's server logs. There is no build-status resource. Poll GET /api/config/releases/ for your ref or sha with a deadline, and if it never shows up, ask whoever operates your control plane to read the worker log.

The operator-side equivalent

Whoever runs your control plane can build the same release without an API token: manage.py create_config_release --account NAME --git-ref REF (--mark-latest, --mark-canary, --fixes, --notes). Useful when the account's token is the thing that is broken.

Tag Tracking

An account can instead track git tags: the control plane polls the repository, builds a release for every new matching tag, and rolls them out one at a time in order. Push a tag, and it joins the back of the queue; when the account is idle, its rollout starts. There is no campaign object — the releases are the queue, and the account's current rollout is the gate.

GET and PATCH /api/config/account read and set this fleet policy. auto_deploy and track_config_tags are writable — the two knobs an upgrade window needs to turn off. The tag pattern, ordering strategy, repo URL, and pinned-release override are exposed read-only there — they are repo wiring rather than fleet operations, so whoever operates your control plane sets them for you. The endpoint always resolves the token's own account, so there is no account UUID in the path and cross-account access is structurally impossible.

Rolling a Release Out

One POST starts a fleet rollout at your lowest-promotion_order environment:

curl -X POST https://app.durantic.dev/api/config/rollouts/ \
  -H "Authorization: Bearer $DURANTIC_TOKEN" -H 'Content-Type: application/json' \
  -d "{\"config_release_uuid\": \"$RELEASE_UUID\"}"

An account has one running rollout. Starting another supersedes the first wholesale — which is exactly what you want for a mid-flight hotfix, and exactly what you do not want by accident. A rollout's status is one of running, halted, completed, cancelled, or superseded.

Pass "protected": true when you start a rollout that must not be superseded implicitly — a quorum-sensitive cluster upgrade, say. While it is running, starting a rollout for a different release is refused unless that call passes "force": true. Re-POSTing the same release still resets it without force, and an explicit halt or a release yank still work, so the emergency levers stay available.

Promotion between tiers is automatic; there is no manual promote verb and no force-advance. It does, however, require the control plane's background worker and its every-minute advance job to be running — see Requirements Today.

Reading the Status Payload

curl -s -H "Authorization: Bearer $DURANTIC_TOKEN" \
  "https://app.durantic.dev/api/config/rollouts/$ROLLOUT_UUID/status?limit=500"

The response is {"summary": {...}, "machines": [...]} — the same live view the admin shows, recomputed from the same hash the agents use.

summary carries active_environment, active_environment_paused, halted_rings (keyed by ring slug, each mapped to the reason it halted), total, eligible, a blocking breakdown, a stall_reason, holding_slots, settling, and a count per state. Each row in machines carries uuid, hostname, environment, ring, bucket, eligible, state, plus queue_position and holds_slot — the last two are meaningful only in a ring with a concurrency cap (see Kubernetes Cluster Upgrades) and are null/false everywhere else.

There are seven machine states:

state	Meaning
applied	Running this release
pending	Eligible now, has not applied it yet
queued	Eligible and waiting behind a ring's concurrency cap — queue_position says where in line
waiting	Not yet eligible — a later environment, or not yet in its ring's wave cohort
failed	Its latest apply failed, or its ring auto-halted in this environment
errored	Eligible, but its config values will not render (an undefined secret or variable)
paused	A circuit breaker is tripped — its own, its ring's, or its environment's

stall_reason is a single sentence explaining why the cursor is not moving. It names a paused active environment, halted rings, an environment where nothing can converge (nothing assigned, or everything paused), machines that are failing, an environment blocked entirely on errored machines, a wave_ladder that tops out below 100% so some machines are never elected, a settle soak still running, or telemetry being unreadable (in which case the accompanying counts are flagged as an approximation). An empty stall_reason with the cursor sitting still usually means a ramp simply has not opened yet.

Reading Apply Results

status is the gating truth. For why one machine failed, read its raw apply observations:

curl -s -H "Authorization: Bearer $DURANTIC_TOKEN" \
  "https://app.durantic.dev/api/config/rollouts/$ROLLOUT_UUID/apply-results?machine_uuid=$MACHINE_UUID&limit=100"

Rows are newest first and carry machine_uuid, created_at, started_at, finished_at, overall_success, exit_code, applied_generation, and state_results — the engine's per-unit breakdown as {state_id, result, duration_ms, comment, changes_json}.

The per-unit changes key is changes_json

Not changes. It is a compact JSON string of what that state changed. Raw engine stdout is deliberately not exposed on this endpoint.

When a Rollout Fails

The failure path is short and has no undo button in it.

1. A ring auto-halts. When a ring's count of distinct failing machines reaches its failure_count_threshold, that ring is recorded in halted_rings with a reason. Other rings keep rolling — only the breached ring freezes.
2. A halted ring blocks promotion. When the active environment has otherwise converged but a ring is halted, the whole rollout goes halted rather than promoting. This is the staging→prod gate: a release whose tested tier had failures never reaches the next tier.
3. Nothing un-halts itself. There is no resume verb. A halted rollout is no longer running, so the release stops being advertised to agents.

Recovery Is Roll-Forward

There is no rollback, and this is deliberate. Configuration state is not time-reversible: re-applying yesterday's states does not undo a migration that ran, a package that was removed, or a file that was rewritten. A platform that offered you a "roll back the fleet" button would be lying about what it can restore. So recovery is always forward, in two moves:

# 1. Withdraw the bad release — never served again, cannot be re-marked latest
curl -X POST https://app.durantic.dev/api/config/releases/$BAD_UUID/yank \
  -H "Authorization: Bearer $DURANTIC_TOKEN" -H 'Content-Type: application/json' \
  -d '{"reason": "broke the db ring"}'

# 2. Ship a hotfix that names what it fixes
curl -X POST https://app.durantic.dev/api/config/releases/build-from-git \
  -H "Authorization: Bearer $DURANTIC_TOKEN" -H 'Content-Type: application/json' \
  -d '{"git_ref": "v1.29.1", "fixes_version": "v1.29.0", "mark_latest": true}'

fixes_version is what makes it a hotfix, and a hotfix is front-loaded: any machine whose applied release is the one being fixed becomes eligible immediately, ahead of the wave ladder. The fix reaches exactly the broken machines first, while healthy machines still follow the normal ramp. Yanking a release whose rollout is still running halts that rollout as part of the yank, since an unservable release could never converge.

Re-POSTing the same release that halted resets that rollout row back to running in place — the shipped retry semantic, for when the cause was environmental rather than in the release. Rolling forward with a hotfix remains the guidance when the release itself was wrong.

What the Agent Does on the Node

The node has its own, narrower safety net. It exists to keep one machine serving; it is explicitly not a fleet rollback, and it never hides a failure from the rollout:

• Last-known-good revert. If a newly activated release fails to converge, the agent atomically restores the previous good release and re-converges it from the exact config and facts it applied last time.
• The failure is still reported. The failed result is what goes to the control plane, not the recovery result — so it still counts toward the ring's threshold and still halts the ring. Local recovery cannot mask a rollout-gating failure.
• Flap damping. Failures retry with capped exponential backoff (from 5 seconds up to a 5-minute cap), and after three consecutive failures of the same assignment the agent latches it and stops retrying until something actually changes — a new release, a forced reapply, or an unpause. A broken release cannot hot-loop the engine.
• Failures never take the machine off the network. Config management runs as its own service inside the agent; an engine or artifact failure cannot tear down heartbeats, the mesh dataplane, or command handling.

Ongoing Convergence and Drift

Convergence does not stop when a rollout completes. Each agent re-converges its current release on a jittered timer between 15 and 30 minutes, forever. Someone who edits a managed file by hand on a node gets corrected within half an hour without anyone opening a ticket, and a machine that was offline picks up where it left off.

Two things follow that surprise people:

• The drift run is a real apply, not a preview. It runs the same states with the same engine. Your states must therefore be genuinely idempotent: a converge of an already-correct machine should change zero units. There is no observation-only mode: every converge the agent runs is a real apply.
• Drift runs report results like any other apply. So apply-result rows keep arriving every 15–30 minutes per machine, and a drift run that fails counts toward its ring's failure threshold exactly like a rollout apply. A machine that starts flapping weeks after a release landed can halt its ring.

Pausing Config Management

Before you work on a machine by hand, take it out of the loop. There are three breakers, at three scopes, and a machine is effectively paused if its own, its ring's, or its environment's is set:

Scope	Call
One machine	POST /api/provisioning/machines/{machine_uuid}/config-pause
A whole ring	POST /api/config/rings/{ring_uuid}/pause
A whole tier	POST /api/config/environments/{environment_uuid}/pause

Each takes an optional reason, and each has a matching config-unpause / unpause.

Pausing does more than withhold the next release: the control plane sets an explicit paused flag in the machine's config, and the agent suspends all engine invocation — new applies and its own drift re-converges. That distinction is the whole point. If pausing only removed the assignment, the agent would keep re-applying its last release on its own timer and undo your manual work anyway. Paused means salt-call does not run.

Paused machines are reported as paused, and whether they gate promotion depends on whether they got the release first:

• Paused after applying the current release — does not gate. There is nothing left to wait for, so pausing a converged box for maintenance cannot wedge a rollout.
• Paused before applying it — does gate, and the hold names the machine. Pausing is the remedy for a broken node; it is not permission to promote the release past a tier that node never applied. Otherwise the safety gate that exists to stop an untested release reaching the next environment would be cleared by the very action you took because the release broke something.

The blocker shows up as blocking.paused in the rollout status, kept separate from blocking.pending on purpose: pending means a machine is still coming, while paused means it is not coming at all — a paused machine is not being managed, so that blocker clears only by an operator action.

To promote without the machine, take it out of scope rather than out of management: unassign its environment. Pausing says "stop touching this box"; unassigning its environment says "this box is not part of this rollout". Two levers, two meanings — the hold reason says which one you want.

If everything in the active environment is paused, the cursor holds and says so rather than promoting an untested release onwards.

Forcing a Reapply

Convergence is content-driven, so re-submitting an unchanged release is a no-op — the agent applies it once and then only re-runs it on its drift timer. To make a machine reapply the current release right now, the control plane bumps a monotonic generation token that rides in the assignment; a changed token means changed config, so the agent reapplies.

One POST does this at whichever scope you need, and the applied_generation field on the resulting apply result is how you confirm it landed:

curl -X POST -H "Authorization: Bearer $DURANTIC_TOKEN" \
  https://app.durantic.dev/api/config/rings/$RING_UUID/reapply
# {"machines": 12, "rollout_running": true}

The same verb exists at /api/config/environments/{uuid}/reapply and, per machine, at /api/provisioning/machines/{uuid}/config-reapply. rollout_running: false means the request was recorded but nothing converges yet — a machine only reapplies under a running rollout, so the bump takes effect when one starts. On a ring with max_in_flight set this does not jump the queue: the machines become not-done again and wait for a converge slot in the usual order.

Requirements Today

Read this section before you plan a rollout. These are the constraints that surprise people, and none of them is a bug.

1. A release needs a git repository the control plane can clone. build-from-git is the only ingest path — there is no upload endpoint. An account with no config repository configured cannot create a release at all, and the repo URL is set by whoever operates your control plane rather than through the API.
2. The node must already have salt-call. The agent executes the host binary at /usr/bin/salt-call and refuses to converge if it is not there as an executable file. Install Salt in your OS image or via a cloud-init role — the official Ubuntu and Rocky images already carry it. The agent does not ship an engine runtime of its own, so "no Salt on the node" is not a supported configuration.
3. Releases are integrity-checked, not signed. The agent verifies the release's sha256 before unpacking and never executes an archive that fails it. That detects corruption and truncation in transit, but it is not a cryptographic signature — do not describe your pipeline as supply-chain-signed.
4. Rollout advancement needs the control plane's worker. Promotion between tiers, stall detection, and ring halting run on the background worker plus an every-minute advance job. On Kubernetes that means the controlplane chart at 2e7f5b5 or newer (the dramatiq Deployment and the advance_rollouts CronJob). Without both, a rollout you start never promotes. This is the single most common cause of "nothing is happening".
5. One running rollout per account. Any config change during a long rollout can only ship by superseding it. Plan changes around rollout windows, not into them.
6. A machine needs both an environment and a ring. Missing either, it is served no release and simply sits in waiting.
7. saltstack is the engine that works. The API accepts ansible as a reserved value, but the shipped agent's adapter refuses to run any engine other than saltstack. Do not create Ansible releases expecting them to converge.
8. Config management does not run in the installer. It is deliberately absent from the provisioning initrd and starts on the installed system. Provision-time configuration is cloud-init's job — see Cloud-Init and Roles.

Serial, One-Node-at-a-Time Rollouts

Everything above converges a ring's machines in parallel as its ramp opens. Some changes cannot work that way — a Kubernetes version upgrade has to cordon, drain, upgrade, and health-check one node before it touches the next.

That pattern is built on exactly the mechanisms on this page — environments as the tier order, rings as the partition, releases as the content, plus a ring-level concurrency cap and the slot lease that enforces it — and it has its own runbook: Kubernetes Cluster Upgrades.