Kubernetes Cluster Upgrades

Overview

Durantic upgrades a Kubernetes cluster one node at a time, gated on success: node N+1 is handed the upgrade only after node N has converged, passed its own health check, and settled. Control-plane nodes go first, then workers. Draining and uncordoning happen inside the release, around each node, so workloads move off a node before it is touched and come back when it is healthy again.

Nothing about this is a separate upgrade engine. It is the shipped pull system — a ConfigRelease rolled out across Environments in promotion_order, partitioned into Rings — with one extra brake: a ring can cap how many of its machines converge at the same time. Set that cap to 1 and a ring becomes a queue.

This page is the operator runbook: how to lay the cluster out, what the release must contain, how to run the window, and what to do when a node fails.

The serial controls are off until you set them

max_in_flight, min_settle_seconds, stall_timeout_seconds, and ConfigRelease.timeout_seconds all default to 0, which means a ring converges its whole wave in parallel — the behaviour of every ring you have not deliberately capped. Serialization is something you turn on per ring, for the rings that need it; it is not a fleet-wide mode change.

How Serialization Works

A ring with max_in_flight = 1 has exactly one open slot. When the control plane serves the upgrade assignment to a machine in that ring, it records a slot lease on that machine. The holder keeps the slot — and keeps being served its assignment — until one of four things happens: it reports success, it is paused, the rollout is superseded, or the rollout is reset.

The lease is the load-bearing part. Because the slot is a recorded fact about an assignment already served, nothing else happening in the ring can take it away from a node in the middle of a drain: another machine unpausing, a machine joining the ring, a drift result arriving, a settle window re-arming — the holder is sticky through all of it, and no second node is admitted while the slot is held. Withdrawing an assignment mid-run is precisely what would kill a running kubectl drain, so the control plane never does it to a holder.

On success the lease is released, the machine's settle window opens, and the next machine in queue order is nudged. Queue order is (bucket, uuid), where the bucket comes from a per-release hash — it reshuffles for every release and you do not pick it (see What This Does Not Do).

Serialization is per (ring, environment)

The cap counts open slots within one ring in one environment. One cluster tier = one capped ring is the supported layout. A straggler machine in an already-promoted environment can converge at the same time as the active environment's holder; there is no cross-tier or cross-cluster serialization.

Cluster Layout

Give each cluster two environments with consecutive promotion_order, and one capped ring in each:

Environment	`promotion_order`	Ring	Members
`<cluster>-cp`	e.g. `10`	`<cluster>-cp`	the control-plane nodes
`<cluster>-workers`	`11`	`<cluster>-workers`	the worker nodes

The rollout cursor walks environments low → high, so "control plane first, then workers" needs no new mechanism: the workers environment is not served the release until every machine in the control-plane environment has converged and settled. Within each environment, the ring cap serializes the nodes.

Create them with POST /api/config/environments/ (name, slug, promotion_order, config_values); promotion_order is also PATCHable afterwards. Keep the pair adjacent and do not put anything else between them, or an unrelated tier promotes in the middle of your cluster upgrade.

Ring Settings for a Serial Upgrade

Apply these to both rings. Every value here has a reason; changing one changes what the window guarantees.

Setting	Value	Why
`max_in_flight`	`1`	One open slot per (ring, environment) — exactly one node inside the cordon → drain → upgrade → uncordon arc at a time. `0` (the default) means unlimited.
`wave_ladder`	`[{"after_seconds": 0, "pct": 100}]`	The ladder is a time-based election; the cap is what sequences a serial ring. Make every member electable immediately and let the cap do the work. A partial ladder adds a second brake you then have to reason about — and a machine the ladder has not elected never enters the queue at all.
`failure_count_threshold`	`1` (the default is `10`)	Stop the line on the first failed node. The blast radius of a cluster upgrade should be one node, and a health-gate failure is an apply failure — it counts here.
`min_settle_seconds`	`120`–`300`, cluster's taste	The slot stays occupied this long after the node's first success on this release, so N+1 starts only once N has been healthy and quiet. Environment promotion waits it out too. Anchored on the first success per (machine, release) — the anti-drift successes that arrive every 15–30 minutes afterwards never re-arm it.
`stall_timeout_seconds`	≥ release `timeout_seconds` + reboot allowance	If the holder disappears (powered off mid-upgrade), the ring is halted with a reason naming it, instead of waiting forever. Set it above one full apply plus reboot cycle, or a legitimately slow node halts its own ring.
`ConfigRelease.timeout_seconds`	longer than the slowest node's whole arc (drain + upgrade + health gate)	The agent enforces this per apply. Set `stall_timeout_seconds` above it, never below — otherwise the ring halts while the node is still legitimately working.

Set them on each ring in one PATCH. The ring endpoint takes name, failure_count_threshold, wave_ladder, the three serial fields, and config_values (slug is immutable):

curl -X PATCH https://app.durantic.dev/api/config/rings/$RING_UUID \
  -H "Authorization: Bearer $DURANTIC_TOKEN" -H 'Content-Type: application/json' \
  -d '{"failure_count_threshold": 1,
       "wave_ladder": [{"after_seconds": 0, "pct": 100}],
       "max_in_flight": 1,
       "min_settle_seconds": 300,
       "stall_timeout_seconds": 5400}'

stall_timeout_seconds only means anything on a capped ring — stall detection halts a ring over its slot holder, and an uncapped ring never grants a slot — so setting it while the ring would be left uncapped is a 400 naming max_in_flight. Send the cap and the timeout together, as above, and the check passes: it judges the state your request would leave behind, not each field alone.

Going the other way is always allowed. Setting max_in_flight back to 0 uncaps the ring and releases its slot leases even though that leaves a now-inert stall timeout behind, because uncapping is your release valve and must never be refused.

To confirm the cap is live, read the rollout status: a capped ring shows queued machines, and holds_slot on at most max_in_flight rows. An uncapped ring converges the whole wave at once, which during a cluster upgrade is the outcome this page exists to prevent.

Preconditions: The Upgrade Window Checklist

Work through all of these before creating the release.

1. The control plane can actually advance rollouts. The controlplane chart must be at 2e7f5b5 or newer, which ships the background dramatiq worker Deployment (dramatiq.enabled) and the every-minute advance_rollouts CronJob (advanceRollouts.enabled, advanceRollouts.schedule: "* * * * *"). Without both, a rollout you start never promotes between environments, stalls are never detected, and a freed slot is only picked up on the agent's own poll. This is the single most common reason "nothing is happening":

kubectl -n <ns> get deploy  -l app.kubernetes.io/component=dramatiq
kubectl -n <ns> get cronjob -l app.kubernetes.io/component=advance-rollouts

2. Turn off auto_deploy and tag tracking for the account. Not because they can interrupt the window — they cannot. Both unattended paths yield to a running rollout: the auto-deploy signal bails when the account already has one, and the tag sequencer's gate waits on it. Turn them off so that nothing starts the moment your upgrade completes, before you have verified the cluster: the slot frees, and a queued release would take it unattended. The switches are auto_deploy (default off) and the tag sequencer's track_config_tags / config_tag_pattern / config_tag_strategy.

The path that *can* interrupt a window is an **explicit** start — another operator's
admin "Apply now", or an automated caller (CI, terraform, a test session) posting to
`/api/config/rollouts/`. That supersede does cancel an in-flight arc, leaving a node
cordoned with workloads off it. Step 4 is the guard against it.

# check
curl -sH "Authorization: Bearer $DUR_TOKEN" \
  https://$CP/api/config/account
# clear for the window (restore afterwards — step 5)
curl -sX PATCH -H "Authorization: Bearer $DUR_TOKEN" -H 'Content-Type: application/json' \
  -d '{"auto_deploy": false, "track_config_tags": false}' \
  https://$CP/api/config/account

Why these live on the config surface

GET/PATCH /api/config/account exposes the fleet knobs (auto_deploy, track_config_tags, plus read-only tag/pin/repo context) to any account API token. Account identity configuration stays on the owner-gated POST /api/account/config — an upgrade window is a fleet operation, and operator tokens are never account owners. Secrets are never exposed there.

3. Accept the fleet freeze. An account has one running rollout. For the hours a cluster upgrade takes, that slot belongs to the upgrade: any other config change can only ship by superseding it. This is the shipped invariant, not something new — plan config changes around upgrade windows rather than into them.

4. Protect the rollout. Start it with "protected": true and the control plane refuses to supersede it implicitly — a caller wanting the slot must pass "force": true and say so on purpose. This is the guard against the explicit-start accident described in step 2, and it is the one that matters, because the unattended paths already yield.

Your emergency levers are unaffected: `POST /api/config/rollouts/{uuid}/halt` and
yanking the release both still work on a protected rollout, and re-posting the *same*
release still resets it (the retry semantic) without `force`.

5. The upgrade release is a fleet release. The running rollout's release is served to every env/ring-assigned machine in the account, including machines in already-promoted environments. So the release must carry the whole fleet's normal config, with the k8s-upgrade states targeted in top.sls on a role or cluster grain (from the machine's facts) — otherwise non-cluster machines execute upgrade and health-gate states that mean nothing to them.

6. Drain credentials exist — see Drain Credentials.

7. The release satisfies the state contract — see The Release Contract. This one is not optional and the control plane does not check it for you.

The Release Contract (Hard Requirement)

Every release served to a capped ring must satisfy the upgrade-release state contract. This is release content discipline — no platform mechanism enforces it — and two shipped behaviours punish a release that violates it:

Anti-drift re-converges the same release every 15–30 minutes, forever. An unguarded cordon/drain state therefore re-drains every already-upgraded node, on every drift cycle, for as long as the release is assigned.
Last-known-good re-converges an older release. Once an upgrade succeeds, that release becomes the node's last-known-good; when a later release fails, the agent full-converges the older upgrade release. That is a no-op on an already-upgraded node only because of the guards below — the safety is a property of the contract, not of the mechanism. A guard-violating release defeats it and can attempt a downgrade or uncordon a half-upgraded node.

The rules:

Monotone guards on every disruptive state. Cordon, drain, and upgrade states run onlyif the node's running version is less than the target version. Compare monotonically, never for equality — "not equal" would make an older release act on an upgraded node.
uncordon fires only via onchanges of the upgrade state. It runs because this converge upgraded the node — never as unconditional cleanup.
A converge of an at-target node changes zero states. That is the property the anti-drift loop exercises forever; treat "the drift run reported zero changes" as the acceptance test for the release.
Order the arc with requisites and failhard: True: cordon_drain → upgrade → health_gate → uncordon.

A tested reference implementation of this arc exists for both k3s and rke2, together with the RBAC manifest it needs. Ask your Durantic contact for it — it is not in a repository you can browse — and vendor it into your config repository rather than writing the arc from scratch.

If you would rather implement it directly, the four rules above are the whole contract: there is nothing in the reference tree that they do not specify.

Health gate failures are apply failures

The release's final states poll node and cluster health and fail the run if the node did not come back. There is no separate health concept: a health-gate failure counts toward failure_count_threshold, halts the ring, and freezes the queue on that node — exactly like a failed drain or a failed package install.

Drain Credentials

Draining is kubectl work done by the release, on the node, so each tier needs credentials that survive its own node going away.

Control-plane nodes use the distro admin kubeconfig already on the node (/etc/rancher/k3s/k3s.yaml for k3s, the RKE2 equivalent), but with the server address rewritten to the cluster VIP — https://{{ vars.K8S_VIP }}:6443, never 127.0.0.1 or localhost. The node you are draining and upgrading is the one whose local API server is about to restart; talk to the cluster, not to yourself.

Worker nodes have no admin kubeconfig. They use a durantic-node-maintainer ServiceAccount, created by a control-plane-tier state — the environment ordering is what guarantees it exists before any worker tries to drain. Its RBAC needs:

pods list / get cluster-wide (filtered by spec.nodeName at call time),
create on pods/eviction in every namespace — evictions are namespaced objects created per pod, so a namespace-scoped rule silently fails on the first workload outside it,
nodes get / patch (cordon and uncordon are node patches).

Store the SA token as a Durantic secret and reference it from the ring or environment config values as {{ secrets.K8S_DRAIN_TOKEN }}. It is rendered server-side into the machine's config over mTLS, templated into a transient 0600 kubeconfig for the run, and removed afterwards. The same token backs the workers' health-gate reads.

Drain the VIP holder last

On a control-plane tier the API-server VIP may be living on the very node being drained, so the health gate can be talking through an address that is failing over underneath it. The reference tree's health-gate states tolerate one leadership flap with an internal retry and deadline. Ordering the VIP holder last in the window (move the VIP, or simply check which node holds it and pause/unpause to influence membership order) removes the flap entirely.

Running the Upgrade

1. Create the release. Build it from your config repository — that is the only way a release comes into existence:

curl -X POST https://app.durantic.dev/api/config/releases/build-from-git \
  -H "Authorization: Bearer $DURANTIC_TOKEN" -H 'Content-Type: application/json' \
  -d '{"git_ref": "v1.31.4-upgrade", "engine": "saltstack",
       "timeout_seconds": 0,
       "notes": "k8s 1.30 -> 1.31, cordon/drain arc"}'

timeout_seconds left at 0 means the release carries no per-apply deadline of its own — and on a cordon/drain arc you usually want one. It is settable at creation only: a release's assignment is immutable once published, so changing the deadline means publishing a new release.

2. Start the rollout. One rollout per account. It begins at the lowest promotion_order environment — your <cluster>-cp tier, assuming nothing lower is in the way:

curl -X POST https://app.durantic.dev/api/config/rollouts/ \
  -H "Authorization: Bearer $DURANTIC_TOKEN" -H 'Content-Type: application/json' \
  -d "{\"config_release_uuid\": \"$RELEASE_UUID\"}"

3. Watch the status (next section).

4. Let the tier promote itself. When every machine in <cluster>-cp is converged and its settle window has elapsed, the cursor advances to <cluster>-workers on the next advance tick (≈60 s). There is no manual promote step and no force-advance.

5. Restore auto_deploy (PATCH /api/config/account) and unpause anything you paused when the rollout reaches completed.

Expect roughly per-node apply time + settle + ~10 s of handoff per node, and up to ~60 s of extra handoff latency when a settle window closes between nudges.

Reading the Status Payload

curl -s -H "Authorization: Bearer $DURANTIC_TOKEN" \
  "https://app.durantic.dev/api/config/rollouts/$ROLLOUT_UUID/status?limit=500"

Four things matter during a serial upgrade:

The per-machine rows — hostname, uuid, environment, ring, bucket, eligible, and state. state is one of paused, applied, failed, errored, pending, queued, waiting. In a capped ring, the node currently inside the arc is pending with holds_slot: true; the ones behind it are queued, each with a 1-based queue_position telling you who is next. waiting keeps its original meaning — a later environment, or not yet in the wave cohort — so a queued node is never confused with one whose tier has not opened.
halted_rings — a map of ring → reason. Non-empty means the line has stopped; the reason names the blocking machine.
stall_reason — a single sentence explaining why the cursor is not moving: a paused environment, halted rings, a tier with nothing gateable, machines failing or erroring, a ladder that tops out below 100%, a settle soak in progress, or telemetry being unreadable. Note there is deliberately no ring-wide "no progress for N seconds" sentence: a holder that stops making progress surfaces as a halted ring whose reason names the machine and how long it has held its slot, not as a stall.
holding_slots and settling — the two summary counts that make the arithmetic add up. A slot is consumed both by holders and by machines soaking out min_settle_seconds, and success clears the lease — so mid-soak holding_slots is 0 while the ring is still full. If a node is queued and nothing seems to be happening, a non-zero settling is usually why.

For the failing state itself, read the node's apply results:

curl -s -H "Authorization: Bearer $DURANTIC_TOKEN" \
  "https://app.durantic.dev/api/config/rollouts/$ROLLOUT_UUID/apply-results?machine_uuid=$MACHINE_UUID&limit=100"

Other verbs you will want in the window (note that a rollout is halted, while environments, rings, and machines are paused — they are different levers):

Action	Call
Stop the whole rollout	`POST /api/config/rollouts/{rollout_uuid}/halt`
Hold one tier without losing the queue	`POST /api/config/environments/{environment_uuid}/pause`
Hold one ring	`POST /api/config/rings/{ring_uuid}/pause`
Skip one node (the manual skip)	`POST /api/provisioning/machines/{machine_uuid}/config-pause`
Bring a skipped node back	`POST /api/provisioning/machines/{machine_uuid}/config-unpause`
Withdraw a bad release	`POST /api/config/releases/{release_uuid}/yank`

Failure Modes and What to Do

Every row here has an action. "Nothing" is never one of them — where the system recovers on its own, the action is what you verify.

Scenario	What the system does	Your action
Node at the queue head is offline (holds the slot)	Slot stays held, the queue does not move — there is no automatic skip. The stall timer runs from the grant; at `stall_timeout_seconds` the ring is halted with a reason naming the holder and how long it has been offline.	Follow Stall Recovery. Decide fix or skip; pausing the machine is the skip, and it is always yours to make.
Node reports an apply, health-gate, or drain failure	It keeps the slot — the line stays frozen on an unproven node. The failure counts toward the threshold (at `1`, the ring halts). The node stays cordoned, because uncordon only fires from a successful upgrade in the same run. The agent retries, then latches.	Read the failing state in the machine's apply results. Fix forward: yank the release and ship a hotfix whose states repair and uncordon, or pause the node and reset (below). Do not expect a retry to clear it.
Drain times out — a PodDisruptionBudget blocks eviction	The drain state fails, so `failhard` stops the arc: the upgrade never ran, the node is left cordoned, the holder keeps its slot, the ring halts at threshold `1`.	Fix the workload, not the platform: relax or fix the PDB, scale the deployment so a replica can move, or raise the release's drain timeout. Then reset the rollout. The control plane has no PDB awareness to override.
Health gate fails after the upgrade ran	Apply failure despite a successful upgrade: cordoned, slot held, ring halted. The agent's last-known-good re-converge of the older release is a no-op under the state contract and reports nothing to the control plane, so the failure row and the halt stand.	Triage the node against the cluster (is it `Ready`? is the API server answering through the VIP?). Roll forward with a hotfix that repairs and uncordons.
A converged node flaps (a later drift run fails)	It is not-done again and re-enters the queue at its bucket if the ladder still passes it. It never displaces the holder. Its failure counts toward the threshold, so persistent flapping halts the ring.	Treat it as a real failure on that node — read its apply results. If it is flapping for a reason unrelated to the upgrade, pause it to keep it out of the queue, and deal with it after the window.
Queue churn while a node holds the slot (another machine unpaused, a machine PATCHed into the ring, a drift result arriving)	The holder is never displaced and no second node is admitted. Churners simply queue behind it.	Do not intervene with the holder. Confirm in the status payload that the holder still shows `holds_slot` and that the queue just got longer.
Rollout superseded mid-queue (an explicit start — another operator's "Apply now", or a CI/terraform caller; unattended auto-deploy and tag rollouts yield to a running rollout and cannot do this)	Every lease in the account is cleared, in-flight arcs are cancelled, the new release reshuffles the buckets, and the queue restarts. Nodes interrupted mid-arc stay cordoned.	Stop the source (find the caller — it was an explicit start, not auto-deploy; protect the rollout per precondition 4), then re-POST the upgrade release to restart the window. Check every node for a leftover cordon with `kubectl get nodes` and `kubectl uncordon` what the next converge will not fix.
You pause a machine that holds the slot	The lease is cleared and the slot frees immediately, so the queue moves on — but this is a skip of the converge slot, not of the promotion gate. A machine paused before it applied the release still holds the environment cursor, and the hold names it. Any run in flight is cancelled, so the node can be left cordoned with workloads off it.	Only pause deliberately. Afterwards, `kubectl uncordon <node>` (or let a later successful converge's `onchanges` chain do it), and unpause the machine once the window is over so it converges alone under the ring cap.
You pause the ring or the environment mid-queue	Nobody in it is served. Leases persist — a ring pause is a hold, not a skip — and in-flight runs are cancelled. Unpausing resumes with the queue intact.	Use ring/environment pause to stop the line without losing your place. Before unpausing, check the interrupted node's cordon state.
Settle window is active	The predecessor shows `applied` while its successor sits `queued`, `settling` is non-zero, and the environment will not promote until `first success + min_settle_seconds`.	Wait it out — the settle stall reason clears itself. If you are waiting on it routinely, lower `min_settle_seconds`; do not work around it.
Telemetry outage (the metrics store is unreachable)	Fail-open for holders, fail-closed for new grants: the current holder keeps being served (its gate is the lease and the applied-release marker, both in the main database), and no new slots are granted — the queue pauses rather than guessing at who is done. Surfaced as a telemetry stall reason, which also flags the accompanying counts as a main-DB approximation.	Fix the telemetry store; the queue resumes by itself. Do not reset the rollout to "unstick" it — the node mid-drain is unaffected and a reset would cancel it.
A machine's config fails to render at the queue head	No assignment is serialized, so no lease is granted — but the machine stays first in the queue and blocks it. Its row shows `errored`.	Fix the template or the missing variable/secret it names. The machine picks up the corrected config on its next fetch and the queue moves.
Capped ring is empty, or every member is paused	Nothing is gateable, so the cursor holds where it is and the environment never promotes.	Check ring membership and per-machine pause state. Either put machines in the ring, unpause them, or move the cursor along by removing the empty environment from the promotion path.
A node is dead and you want the window to finish without it	Pausing frees its slot but still holds the cursor, because it never applied the release. `blocking.paused` counts it and the hold reason names it.	Pause it (so nothing touches it and the queue moves), then unassign its environment to take it out of the rollout's scope. Put it back in the environment once it is repaired.
A node is reimaged during or after the window	It comes back at the k8s version baked into its OS image and self-heals on its next converge — see Reimaged Nodes.	Confirm a rollout is `running`; if the upgrade rollout has already reached `completed`, re-POST the release after the rebuild.

Stall Recovery

A halted ring does not un-halt itself and there is no resume verb. Recovery is roll-forward:

Read the reason. halted_rings and stall_reason in the rollout status name the blocking machine — and, for an offline holder, how long it has been out.
Choose: fix it, or skip it. There is no automatic skip, by design (a cluster quorum is not something to skip past on a timer). Pausing the machine is the manual skip, and it is the only one.
If you fix it: bring the node back or repair whatever failed. A returning holder still owns its lease and simply resumes being served — it finishes its arc.
If you skip it: pause the machine. That clears its lease and frees the slot, and cancels any run in flight, so check the node for a leftover cordon (kubectl uncordon <node>).
Re-POST the same release — POST /api/config/rollouts/ with the same config_release_uuid — to reset the rollout in place. This clears halted_rings and every lease in the account, and the queue fast-forwards through machines that are already done — expect one advance tick per environment, up to ~60 s each, before the remaining node is granted a slot.
Afterwards, unpause anything you paused. A skipped node converges on its own under the ring cap, one node at a time as always.

Reimaged Nodes

Kubernetes binaries are installed by the OS image's bootstrap at provision time, so after a config-managed upgrade the version on disk is owned by both layers. A reimaged node therefore rejoins the cluster at the image's baked version.

It self-heals: the agent's config state is wiped by the rebuild, the served assignment no longer matches, the config manager re-applies it, and the monotone guards carry the node up to the cluster's target version on that first converge. No rollout choreography is involved.

Two honest limits:

It needs the release to be served — that means a running rollout. If the upgrade rollout has already reached completed, re-POST the release after rebuilding the node. The queue fast-forwards through the machines already done and the rebuilt node converges alone under its ring cap.
There is a real skew window. Between rejoining and finishing that first converge, the node runs the image's older version in the cluster — the config sync loop (up to 60 s) plus the upgrade's own runtime. Keep OS images reasonably current with the cluster version to keep the window short, and do not reimage a node while you are counting on cluster-wide version uniformity.

What This Does Not Do

Assumptions worth discarding before the window:

It never skips a stuck node. Not on a timeout, not on a threshold, not ever. The stall timer halts the ring and pages you; pausing the machine is the only skip.
There is no imperative drain, cordon, or health command. Draining is content inside the release, executed by the node on itself. There is no API call that drains a node for you.
The control plane knows nothing about PDBs, topology, or spread. A PDB that blocks eviction surfaces as a failed drain state, not as a scheduling decision the platform routes around.
You cannot pin the node order. Order within a ring is (bucket, uuid) from a per-release hash and reshuffles every release. You control membership (rings, pause) and tier order (environments), not the sequence inside a ring.
There is no rollback. Recovery is roll-forward: yank the release, ship a hotfix that repairs and uncordons. Front-loaded hotfix targets still go through the queue, still one node at a time.
There is no cross-tier or cross-cluster serialization. The scope is one (ring, environment) pair; see the note in How Serialization Works.
There is no live "applying right now" feed. What you get is the per-machine row (pending, holds_slot) plus the machine's apply results. That is the whole observability surface.