Monitoring & Metrics

The Durantic agent can expose a Prometheus-compatible /metrics endpoint on each machine. It is disabled by default and, when enabled, is scraped by your own Prometheus, VictoriaMetrics, or any Prometheus-compatible collector — nothing is sent to Durantic. Because the metrics are read from the agent's live in-memory state at scrape time, they keep working even when the machine's link to the Durantic control plane is down.

What you get

Per-mesh-peer link metrics (throughput, last handshake, RTT, quality, BGP session state) plus, most usefully, whether each link is direct or routed via a transit relay — so you can alert the moment a peer falls back to relaying. There are also control-plane connectivity, gossip, VIP health, agent build, and per-interface NIC metrics (throughput, errors, drops, link flaps) for detecting traffic spikes and physical-layer problems. See the full metric reference below.

Enabling the endpoint

Disabled by default — you must turn it on

The /metrics endpoint is off by default and is never enabled automatically. Each machine that should expose metrics has to be enabled explicitly — there is no account-wide "on" switch. Enable it per machine (systemd drop-in or config file), or roll it out to a group by adding the same setting to a Machine Role; either way it only applies to the machines you opt in.

The endpoint is controlled by a single setting, the listen address. If it is set, metrics are on; if it is empty (the default), metrics are off. You can set it three equivalent ways.

On a single machine (systemd drop-in)

sudo mkdir -p /etc/systemd/system/durantic.service.d
sudo tee /etc/systemd/system/durantic.service.d/metrics.conf <<'EOF'
[Service]
Environment=DURANTIC_METRICS_LISTEN=mesh:9342
EOF
sudo systemctl daemon-reload && sudo systemctl restart durantic

To disable again, remove the file, daemon-reload, and restart.

Across a fleet (Machine Role)

Add the same drop-in to a Machine Role so every machine provisioned with that role comes up with metrics enabled:

write_files:
  - path: /etc/systemd/system/durantic.service.d/metrics.conf
    permissions: '0644'
    content: |
      [Service]
      Environment=DURANTIC_METRICS_LISTEN=mesh:9342
runcmd:
  - systemctl daemon-reload
  - systemctl restart durantic

Config file

If you manage /etc/durantic/config.toml directly:

[metrics]
listen = "mesh:9342"

Listen address reference

The listen address is a host:port. The host part decides which network interface the endpoint is reachable on — choose it deliberately, since the endpoint is unauthenticated (see Security).

Value	The agent binds	Use it for
(empty / unset)	nothing — metrics disabled	the default
`mesh:9342`	this machine's own WireGuard mesh IP, once assigned	fleet rollout via one role; reachable only from inside the mesh
`127.0.0.1:9342`	localhost only	a scraper running on the same machine
`10.0.5.3:9342`	that exact interface IP	a specific network layout
`0.0.0.0:9342`	all interfaces	not recommended — also exposes the endpoint on the provisioning network

Notes:

mesh is a keyword, not a hostname. The mesh IP is assigned by the control plane and may not exist the instant the agent starts, so the agent resolves mesh at runtime and retries every 60 seconds until the mesh IP is available. If the machine is never assigned to a mesh, the endpoint simply never comes up (logged as a warning; the agent is otherwise unaffected).
The port is your choice. 9342 is the value we document by convention (it is not used by common exporters), but any free port works.
Changing the listen address takes effect on the next systemctl restart durantic.

One mesh IP per machine

With mesh:9342 on three machines, each binds its own mesh IP on port 9342 (for example 10.0.0.5:9342, 10.0.0.6:9342, 10.0.0.7:9342). Point your scraper at each machine's mesh IP.

Port conflicts

If the chosen port is already in use (for example by another exporter you install through the same role), the agent logs a warning and retries every 60 seconds — it never crashes the agent and never disturbs the other listener. Because both the agent's metrics config and any exporters you install arrive through the same provisioning role, you are always the one choosing the ports; pick a free one, or move the other service.

Scraping

Point your Prometheus at each machine's mesh IP:

scrape_configs:
  - job_name: durantic-agents
    scheme: http
    scrape_interval: 30s
    static_configs:
      - targets:
          - 10.0.0.5:9342
          - 10.0.0.6:9342
          - 10.0.0.7:9342

The same target list works for a VictoriaMetrics vmagent. Your scraper must be able to reach the machines' mesh IPs — for a mesh:9342 endpoint that means the scraper is itself a member of the mesh, or routes into it.

Scrape interval

The scrape cadence is set entirely on your scraper (the agent just answers each request). 30–60s is recommended. Mesh path state (direct/transit, RTT, quality) changes on the agent's ~15s probe cycle, so faster scraping doesn't reveal more; interface byte counters move faster, but 30s still resolves spikes lasting more than a scrape or two. The endpoint bounds concurrent scrapes internally, so an over-aggressive interval can't overload the agent — it's just wasteful. Sub-scrape microbursts are not visible (true of any pull-based exporter).

Example queries

# Any mesh link currently running over a relay instead of direct
durantic_mesh_peer_path_type{type="transit"} == 1

# Peers whose tunnel looks dead (no handshake in 3 minutes)
time() - durantic_mesh_peer_last_handshake_timestamp_seconds > 180

# Per-link send throughput
rate(durantic_mesh_peer_tx_bytes_total[5m])

# Agent running on stale cached config (control plane unreachable)
durantic_controlplane_using_cached_config == 1

# Egress interface utilization % (throughput vs negotiated line rate)
100 * 8 * rate(durantic_egress_tx_bytes_total[5m]) / (1e6 * durantic_egress_speed_mbps)

# Traffic spike on any egress interface
rate(durantic_egress_tx_bytes_total[1m])

# Non-mesh traffic on the egress NIC (total minus the sum of mesh links)
rate(durantic_egress_tx_bytes_total{interface!~"durantic-.*"}[5m])
  - sum(rate(durantic_mesh_peer_tx_bytes_total[5m])) without (peer_mesh_ip, peer_hostname)

# Physical link flapping
increase(durantic_egress_carrier_changes_total[15m]) > 0

Diagnostic combination: a mesh link degrading (durantic_mesh_peer_quality down) with high durantic_egress utilization points to congestion; the same degradation with rising carrier_changes and flat utilization points to a physical NIC/cable/uplink problem instead.

Metric reference

Per-peer series are labelled peer_mesh_ip (the authoritative identity) and peer_hostname (for display).

Metric	Type	Notes
`durantic_mesh_peers`	gauge	Tracked mesh peers (0 when no tunnel is active)
`durantic_mesh_peer_rx_bytes_total`	counter	Bytes received from the peer
`durantic_mesh_peer_tx_bytes_total`	counter	Bytes sent to the peer
`durantic_mesh_peer_last_handshake_timestamp_seconds`	gauge	Last WireGuard handshake time
`durantic_mesh_peer_up`	gauge	1 if the peer has an active path
`durantic_mesh_peer_path_type`	gauge	State set with label `type=direct\\|transit`
`durantic_mesh_peer_transit_via_info`	gauge	Present (1) only while transit; labels `via_mesh_ip`, `via_hostname`
`durantic_mesh_peer_rtt_seconds`	gauge	Smoothed RTT of the active path
`durantic_mesh_peer_quality`	gauge	Effective path quality bucket (0–7)
`durantic_mesh_peer_failovers_total`	counter	Lifetime active-path failovers
`durantic_mesh_peer_bgp_session_state`	gauge	BGP FSM state (6 = Established)
`durantic_mesh_peer_tunnel_type`	gauge	State set with label `type=wireguard\\|geneve`
`durantic_gossip_members`	gauge	Gossip cluster size
`durantic_gossip_*_total`	counter	Gossip message / hint / node-event counters
`durantic_vip_healthy`	gauge	Per-VIP (`vip_id`) health, 1 = passing
`durantic_vip_response_time_seconds`	gauge	Per-VIP last check response time
`durantic_vip_consecutive_failures`	gauge	Per-VIP consecutive check failures
`durantic_controlplane_connected`	gauge	1 if the control plane event stream is up
`durantic_controlplane_stream_reconnections_total`	counter	Lifetime event-stream reconnections
`durantic_controlplane_heartbeat_rtt_seconds`	gauge	Last heartbeat RTT
`durantic_controlplane_using_cached_config`	gauge	1 if running on cached config
`durantic_agent_info`	gauge	Constant 1 with `version`, `commit` labels
`durantic_agent_startup_duration_seconds`	gauge	Process start to ready
`durantic_metrics_collect_errors_total`	counter	Scrape-time collection errors

Standard Go runtime and process collectors (go_*, process_*) are also exported.

Interface metrics

Per-interface counters for the NICs actually carrying mesh traffic — the underlay NIC(s) the mesh egresses through, plus the tunnel interfaces durantic-wg and durantic-gnv. These let you detect traffic spikes, link saturation, and physical-layer trouble, and they surface Geneve dataplane throughput (Geneve has no per-peer byte counters, so durantic_egress_*{interface="durantic-gnv"} is where its traffic shows up). Labelled by interface; this is a small bounded set, not every NIC on the host.

Metric	Type	Notes
`durantic_egress_rx_bytes_total` / `_tx_bytes_total`	counter	Interface throughput (use with `rate()`)
`durantic_egress_rx_packets_total` / `_tx_packets_total`	counter	Packet rate
`durantic_egress_rx_errors_total` / `_tx_errors_total`	counter	NIC frame errors
`durantic_egress_rx_drops_total` / `_tx_drops_total`	counter	NIC dropped packets (queue/buffer pressure)
`durantic_egress_carrier_changes_total`	counter	Link-flap counter (cable/SFP/driver/switch trouble)
`durantic_egress_speed_mbps`	gauge	Negotiated line rate (omitted when unknown, e.g. virtual interfaces)

Cardinality

Series count scales with mesh size: roughly a dozen per-peer series per peer, so a 200-peer mesh produces on the order of a few thousand series per machine — well within a normal Prometheus/VictoriaMetrics budget. Peers are labelled by mesh IP and hostname only (never by endpoint IP or public key), which keeps cardinality bounded as NAT mappings and keys change.

Security

The endpoint is unauthenticated. Bind it to your mesh (mesh:9342) or to localhost (127.0.0.1:9342) so it is reachable only from trusted networks; avoid 0.0.0.0. Left unset (the default), nothing is exposed at all.