agentd — Maturity & Production Readiness

Purpose: honest, dated snapshot of what's production-ready, what's pre-production-ready-with-caveats, and what's still aspirational. Read before deploying.

Date: as of v1.3.0 (2026-06-11); kept current with the release tags. Reassess quarterly or after any RFC-scale change.

Scope: crates/agentd/ only.

TL;DR. The runtime is production-ready for the deployment shape it targets (single-tenant micro-agent, one workflow per process, standalone or small horizontal fan-out on Linux). Every posture concern that's a runtime responsibility — bounded execution, policy, TLS / mTLS / OIDC / HMAC, signed workflows, audit, hot reload, Prometheus, OTLP — is Green with tests. The remaining Red cells are shape choices, not gaps: durability (use an upstream queue), fleet-wide rate limits (use your LB), multi-workflow inside one process (run multiple processes), browser surfaces (front it with a real gateway). Secret-fetching is now in-runtime via [[secrets]] (env / file / command / OAuth2); orchestrator-mounted env/files keep working unchanged.

When this runtime is the right tool. You want a small, fast, single-binary runtime that executes a declarative TOML workflow, refuses to invent capabilities at run time, and fits in a container or a systemd unit. You're happy for durability, secret- fetching, and fleet concerns to live upstream. You expect to rotate certs / tokens / policy in-place without dropping traffic.

When to reach for something else. You need multi-tenant shells, durable queueing built into the runtime, browser-facing HTTP (cookies / CORS / WebSocket), HTTP/2 or HTTP/3, or Windows- native production deployments (Windows compiles but the hot-reload surface is trimmed — see §1.5).

1. Readiness by concern

Legend

• Green — covered end-to-end by tests, safe to rely on.
• Yellow — works and tested, but has a known sharp edge; call it out in your runbook.
• Red — not wired, not advertised as working, or a known gap.

1.1 Correctness / DAG runtime

1.2 HTTP surface (serve mode)

1.3 Security posture

1.4 Observability

1.5 Operations

1.6 Integrations

2. Named gaps with effort sizing

Everything below is deliberately deferred. No work is in flight on these right now; each entry is a handle to grab if a deployment needs it.

2.0 Supply-chain: workflow signature verification

Closed. (RFC 0002 v1). Ed25519 detached signatures over the workflow TOML bytes; fail-closed when [signing].required = true or --signing-required / AGENTD_SIGNING_REQUIRED=1 is set. signing Cargo feature; off by default. External path looks for <config>.toml.sig alongside the TOML; embedded mode bakes the signature via AGENTD_EMBED_CONFIG_SIG in build.rs. v2 follow-up: ECDSA-P256 + Sigstore keyless via cosign.

2.1 diff_compute node

Closed. Structural JSON diff between two context values; emits {added, removed, changed, unchanged} with dot/bracket-notation paths. 8 unit tests cover the edge cases (nested objects, arrays by index, type mismatches, root-scalar change, identical-values unchanged). See docs/agent/capabilities.md §diff_compute.

2.2 Dedicated audit-event file sink with redaction

Closed. [logging.audit] block renders a dedicated tracing layer keyed on target agentd::audit, writing JSONL with field-level redaction (built-in list + operator extension via redact_fields). Default-redacted names: token, secret, password, authorization, api_key, bearer, jwt, cookie, session, reason. Events continue flowing to the main stream in parallel so ops dashboards aren't affected.

2.3 Prometheus /metrics endpoint

Closed 2026-04-23. MetricsSnapshot::to_prometheus emits 0.0.4 text-exposition content; GET /metrics is always live alongside /healthz. Every counter is labelled with the workflow name plus an agentd_build_info gauge carrying the crate version. Tests: metrics::prometheus_text_is_well_formed, triggers::http::metrics_endpoint_returns_prometheus_text.

2.4 TLS cert hot reload

Closed. SIGHUP re-reads the workflow TOML, rebuilds rustls::ServerConfig, and swaps via arc_swap::ArcSwap without dropping in-flight TLS sessions. Also rotates mTLS client CAs on the same signal. Audit events: reload.tls, reload.started, reload.succeeded, reload.failed. Bad reloads keep the old config live — fail-forward-then-recover rather than fail-closed.

2.5 Fleet-wide rate limiting

Per-process today. A lightweight Redis-backed token-bucket adapter is the shortest path, but most deployments should lean on an upstream LB instead.

Effort: ~1–2 days including tests for the distributed case.

2.6 HTTP keep-alive / connection reuse

Closed. Up to 100 requests per connection; the socket's existing read_timeout (30s) doubles as the idle-timeout between requests. Clients opt out per-request via Connection: close; the server reflects the decision via Connection: keep-alive|close response header. Engine-level errors always close the connection (fail-closed on server misbehaviour). Tests: keepalive_serves_two_requests_on_one_connection, keepalive_closes_when_client_requests_close.

2.7 x509 principal extraction (mTLS CN / SAN)

Closed. x509-parser (pure-Rust, added under the server-tls feature) extracts Common Name + DNS SANs from the peer cert and attaches them to trigger.principal:

{
  "kind": "mtls",
  "name": "sha256:<hex>",   // fingerprint — authoritative identity
  "cn": "svc-a",
  "sans": ["svc-a.internal", "svc-a.prod.internal"]
}

Fingerprint remains the pinned identity for auth decisions; CN / SAN are advisory — they drive workflow routing AFTER policy has already authorised the request. SANs are lowercased for case-insensitive matching. Malformed DER returns empty CN / SAN rather than panicking — the fingerprint stays load-bearing even when the parser can't help.

Tests: parse_subject_extracts_cn_and_sans, parse_subject_handles_garbage.

2.8 Workflow hot reload — full scope (SIGHUP v2)

Closed. Key insight: we never had to rebuild the engine. Every lifecycle-sensitive component got its own ArcSwap-wrapped adapter (ReloadablePolicy, ReloadableIntelClient, ReloadableMcpClient, ReloadableMcpAllowlist, HttpReloadable). Tool handlers keep the exact same Arc<dyn …> they captured at registration; the wrapper's Policy / McpClient / IntelligenceClient impl defers to whatever the ArcSwap currently points at. Swapping is lock-free, in-flight calls complete against their pre-swap snapshot, next call sees the new state.

Reload sequence on SIGHUP (each step fail-forward — a single failure keeps the old value live, emits reload.failed stage=<name>, and the process stays healthy):

1. TLS — rebuild rustls::ServerConfig (v1 behaviour).
2. Auth — re-parse JWKS + bearer/HMAC bindings (v1).
3. Policy — rebuild ManifestPolicy (re-compiles Rego, re-reads inline data). Per-thread Rego engines self-invalidate via the fresh RegoSpec.id.
4. MCP allowlist — swap from the new [policy.mcp] block.
5. MCP stdio child — spawn new process, swap, old child drops when the last snapshot goes away (kills + waits).
6. Intelligence client — rebuild from CLI args; bearer file / AGENTD_INTEL_HTTP_BEARER env var re-read so rotation picks up.
7. Routes + rate-limit buckets — rebuild HttpReloadable, swap. Token counters reset to full capacity (a policy-rotation shouldn't let a flooding client keep their allowance).

Audit events: reload.started, reload.tls, reload.auth, reload.policy, reload.mcp_allowlist, reload.mcp_respawn, reload.mcp_respawn_failed, reload.intel, reload.routes, reload.succeeded, reload.failed.

Tests: route add / route remove / rate-limit bucket rebuild / policy swap round-trip / intel client swap / MCP client swap / MCP allowlist swap — all exercise the Arc<dyn …> trait-object boundary (the thing handlers actually hold) to prove the swap takes effect without re-registering.

Scope boundary (still restart-required): bind address, --mcp-stdio command vector, --intel-unix / --intel-http endpoint. These are CLI-arg-shaped at process start and the reload preserves the same arg set — matches the blast radius operators expect from SIGHUP.

2.9 Log rotation for file: target

Closed. [logging].rotation opts a file: target into time-based rollover via tracing-appender:

[logging]
target = "file:/var/log/agent.log"
rotation = "daily"    # "daily" | "hourly" | "minutely" | "never"

Yields agent.log.2026-04-23, agent.log.2026-04-24, ... — the operator's filename is split into dir=/var/log, filename_prefix=agent.log so the rotation suffix lands at the end. "never" (the default) keeps the hand-rolled synchronous FileWriter, so external logrotate-style setups still work unchanged. Size-based rotation is out of scope — tracing-appender is time-based only, and enterprise ops runbooks almost always ask for daily anyway.

Tests: rotating_writer_writes_and_rotates, rotation_never_is_not_accepted_by_rotating_writer, logrotation_parses_from_toml.

2.10 Direct in-process OTLP exporter

Closed. New otel Cargo feature installs a tracing-opentelemetry layer alongside the main fmt + audit layers. Every tracing::Span is exported over OTLP gRPC (tonic transport) to an operator-configured collector endpoint. Sampling is ratio-based at the SDK layer — [otel].sample_ratio = 0.1 caps the fleet-wide export rate without losing the 10% trail.

[otel]
endpoint = "http://otel-collector:4317"
service_name = "agent"
resource_attrs = { env = "prod", region = "eu-west-1" }
sample_ratio = 1.0

Implementation: dedicated agent-otel tokio runtime (1 worker), OnceLock-pinned TracerProvider, graceful flush on shutdown via force_flush + shutdown_tracer_provider. The JSON logs path through the filelog receiver still works for operators who don't want the ~50-crate tokio/tonic dep expansion — it's just the otel feature they don't compile in.

Outbound traceparent propagation (closed 2026-04-23). When the HTTP trigger parses an inbound traceparent, the trace id + flags are threaded onto ExecutionContext.trace_context. The http_request tool reads ctx.outbound_traceparent() and emits a traceparent: <version>-<trace_id>-<fresh_span_id>-<flags> header — the parent id is a per-run 16-hex span id, W3C-correct, so downstream services see the agent as their direct parent rather than whoever called the agent. Cron / fs-watch / manual runs emit nothing; the runtime is a propagator, not an originator.

2.11 Retry jitter

Closed. [[nodes]].retry.jitter multiplies the linear backoff by a random factor in [1 - j, 1 + j], clamped to [0.0, 0.5]:

retry = { max_attempts = 3, backoff_ms = 500, jitter = 0.2 }

Default 0.0 (fully deterministic) so existing workflow tests stay reproducible; operators opt in per-node when the upstream is shared across a fleet. Implementation keeps the backoff function pure (backoff_for(attempt, rng_bits)) — the engine supplies entropy, so unit tests pass fixed bits and assert exact backoffs without a Rng trait.

3. Target deployment bars

"OK for dev"

Any shape. Defaults and default feature set are fine.

"OK for pre-production / internal staging"

• Build Shape A or B (default features + maybe server-tls).
• [policy] declared (no allow-all).
• Bearer or HMAC auth on every route you don't want public.
• --log-format json --log-level info.
• Healthcheck wired on /healthz.
• Stateless — no persistence assumed.

"OK for production (with caveats)"

Everything above, plus:

• TLS termination: either in-process (server-tls feature, with SIGHUP hot reload of the cert / key — see §1.5) or upstream at a real LB.
• mTLS for inter-service traffic when that's your posture.
• Audit events piped to a retention-compliant sink (either a dedicated log collector with target filtering, or wait on gap §2.2).
• Upstream fleet rate-limit when request volumes matter.
• Durable queue in front if you need at-least-once across a fleet. Single-node crash recovery is opt-in (§1.5 — --checkpoint-each-node); a queue boundary with idempotency is still upstream of the runtime.
• drain_timeout_secs + orchestrator grace period set consistently.

"NOT for production today"

• Browser-facing surfaces (no CORS, no cookies, no CSRF).
• Workloads that need sub-millisecond tail latencies (no keep-alive, one thread per connection).
• Workloads needing durability beyond single-node checkpoint/resume — multi-day, queue-backed, or exactly-once execution (durability is opt-in and single-node; see §1.5).
• Multi-tenant shells where different workflows need different auth backends inside one process (bindings are shared across routes).

4. Test coverage snapshot

• Main crate: ~26.4k lines of Rust source.
• Integration tests: ~2.1k lines across 4 test binaries.
• Total #[test] blocks: 471.
• Stability: the test suite has been run three times in a row without flake under default features and with every feature combo in §2 of configuration.md.
• Fixture suites: intel_classify, linear, switch_branch under tests/fixtures/ — each is a complete TOML + expected-outcome pair exercised by testing::fixture::Fixture::run.

What the tests don't cover (deliberate):

• End-to-end TLS + mTLS handshake failures on real sockets — we test the config loader and accept path, not every libssl-side edge case.
• Load / stress testing — no k6 / vegeta harness checked in. Run your own for your expected RPS.
• Chaos (kernel-level signals other than SIGTERM/SIGINT, out-of-descriptors, disk-full on log file) — handled at the "does it crash vs. does it degrade" level, not under a chaos framework.

5. What "not in scope" means

The runtime intentionally does not:

• Persist state by default. A crash loses in-flight work unless --checkpoint-each-node is enabled — opt-in checkpoint/resume gives single-node recovery (§1.5), but it is not on by default.
• Distribute work across a fleet, or guarantee exactly-once across a queue boundary (single process is the unit of correctness; see the roadmap's scale-out section).
• Broker between multiple workflows in one process (one workflow per process is the deployment shape).
• Take a non-TOML input at run time — the TypeScript SDK authors workflows, but it compiles to the TOML the runtime executes; TOML stays the only thing the engine loads.
• Act as an LLM orchestration framework — it runs inference via the intelligence adapter, but chaining / planning logic stays in the workflow graph.

If a deployment wants those behaviours, the right move is an upstream or sibling service that composes agentd as a building block, not extending agentd itself.

6. Feedback / track new gaps

File an issue referencing this document and the concern row. When a gap closes, the row moves from its current colour to Green with a linked commit / PR. Quarterly review rewrites §1 and §2 from scratch.