Wiring harvest metrics into your Prometheus / OTLP pipeline

Autumn-harvest ships a MetricsRecorder trait and nine pre-defined metric instruments that cover every production-observable aspect of the engine. The default implementation (NoOpMetrics) is zero-cost; no samples are emitted until you register a recorder.

Recommended: the built-in scrape endpoint

If you're embedding Harvest via autumn-harvest-plugin's HarvestPlugin in an autumn-web app, this is the fastest path to a working Prometheus scrape — one builder call, no exporter to install, no route to wire, and no new dependency (rendering is done entirely through autumn-web's existing MetricsSource contract):

[dependencies]
autumn-harvest-plugin = { version = "0.10", features = ["metrics"] }

let app = autumn_web::app().plugin(
    HarvestPlugin::new()
        .workflows(workflows![onboarding])
        .activities(activities![send_email])
        .api("/api/harvest")
        .with_metrics_scrape(),
);

with_metrics_scrape() registers a HarvestMetricsRecorder as both the engine's MetricsRecorder (via HarvestBuilder::telemetry) and an autumn-web MetricsSource (via AppBuilder::metrics_source). The metrics it covers then appear on the app's one, already-shared /actuator/prometheus endpoint, alongside the app's own autumn_http_* families and any other plugin's metrics — there is deliberately no separate, Harvest-owned scrape route. This is the "one endpoint for autumn-web apps and their plugins" model: every plugin registers its metrics into the same scrape, instead of each mounting its own.

curl http://localhost:8080/actuator/prometheus

Auth posture is governed entirely by the app's own [actuator] config (actuator.prometheus, profile-aware sensitive-endpoint gating) — the same policy that already protects the app's other actuator endpoints, rather than a second, Harvest-specific toggle.

Coverage: this endpoint aggregates the nine ADR-0001 §7 catalogue metrics named in issue #355 (harvest.workflow.started, harvest.workflow.duration, harvest.activity.duration, harvest.timer.started, harvest.queue.depth, harvest.queue.paused, harvest.dlq.entries, harvest.schedule.runs, harvest.schedule.skipped, harvest.retention.deleted), plus harvest.queue.oldest_pending_age, harvest.worker.slots_in_use/slots_available/slot_target, and harvest.shard.stranded_pending — the engine's own background samplers already compute these under the same is_enabled() gate the nine required metrics live behind, so they're implemented too rather than sampled and discarded. It also covers the four broker-connector families (harvest.connector.received, harvest.connector.dispatched, harvest.connector.poisoned, harvest.connector.lag), which back the shipped connector dashboard panels — without them a dropped metric would be indistinguishable from an idle consumer. It also covers harvest.build.ramp_aborted from the build ramp guard, which backs the starter dashboard's ramp-abort panel. It also covers harvest.cost.hold from a worker cost hold (autumn#1720). It does not back the full starter alert pack (docs/alerts/starter-pack-v0.1.0.json), which also references metrics this endpoint never emits (e.g. harvest_workflow_terminal_total, harvest_activity_attempts_total/retries_total, harvest_schedule_fire_attempts_total, harvest_no_active_workers). If you want the full starter alert pack to work, use the metrics-rs adapter escape hatch below, which bridges every MetricsRecorder method. The optional SLO burn-rate pack (docs/alerts/slo.md) also needs that adapter. It reads the task-timeout counter, the canary counters, and histogram buckets.

Trade-off: autumn_web::actuator::MetricsSource's MetricKind only supports Counter/Gauge (no histogram variant), so harvest.workflow.duration and harvest.activity.duration are rendered as _count/_sum counter pairs rather than a bucketed histogram. That's enough for an average-latency query (rate(harvest_workflow_duration_sum[5m]) / rate(harvest_workflow_duration_count[5m])) but not for histogram_quantile(...). If you need full bucketed histograms (e.g. for the starter alert pack's harvest_queue_schedule_to_start_high p99 alert), OTLP export, or a custom recorder, use the metrics-rs adapter escape hatch below instead — a fresh HarvestBuilder::telemetry(...) call (or a second HarvestPlugin that never calls .with_metrics_scrape()) overrides nothing else in your app.

See autumn-harvest-plugin/examples/metrics_scrape_quickstart.rs for a complete runnable example: cargo run, curl .../actuator/prometheus, see all nine metrics after a single workflow run.

Escape hatch: metrics-exporter-prometheus for full histograms, OTLP, or a custom recorder

Reach for this path if you need bucketed histogram _bucket series, OTLP push, or you already have a global metrics-crate recorder installed for app-level metrics.

Enable the in-tree metrics-rs adapter in your application's Cargo.toml:

[dependencies]
autumn-harvest = { version = "0.10", features = ["metrics-rs"] }
metrics-exporter-prometheus = "0.16"

Then install an exporter and wire the recorder before starting any workers:

use autumn_harvest::metrics_rs_adapter::MetricsRsRecorder;
use autumn_harvest::telemetry::TelemetryConfig;
use metrics_exporter_prometheus::Matcher;
use std::sync::Arc;

// 1. Install the Prometheus exporter (do this once at process start).
//
//    `metrics-exporter-prometheus` renders every histogram as a Prometheus
//    *summary* (client-side quantiles) UNLESS you configure explicit bucket
//    boundaries. The starter alert pack pages on
//    `histogram_quantile(0.99, rate(harvest_queue_schedule_to_start_bucket[5m]))`,
//    which only exists when buckets are set — and server-side `_bucket` series
//    are the only form you can aggregate across replicas.
//
//    Scope the seconds-oriented boundaries to the *latency* histograms with
//    `set_buckets_for_metric` rather than the global `set_buckets`. A global
//    bucket set is applied to every histogram, including non-duration ones such
//    as `harvest.workflow.history_size` (durable event counts, soft threshold
//    10k) and the payload-size histogram (bytes) — seconds buckets would dump
//    every sample above 600 into `+Inf` and break count/byte dashboards.
//
//    NOTE: `Matcher::Full` is matched against the *sanitized* metric name —
//    `metrics-exporter-prometheus` calls `sanitize_metric_name` before looking up
//    the per-metric distribution — so the matcher must use the Prometheus form
//    with underscores (`harvest_queue_schedule_to_start`), NOT the dotted name
//    registered via the `metrics` macros. A dotted matcher silently fails to match
//    and the metric falls back to a summary (no `_bucket` series).
const LATENCY_BUCKETS: &[f64] = &[
    0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0,
    10.0, 30.0, 60.0, 120.0, 300.0, 600.0,
];
metrics_exporter_prometheus::PrometheusBuilder::new()
    .set_buckets_for_metric(
        Matcher::Full("harvest_queue_schedule_to_start".into()),
        LATENCY_BUCKETS,
    )
    .expect("valid bucket boundaries")
    .set_buckets_for_metric(
        Matcher::Full("harvest_workflow_duration".into()),
        LATENCY_BUCKETS,
    )
    .expect("valid bucket boundaries")
    .set_buckets_for_metric(
        Matcher::Full("harvest_activity_duration".into()),
        LATENCY_BUCKETS,
    )
    .expect("valid bucket boundaries")
    // Issue #1815: pool wait and DB op latency use the same seconds buckets.
    .set_buckets_for_metric(
        Matcher::Full("harvest_db_pool_wait_duration".into()),
        LATENCY_BUCKETS,
    )
    .expect("valid bucket boundaries")
    .set_buckets_for_metric(
        Matcher::Full("harvest_db_query_duration".into()),
        LATENCY_BUCKETS,
    )
    .expect("valid bucket boundaries")
    .set_buckets_for_metric(
        Matcher::Full("harvest_db_persist_phase_duration".into()),
        LATENCY_BUCKETS,
    )
    .expect("valid bucket boundaries")
    // history-size is a *count* histogram (durable events), not seconds — give
    // it its own boundaries so it stays usable for count dashboards/alerts.
    .set_buckets_for_metric(
        Matcher::Full("harvest_workflow_history_size".into()),
        &[10.0, 50.0, 100.0, 500.0, 1_000.0, 5_000.0, 10_000.0, 50_000.0],
    )
    .expect("valid bucket boundaries")
    .install()
    .expect("failed to install Prometheus exporter");

// 2. Build a TelemetryConfig that forwards samples to the global registry.
let telemetry = TelemetryConfig::builder()
    .metrics(Arc::new(MetricsRsRecorder))
    .build();

// 3. Pass it to HarvestBuilder.
let harvest = autumn_harvest::HarvestBuilder::new(pool)
    .telemetry(telemetry)
    // ... rest of your config
    .build();

The exporter now serves GET /metrics in Prometheus text format. Grafana / your OTLP collector can scrape it directly.

OTLP / OpenTelemetry

Replace step 1 with your OTel SDK setup and a metrics-exporter-otlp or opentelemetry-prometheus bridge. The MetricsRsRecorder adapter is backend-agnostic — it writes to whichever exporter the metrics crate's global recorder is pointing at.

To keep the Prometheus exporter and send OTLP from an OpenTelemetry Collector, use the Collector recipe. It also copies the metrics that match an OTel messaging semantic convention under the semconv name.

For spans, OTel has no workflow conventions in October 2026. The workflow conventions draft proposes them upstream.

Custom recorder

If you already have a metrics backend that does not use the metrics crate facade, implement MetricsRecorder directly:

use autumn_harvest::telemetry::{
    ActivityStatus, MetricsRecorder, WorkflowStatus,
};

struct MyRecorder { /* … */ }

impl MetricsRecorder for MyRecorder {
    fn record_workflow_started(&self, workflow_name: &str, queue: &str) {
        my_counter!("harvest.workflow.started", workflow = workflow_name, queue = queue);
    }
    // implement the other methods you care about; all have default no-ops
}

Custom workflow/activity metrics

Workflow and activity authors can emit business KPIs ("orders processed", "dollars charged") into the same MetricsRecorder pipeline the engine uses — without standing up a parallel metrics stack or risking double-counting on replay.

API

// Inside a #[workflow] function — emission is suppressed during replay:
#[workflow]
async fn checkout(ctx: &WorkflowContext, order: Order) -> Result<String, String> {
    let result = ctx.execute_activity(&charge_card_info(), order.clone()).await?;
    // Suppressed on every replay cycle; emitted at the live execution frontier.
    ctx.metrics().counter("orders_processed", 1, &[("tier", &order.tier)]);
    ctx.metrics().histogram("order_amount_usd", order.amount_usd, &[("tier", &order.tier)]);
    Ok(result)
}

// Inside a #[activity] function — always emitted (activities run once per attempt):
#[activity(start_to_close = "30s")]
async fn charge_card(ctx: &ActivityContext, order: Order) -> Result<String, String> {
    // Each retry of this activity is a separate execution — each emits once.
    ctx.metrics().counter("charge_attempts", 1, &[("payment_method", &order.payment_method)]);
    // … charge logic …
    Ok("txn-123".to_string())
}

Replay safety

Workflow metrics are suppressed during deterministic replay (WorkflowContext::is_replaying() == true). A workflow that has been suspended and resumed 50 times will not emit ctx.metrics() calls during those 50 re-invocation cycles the replay engine uses to reconstruct state — only at the live execution frontier are metrics emitted. No new WorkflowEvent variants are produced; the history remains byte-identical whether or not the workflow author emits custom metrics.

At-least-once in crash-recovery scenarios: if the worker crashes after a ctx.metrics() call but before the next event (activity schedule, timer, completion) commits to the database, the task is retried from the same frontier. The replay engine re-runs to that point, finds no new committed history past it, and emits the metric again. This is the same at-least-once behaviour Temporal's own workflow.GetMetricsHandler() exhibits. In practice, workflow-task durations are measured in milliseconds, so duplicate emissions are rare transients; design counter dashboards to be idempotent (sum/rate) rather than expecting exact counts.

Activity functions always run exactly once per attempt. Retries are separate executions and each emits independently — this is intentional and matches the "each attempt counts" semantics activities already have.

Namespacing

All names are automatically prefixed with harvest.user.:

Do not pass a name that starts with harvest. — it is reserved for engine metrics and will be rejected with a tracing::warn! (the call is dropped, not a hard error, so the workflow continues running).

Label cardinality rule (ADR-0001 §7)

The following label keys are forbidden because they carry per-execution cardinality that would blow up any metrics backend:

execution.id, activity.id, workflow.id, harvest.execution.id, harvest.workflow.id, harvest.activity.id, idempotency_key, run_id

Using any of these keys logs a tracing::warn! and silently drops the metric call. Use low-cardinality labels only (e.g. tier, region, payment_method). Additional limits: up to 16 labels per call; label keys and metric names up to 200 characters.

Zero-overhead when telemetry is off

If no MetricsRecorder is configured (the default no-op path), the is_enabled() → false short-circuit exits before validation, so there is zero overhead in the common "telemetry off" case.

metrics-rs bridge

When the metrics-rs feature is enabled, the three user-metric methods are bridged through MetricsRsRecorder using the low-level metrics::with_recorder API with dynamic names and Vec<Label>:

// Emits to whatever metrics exporter is installed globally (e.g. Prometheus).
ctx.metrics().counter("orders_processed", 1, &[("tier", "gold")]);

Metric catalogue (ADR-0001 §7)

See docs/adr/0001-otel-trace-contract.md for the canonical, authoritative catalogue. The table below lists where each metric is emitted in the source code.

Label sets

Cardinality rule: execution.id is never a metric label. It is span-only (see ADR-0001 §4). The MetricsRecorder API enforces this by construction — no record_* method accepts an ExecutionId.

build_id label

Five families carry a build_id label: harvest.workflow.terminal, harvest.activity.attempts, harvest.workflow.nondeterministic_block, harvest.workflow.duration and harvest.activity.duration. The value is the build of the worker that ran the task. An outcome that no worker code produced reports none. Examples are the timeout scanner, a cancel through a signal and a race-loser cancel.

telemetry::build_id_label caps the values. A process admits the first 16 distinct builds that it sees (MAX_BUILD_ID_LABELS), and it never evicts one. A later build, or a build id longer than 128 bytes, reports __other__. A real build id equal to a sentinel, or one that starts with build:, gets a build: prefix, so it never merges with a sentinel or another build. A worker process normally reports only its own build. harvest.workflow.non_determinism goes through the same cap.

A custom recorder gets the build through the *_for_build methods of MetricsRecorder. Their defaults drop the build and call the method without it, so an existing recorder needs no change. A recorder that forwards to another recorder must forward these methods too, or the build is lost.

The ramp guard does not read these metrics. It counts runs in the database by assigned_build_id. See docs/operations/build-ramp-guard.md.

Saga compensation metrics

harvest.saga.compensated counts compensation sequences that started running forward (unwind start — the earliest outage signal), and harvest.saga.compensation_failed counts unwinds that finished with at least one error (the dangling-state page). The exactly-once contract is keyed to durable MarkerRecorded dedup markers (saga_compensated:{seq} / saga_compensation_failed:{seq}): the counter fires only when the marker is first recorded on the live frontier, so the documented "compensations re-run on every replay" contract never double-counts, and pre-#801 marker-less histories are never counted retroactively. The failure counter is emitted in-Saga rather than at the worker terminal boundary, so it is separable from harvest.workflow.terminal{outcome=failed} and fires even when the workflow author catches SagaCompensationFailed and the run completes.

Per-unwind coherence (post-review hardening): the unwind's disposition is resolved once at unwind start and the failure counter follows it, so failed ≤ compensated holds per unwind. A counted unwind's failure is never suppressed by a trailing un-awaited signal (the failure marker is recorded past the drained signal), the cancel-and-compensate pattern is counted (a trailing WorkflowCancelled is transparent to the marker matcher), and an unwind entered at a drained-signal frontier — canonically a signal-with-start run whose unwind starts before the staged signal is awaited — is conservatively uncounted as a whole (the ctx.patched() signal-with-start caveat, inherited).

Crash-window caveat (at-least-once, both counters, durable and in-memory unwinds alike): each sample is emitted in-process within the single-decision-cycle gap before its dedup marker's batch commits — the start marker persists with the unwind's first dispatch batch, the failure marker with the post-unwind batch — so a worker crash (or a #383 pause-race decision discard) inside that gap re-emits after resume. A crash mid-unwind — between compensations, after the first batch committed — is exactly-once. A pure in-memory unwind (zero durable footprint) is the maximal case: a crash-resume anywhere within its single cycle can re-count — the metric mirror of the "compensations re-run wholesale" idempotency contract in docs/saga.md.

Grafana dashboard example queries

# Activity failure rate
rate(harvest_activity_duration_count{status="failed"}[5m])

# Current queue backlog, exact below 10,000 (issue #2356)
harvest_queue_depth{queue="default"}

# p99 schedule-to-start latency per queue (canonical worker-capacity SLI, issue #501)
histogram_quantile(0.99, sum by (le, queue) (rate(harvest_queue_schedule_to_start_bucket[5m])))

# Age of the oldest unclaimed eligible task per queue (fast-detection gauge, issue #501)
harvest_queue_oldest_pending_age{queue="default"}

# DLQ depth per shard
harvest_dlq_entries{shard="0"}

# Workflow history-bloat early warnings per type, per 15m window (issue #704).
# One increase per still-running execution the moment it first crosses the
# configured soft threshold (default 20.48% of event_hard_cap) -- pair a
# crossing with `GET /workflows?history_bloat_min_events=<N>` to rank offenders.
sum by (workflow) (increase(harvest_workflow_history_bloat_total[15m]))

# Worker dispatch-slot utilization per slot type (issue #531):
# "are my workers saturated?" — pairs with queue.depth to tell a worker
# bottleneck apart from a queue/DB bottleneck.
harvest_worker_slots_in_use / (harvest_worker_slots_in_use + harvest_worker_slots_available)

# Adaptive slot tuner's current target vs. the operator-configured band
# (issue #548) — confirms live behaviour matches [min_slots, max_slots].
harvest_worker_slot_target{slot_type="activity"}

# Tuner grow/shrink/hold rate per slot type — a steady stream of grow/shrink
# under flat load usually means the band or step size needs retuning.
rate(harvest_worker_tuner_decisions_total{decision!="hold"}[5m])

# Effective schedule run rate (runs - skips)
rate(harvest_schedule_runs_total[1h]) - rate(harvest_schedule_skipped_total[1h])

# p99 durable-mutex acquire wait per workflow (issue #691) — how long
# contenders queue before entering the critical section.
histogram_quantile(0.99, sum by (le, workflow) (rate(harvest_mutex_wait_duration_bucket[5m])))

# p99 durable-mutex held duration per workflow — long holds serialize peers
# and are the first thing to look at when acquire waits climb.
histogram_quantile(0.99, sum by (le, workflow) (rate(harvest_mutex_held_duration_bucket[5m])))

# Live durable-mutex contention depth (FIFO waiter-queue length at last grant).
harvest_mutex_contention_depth{workflow="apply_ledger_op"}

Full importable dashboard pack

The queries above are seed examples. A complete, versioned, importable Grafana dashboard covering the full metric catalogue (every METRIC_* constant in telemetry.rs, plus the literal-named concurrency gauges) ships at docs/dashboards/starter-pack-v0.1.0.json, with import instructions, prerequisites, and the alert-rule ↔ panel mapping table in docs/dashboards/README.md. Coverage is CI-enforced by autumn-harvest/tests/integration/dashboard_pack_docs.rs: a new METRIC_* constant with no dashboard panel turns the test red.