App reference

ADR-0013: Metrics instrumentation

Metadata

  • Status: Accepted
  • Date: 2026-07-25
  • Deciders: Eagraí Clainne Team
  • Related: ADR-0007 (JWT authentication) and ADR-0008 (RBAC) — the auth surface counted here; ADR-0009 (service-layer authorization) — the svc seam the entity counter hangs off; ADR-0002 (centralised RPC authorization behind auth.Allow) — the interceptor-layer denial point

Context

The server already exports three OTel signals over one OTLP pipeline (push to a collector when EAG_OTLP_ENDPOINT is set, stdout exporters otherwise), but the only metrics on it are what otelconnect emits per RPC: rpc.server.duration, request/response sizes and message counts, tagged by service and method. Nothing else is instrumented — no domain events, no database health, no runtime, no auth outcomes. There is no Meter() call in the codebase, and the MeterProvider built in cmd/server is handed only to the otelconnect interceptor.

Two audiences want more, with equal weight:

  • Operations: is the system healthy — DB pool saturation, query latency, GC pressure, error and denial rates.
  • The family: what is the system doing — sign-ins, chores completed, rewards claimed.

The codebase gives natural seams to hang this on: every entity write funnels through svc.Create/svc.Mutate/svc.Delete (ADR-0009), every query through the typed table methods in internal/database, and every authorization decision through auth.Allow or the svc guards.

Decision

  1. One pipeline, no scrape endpoint. Metrics continue to leave the process exclusively via the existing OTLP push (stdout fallback). No Prometheus /metrics endpoint; a collector fans out to whatever backend wants the data. cmd/server now also calls otel.SetMeterProvider(mp), so packages reach the meter through the global — matching how the log bridge already uses the global provider — rather than threading it through constructors.

  2. All custom instruments live in internal/telemetry. One package owns instrument creation (lazy, sync.Once) and exposes intent-named recording functions (CountLogin, RecordDBOperation, …). Call sites never touch the OTel API directly; auth, svc, database and handlers depend only on internal/telemetry, avoiding instrument duplication and import cycles.

  3. Domain metrics are hybrid: one generic seam counter plus a few named counters.

    • eagraiclainne.entity.mutations {entity, action, outcome} is recorded once in the svc seam (Create/Mutate/Delete), covering all CRUD on all entities with zero per-handler code. The entity name is derived from the proto message descriptor, so new entities are covered automatically. outcome is success, skipped (idempotent no-op via ErrSkipMutation) or error.
    • Where the seam cannot see semantics, an explicit counter records the domain event: eagraiclainne.auth.logins {outcome}, eagraiclainne.auth.tokens.issued, eagraiclainne.item.completions {complete}, eagraiclainne.reward.claims.

  4. Authorization denials are counted at both layers. eagraiclainne.authz.denials carries layer=interceptor with the RPC procedure (a closed set) when auth.Allow refuses, and layer=service with the guard name (owner, event_member, admin) when an ADR-0009 guard refuses. Failed logins plus denial rates are the brute-force and misconfigured-permission signals.

  5. Database instrumentation sits on our own seam, not a driver wrapper. eagraiclainne.db.operation.duration {table, operation, outcome} is recorded in the typed table methods — table-aware attributes a driver-level wrapper cannot provide, and no new dependency. Pool health comes from sql.DBStats observable gauges (open/idle/in-use connections, wait count and wait duration), registered when the production pool is created and unregistered on close.

  6. Go runtime metrics come from go.opentelemetry.io/contrib/instrumentation/runtime started at boot — memory, GC, goroutines under the standard go.* semconv names.

  7. Cardinality is bounded by construction: no user identifiers in metric attributes. Every attribute is a closed set — entity, action, outcome, procedure, guard, table, operation. Per-person questions ("who completed the most chores?") are answered from the database and its mutation audit trail, which already record every actor; the metrics pipeline stays fixed-size forever.

  8. Naming: custom instruments are OTel-style under the eagraiclainne. prefix; standard areas (RPC, runtime) keep their semconv names. Default histogram buckets and temporality — no custom views until a dashboard proves the need.

  9. Testing bar: unit tests with sdk/metric.NewManualReader assert the load-bearing instruments fire with the right attributes — the seam counter, the DB histogram, logins and denials — not every attribute combination.

Consequences

Positive

  • Both halves of the audience are served from one pipeline: RED-style ops signals (RPC metrics + DB + runtime + denials) and family-level activity (logins, completions, claims) with no second export path to run.
  • The seam counter means CRUD coverage is automatic: a future entity type gets mutation metrics the day svc.Mutate handles it.
  • Bounded attribute sets mean the series count cannot grow with usage — safe for any backend, nothing PII-ish in telemetry.
  • internal/telemetry keeps the OTel API out of business code; swapping buckets, adding views or renaming instruments is a one-package change.

Negative / trade-offs

  • No pull endpoint: a bare Prometheus cannot scrape the binary; a collector (or one deployed later) is a hard prerequisite for metrics in production.
  • Global meter provider trades explicit dependency wiring for convenience; tests must set the global before instruments first fire in the process.
  • Per-person analytics deliberately excluded from metrics — Grafana alone cannot draw a leaderboard; that view must query the DB.
  • The generic entity.mutations counter records that an update happened, not what changed; anything needing more semantics gets its own counter, which is a judgement call each time.

Alternatives considered

  • Native Prometheus /metrics endpoint (alongside or instead of OTLP): works collector-less, but adds a second pipeline to keep consistent and diverges metrics from the traces/logs path. Rejected while the OTLP collector route serves the homelab.
  • Per-user attributes on domain counters: a family is small, so cardinality is bounded in practice — but the DB already answers per-person questions exactly, and identity labels in telemetry are a liability with zero ops value. Rejected.
  • otelsql-style driver wrapper: per-query spans and metrics for free, but attributes are raw-SQL-level (no table/operation semantics) and it adds a dependency for what four table methods can record themselves. Rejected.
  • Injecting the MeterProvider through constructors: more explicit, but touches every constructor and fixture for no behavioural gain when the global is already the pattern for logs. Rejected.
  • Per-handler explicit counters everywhere (no seam counter): maximally explicit, ~25 call sites to keep in step, drift guaranteed. Rejected.