Telemetry Debt: Why Your Observability Bill Keeps Climbing While Nobody Looks at the Dashboards

Most teams we work with can tell you exactly how much their observability platform costs per month. Far fewer can tell you which of those dashboards anyone actually opened last quarter, or which alert rules fire weekly and get dismissed without investigation. That gap is telemetry debt: years of metrics, logs, and traces instrumentation accumulated without anyone going back to prune it.

It behaves like technical debt, but the interest compounds in your monitoring bill and in your on-call engineers' attention span. Left alone, it doesn't just cost money — it actively degrades incident response, because the signal you need during a real outage is buried under noise nobody bothered to clean up.

How Telemetry Debt Actually Accumulates

It rarely arrives as one bad decision. It's the sum of small, individually reasonable ones:

Reactive alert rules. An incident happens, someone adds an alert so "this never happens again," and it's never revisited once the underlying issue is fixed. Two years later that alert still fires monthly against a threshold nobody remembers choosing.
Orphaned dashboards. A dashboard gets built for a project, a migration, or a team that no longer exists in its original form. It stays in the shared folder, still consuming query load every time someone opens the tool, still costing you seat and API budget.
Unbounded custom metric cardinality. Someone tags a metric with user_id, order_id, or a raw request path instead of a bounded label set. Each unique value creates a new time series. On a busy service this turns one metric definition into millions of series, and most observability vendors price on active series count.
Debug-level logging left on in production. A verbose log line added to chase down a bug ships to production and is never turned back down. Multiply by every service and every deploy, and log ingestion volume — and cost — quietly triples.

None of these are mistakes in isolation. The debt is the absence of a process that ever comes back and asks "do we still need this."

A Practical Audit Method

Before proposing remediation to a client, we run a structured audit across three dimensions:

1. Dashboard usage. Most modern observability platforms (Grafana, Datadog, New Relic) expose view/access analytics per dashboard — either in the UI or via API. Pull last-90-day view counts. Anything with zero or near-zero views from a real user (not a scheduled export or synthetic check) is a deletion or archive candidate.

2. Alert fire-and-ignore ratio. Cross-reference alert history against incident/ticket history. An alert that fired 40 times in a quarter and correlated with an opened incident twice is not protecting anyone — it's training your on-call rotation to ignore Slack notifications. We look specifically for alerts with high fire count and low resolution-action rate.

3. Cardinality cost drivers. Most vendors expose a way to rank metrics by active series count or ingestion volume. If you're self-hosting Prometheus, you can find this directly:

# Top 10 metric names by number of active time series
topk(10, count by (__name__)({__name__=~".+"}))

# Find which label on a specific metric is driving cardinality
count by (label_name) (
  count by (label_name, __name__) (http_requests_total)
)

In practice, a handful of metrics — often ones with a user_id, session_id, or unbounded path label — account for a disproportionate share of total series. Fixing three or four metric definitions is often enough to meaningfully move the bill.

Remediation: The Technical Playbook

OpenTelemetry sampling

If you're instrumenting with OpenTelemetry (now the de facto standard for traces, metrics, and logs), sampling strategy is the single highest-leverage lever for trace volume:

Head-based sampling decides whether to keep a trace at the moment it starts, based on a fixed probability. It's cheap and simple, but it can drop the exact slow or erroring trace you'd want to investigate.
Tail-based sampling waits until a trace completes, then decides based on its actual outcome — keeping all errors and slow traces while dropping a sample of routine, fast, successful ones. It costs more (you need a collector that buffers spans and makes the sampling decision downstream), but it preserves the traces that matter most.

A simplified tail-sampling policy in the OpenTelemetry Collector might look like this:

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow-requests
        type: latency
        latency:
          threshold_ms: 500
      - name: sample-the-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

This is illustrative — production configs need tuning per policy type and combination logic — but the shape holds: always keep errors and outliers, sample the routine traffic aggressively.

Log level and structured logging discipline

Set an explicit policy: debug level is for local and staging environments only, and any debug line that ships to production requires a follow-up ticket to remove it or gate it behind a feature flag. Combine this with structured (JSON) logging so you can filter and aggregate on fields instead of grepping free text — which also reduces the incentive to over-log "just in case."

Alert consolidation tied to SLOs

Instead of a dozen low-signal alerts on individual infrastructure symptoms (CPU, memory, queue depth, disk), define a small number of SLOs (availability, latency, error rate) and alert on error budget burn rate. One well-tuned burn-rate alert replaces many noisy threshold alerts and directly reflects user impact rather than resource trivia.

Retention tiering

Not all telemetry needs the same retention or resolution. A reasonable default: full-resolution metrics and traces hot for 15–30 days, downsampled aggregates retained for 6–13 months for trend analysis, and raw high-cardinality data dropped after the incident-investigation window closes. Most platforms support this natively; if self-hosting, Prometheus's recording rules plus a long-term storage backend can achieve the same effect.

Ownership and Cadence

Debt accumulates because nobody owns removal — only addition. We recommend two things together:

A quarterly observability audit, treated with the same seriousness as a security review: dashboard usage report, alert fire-and-ignore report, top-N cardinality report, reviewed by SRE/platform team plus one representative from each service team.
Ownership at the source. The team that creates an alert or dashboard owns its lifecycle, including deletion. SRE/platform owns the audit process and tooling, but doesn't unilaterally delete other teams' instrumentation — it flags candidates and requires sign-off, which keeps the process fast without becoming adversarial.

Checklist

Pull dashboard view analytics for the last 90 days; archive anything unused
Cross-reference alert fire count against incident tickets; consolidate or delete fire-and-ignore alerts
Rank metrics by active series count; fix the top cardinality offenders first
Confirm debug-level logging is disabled in production by default
Adopt tail-based sampling for traces where error/latency visibility matters most
Define SLO-based burn-rate alerts to replace symptom-level threshold alerts
Set retention tiers: hot, downsampled, and dropped
Assign audit ownership (SRE/platform) and instrumentation ownership (originating team)
Schedule the next quarterly audit before closing this one

Telemetry debt doesn't announce itself the way an outage does. It shows up as a bill that grows every renewal and an on-call rotation that's slowly learning to tune out your monitoring. Treating pruning as a recurring practice, not a one-time cleanup, is what keeps it from coming back.