> ## Documentation Index
> Fetch the complete documentation index at: https://docs.obsy.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Sampling

> Reduce telemetry volume and cost with smart trace sampling.

Sampling lets you collect a representative subset of traces instead of every single span. This reduces the volume of data sent to your observability platform — cutting storage costs while keeping enough data to find problems.

Go to **Sampling** in the sidebar.

***

## Sampling modes

### Probabilistic sampling

A fixed percentage of all incoming traces is kept. Simple and low-overhead.

* **Global rate** — percentage of all traces to keep (e.g. `10` = keep 10%)
* All decisions are made at ingestion time, before the full trace is assembled

**Best for:** High-volume services where you want a consistent cost reduction across the board.

### Tail-based sampling

Sampling decisions are made *after* the full trace is assembled, so you can make smarter decisions based on outcome.

* **Global rate** — percentage of *normal* (error-free, latency-normal) traces to keep
* **Error traces** — always kept (100%)
* **Slow traces** (above latency threshold) — always kept (100%)
* **Service overrides** — set a different rate for specific services

**Best for:** Mixed workloads where you need 100% of error and slow traces but can drop healthy fast traces.

***

## Enabling sampling

1. Go to **Sampling**.
2. Toggle **Enable sampling** on.
3. Choose **Probabilistic** or **Tail-based**.
4. Set the global rate.
5. Click **Save**. Obsy syncs the config to your OTel collector within seconds via a Helm upgrade.

***

## Filter rules

In addition to sampling, you can completely drop certain traffic from being collected at all. These rules run before sampling.

### Drop by endpoint

Drop all spans for specific HTTP paths. Useful for noisy health check endpoints.

```
/health
/healthz
/readyz
/livez
/ping
/metrics
```

<Note>
  Obsy always drops `/health`, `/healthz`, `/readyz`, `/livez`, `/ping`, and `/metrics` by default regardless of your custom drop rules. This reduces noise from k8s liveness probes.
</Note>

To add custom endpoints to drop:

1. Go to **Sampling → Filter rules**.
2. Toggle **Custom drop rules** on.
3. Enter endpoint paths (one per line, supports prefix matching).

### Drop by HTTP method

Drop all spans for specific HTTP methods. Example: drop `OPTIONS` (CORS preflight) and `HEAD` requests.

### Drop by user agent

Drop spans from specific user agents. Useful for excluding internal monitoring bots, load balancer health checks, or synthetic monitors you don't want appearing in traces.

***

## Service-level overrides (tail-based only)

When using tail-based sampling, you can set a different sampling rate for individual services:

1. Go to **Sampling → Service overrides**.
2. Click **Add override**.
3. Select the service from your catalog and enter a rate (0–100%).
4. Click **Save**.

Example: set `payment-service` to 100% (keep all) while everything else runs at 10%.

***

## Cost impact

The **Sampling** page shows an estimated cost impact card based on your current ingestion volume and the configured rate. This is an estimate — actual savings depend on trace size distribution.

***

## How changes are applied

Sampling config changes trigger an automatic `helm upgrade` on all clusters. The upgrade is non-blocking (run with `--no-wait`) so changes appear in the UI immediately. New sampling settings take effect when the collector pods finish their rolling restart, typically within 30–60 seconds.