Support Center Home

Sampling Trace Data

You can use a number of sampling methods within the OpenTelemetry specification to limit or focus the volume of data you capture in your application. For example, you may want to exclude long-running background jobs or traces with a 200 OK status code. Two methods that we recommend for sampling are head-based sampling and tail-based sampling.

Head-based sampling is a probabilistic random sampling decision that is made at the beginning of the trace. To implement head-based sampling decisions, add the following OpenTelemetry trace sampler into your application:

SdkTracerProvider tracerProvider = SdkTracerProvider.builder()

By default, the tracing decision is made at the parent span and the decision is propagated through the entire trace. The above example samples 50% of traces. To change this, see configuration and additional details in OpenTelemetry’s sampling documentation.

Tail-based sampling

Tail-based sampling is a sampling decision made at the end of the trace and kept based on a set of defined criteria. Tail-based sampling decisions can be set at the Collector-level and need to be configured by the user.

OpenTelemetry supports the following policies:

Policy Description
always_sample Sample all traces.
numeric_attribute Sample based on number attributes.
string_attribute Sample based on string matches.
rate_limiting Sample based on rate.

The following example shows a configuration for rate limiting:

    decision_wait: 10s
    num_traces: 100
    expected_new_traces_per_sec: 10
         name: test-policy-1,
         type: rate_limiting,
	 Rate_limiting: {spans_per_second: 35}

See the full list of policies and additional details in OpenTelemetry Tail Sampling Processor documentation.

Adding custom tags to traces

Regardless of the instrumentation language, LogicMonitor requires the following custom tags to be specified during implementation if you want the traces to be mapped to existing monitored resources: ip, resource.type (which should be set to kubernetes-pod, cloud, or host), and (which should be the pod name for Kubernetes).

Capturing the above tags statically on your traces is important for mapping within LogicMonitor, but you may want to include additional attributes to your trace data to find more correlations and context while exploring and searching your data. For example, capturing metadata such as customer tier, product category, or geographical data, would provide important context to better understand what issue might be occurring.

You can set a key-value pair or group of pairs on any manually instrumented span. Below is an example for Java:

Span span = tracer.spanBuilder("/resource/path").setSpanKind(SpanKind.CLIENT).startSpan();
span.setAttribute("http.method", "GET");
span.setAttribute("http.url", url.toString());

For more information about span attributes and naming, refer to OpenTelemetry’s documentation and their defined list of semantic conventions.

In This Article