Following a Request Across Services with OpenTelemetry Trace IDs

When an application runs as a single service, investigating a failed request is usually straightforward. Developers can inspect application logs, check the database, and identify where the request failed. The situation becomes more complicated when one request passes through multiple services.

Consider an ASP.NET Core API that receives an order, calls a payment service, and then sends a request to an inventory service. If the payment operation takes too long, checking each service's logs separately may not reveal the complete sequence of events. Each service knows what happened locally, but developers may struggle to connect those events to the original request.

OpenTelemetry distributed tracing helps solve this problem by connecting related operations through trace and span identifiers. Instead of investigating independent log entries, developers can follow a request across service boundaries and understand where time was spent or where a failure occurred.

This article explains how trace IDs work, how to configure distributed tracing in ASP.NET Core, and how to use trace information when diagnosing production issues.

What Is Distributed Tracing?

Distributed tracing records the path of an operation as it moves through an application and its dependencies. Each operation is represented by a span, which contains information such as its name, timing, parent relationship, and status.

A trace groups related spans into a single request journey.

For example, an order request might produce this trace:

Order API
  ├── Validate Order
  ├── Call Payment Service
  │     └── Process Payment
  └── Call Inventory Service
        └── Reserve Stock

Each span represents an individual operation. Together, they describe the work performed for the original request.

OpenTelemetry provides APIs, instrumentation libraries, and exporters for collecting and transmitting this telemetry. It can send trace data to a compatible observability backend, where developers can inspect the request flow.

Trace ID vs. Span ID

Trace IDs and span IDs serve different purposes.

A trace ID generally remains the same as a request moves between instrumented services. Each span has its own span ID, allowing the tracing system to represent the relationships between operations.

For example, the order API and payment service can have different span IDs while belonging to the same trace.

This distinction matters during troubleshooting. Searching by trace ID reveals the overall request journey, while examining individual spans helps identify the operation responsible for a delay or failure.

How Trace Context Moves Between Services

For distributed tracing to work, services must exchange trace context when making requests to one another.

OpenTelemetry commonly uses the W3C Trace Context standard. Its traceparent HTTP header carries the trace ID, the caller's span ID, and trace flags.

A simplified example looks like this:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

The header contains four fields:

1. Version: Identifies the trace context format version.

2. Trace ID: Identifies the distributed trace.

3. Parent ID: Identifies the span that initiated the outgoing request.

4. Trace flags: Carry tracing flags, including the sampling decision.

When an instrumented service receives a request with valid trace context, it can create a new span associated with the incoming trace. When it makes another instrumented HTTP request, the outgoing request can carry updated context that identifies the current span as the caller.

The result is a connected trace rather than several unrelated traces.

In typical ASP.NET Core applications, the OpenTelemetry ASP.NET Core and HTTP client instrumentations work with .NET's activity infrastructure to handle this process. Developers generally do not need to construct traceparent headers manually.

Step 1: Install the OpenTelemetry Packages

For an ASP.NET Core application, install the hosting integration, ASP.NET Core instrumentation, HTTP client instrumentation, and OTLP exporter.

Run these commands in the project directory:

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

Use package versions that are compatible with your target .NET framework and the rest of your application's dependencies. The examples below assume the corresponding APIs are available in the installed package versions.

The packages have separate responsibilities:

• OpenTelemetry.Extensions.Hosting integrates telemetry configuration with the .NET hosting model.

• OpenTelemetry.Instrumentation.AspNetCore captures incoming ASP.NET Core request activity.

• OpenTelemetry.Instrumentation.Http captures outgoing HTTP client activity.

• OpenTelemetry.Exporter.OpenTelemetryProtocol exports telemetry using OTLP, which supported observability backends and OpenTelemetry Collectors can receive.

These packages establish the foundation for tracing requests entering and leaving the service.

Step 2: Configure OpenTelemetry Tracing

Configure tracing in Program.cs. The following example enables ASP.NET Core and HTTP client instrumentation and sends trace data to an OTLP endpoint.

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddControllers();

builder.Services
    .AddOpenTelemetry()
    .ConfigureResource(resource =>
        resource.AddService(
            serviceName: "order-api"))
    .WithTracing(tracing =>
    {
        tracing
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddOtlpExporter(options =>
            {
                options.Endpoint =
                    new Uri("http://localhost:4317");
            });
    });

builder.Services.AddHttpClient<PaymentClient>(client =>
{
    client.BaseAddress =
        new Uri("http://localhost:5001");

    client.Timeout = TimeSpan.FromSeconds(10);
});

var app = builder.Build();

app.MapControllers();

app.Run();

Replace the example endpoint with the address of your actual OTLP receiver. The address shown is a local development example, not a production endpoint. Ensure the receiver supports the configured OTLP transport.

The ConfigureResource call sets the service name. This is useful when multiple applications send telemetry to the same backend because developers can distinguish the order API from the payment or inventory service.

The two instrumentation calls serve different purposes:

• AddAspNetCoreInstrumentation() captures incoming HTTP requests handled by ASP.NET Core.

• AddHttpClientInstrumentation() captures outgoing requests made through supported .NET HTTP client infrastructure.

The exporter transmits the collected spans to the configured endpoint. Without an active exporter or another configured telemetry destination, enabling instrumentation alone does not make traces available in an external tracing interface.

Use the Correct Endpoint Configuration

The endpoint must match the receiver's OTLP protocol configuration. OTLP commonly uses port 4317 for gRPC and port 4318 for HTTP, but the actual endpoint and transport depend on the collector or backend configuration.

In a deployed environment, avoid hardcoding environment-specific endpoints in application source code. Use configuration or environment variables instead, and ensure the application has network access to the receiver.

Step 3: Send Requests Through HttpClient

Outgoing requests should use an instrumented HTTP client. In ASP.NET Core, IHttpClientFactory is a practical way to manage client configuration and lifetime.

For example, create a typed client for the payment service:

public sealed class PaymentClient
{
    private readonly HttpClient _httpClient;

    public PaymentClient(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }

    public async Task<string> ProcessPaymentAsync(
        CancellationToken cancellationToken)
    {
        using var response = await _httpClient.PostAsync(
            "/payments",
            content: null,
            cancellationToken);

        response.EnsureSuccessStatusCode();

        return await response.Content.ReadAsStringAsync(
            cancellationToken);
    }
}

The example uses a POST request without a body only to demonstrate the tracing flow. A real payment operation would send the required payment data using an appropriate request model and serialization strategy.

Registering the typed client with AddHttpClient lets the application use the configured HttpClient instance. With the HTTP client instrumentation enabled, the outgoing request can create a client span and propagate trace context automatically when the request participates in an active trace.

The receiving payment service must also have tracing configured to continue the trace. If it does not, the order API may record its outgoing request, but the payment service's internal work will not appear as instrumented spans.

Avoid Manually Creating HttpClient Instances

Creating an HttpClient directly is not inherently incompatible with tracing, but it can bypass your application's configured client factory, handlers, and other conventions. It can also lead to connection-management problems when instances are created and discarded incorrectly.

Prefer IHttpClientFactory or a deliberately managed client where appropriate. The important tracing requirement is that outgoing requests pass through the instrumented HTTP client pipeline and preserve their trace context.

Step 4: Read the Current Trace ID in Application Code

.NET uses the Activity abstraction to represent operations that can participate in distributed tracing. The current operation is commonly accessible through Activity.Current.

You can read its trace and span IDs when adding useful diagnostic information to application logs.

using System.Diagnostics;

public static class TraceDiagnostics
{
    public static void LogCurrentTrace(
        ILogger logger)
    {
        var activity = Activity.Current;

        if (activity is null)
        {
            logger.LogDebug(
                "No active tracing activity is available.");

            return;
        }

        logger.LogInformation(
            "Processing operation. TraceId: {TraceId}, SpanId: {SpanId}",
            activity.TraceId,
            activity.SpanId);
    }
}

This helper checks whether an activity exists before attempting to log its identifiers. Some code paths may run without an active tracing activity, depending on how they are invoked and how instrumentation is configured.

Use structured logging placeholders rather than building the message through string concatenation. Structured properties are easier for compatible logging systems to index and query.

However, do not assume every log entry will automatically contain trace identifiers. That behavior depends on the logging provider and configuration. If your logging pipeline already enriches records with trace and span IDs, adding them manually to every message can create unnecessary duplication.

For request-level troubleshooting, the goal is to make it possible to move from a log entry to the corresponding distributed trace.

Step 5: Follow a Request Across Multiple Services

Consider a request that starts at order-api, calls payment-api, and then calls inventory-api.

The trace might contain spans similar to these:

The trace viewer uses span relationships and timing information to display these operations together.

Suppose the payment HTTP span takes several seconds, while the payment service's server span is short. This may indicate that part of the delay occurred outside the measured server operation, such as network transit, queuing, or work performed in an uninstrumented component. It does not prove which factor caused the delay; additional evidence is required.

If the payment service's server span is itself long, inspect its child spans, logs, and dependencies to determine where the time was spent.

A useful investigation process is:

1. Find the failed request or slow operation in the tracing backend.

2. Open the complete trace using its trace ID.

3. Identify the longest spans and any spans marked with errors.

4. Compare client-side and server-side spans for the same dependency call.

5. Inspect relevant logs and application metrics.

6. Verify the suspected cause with additional diagnostics before making a change.

This approach narrows the investigation from an entire application to the operations associated with one request.

Step 6: Add Custom Spans for Important Operations

Automatic instrumentation captures many HTTP operations, but it cannot understand every business operation inside your code. Custom spans help you measure important steps such as order validation, inventory checks, and payment preparation.

You can create a custom activity using ActivitySource:

using System.Diagnostics;

public static class OrderTracing
{
    public static readonly ActivitySource ActivitySource =
        new("OrderApi.OrderProcessing");

    public static async Task ValidateOrderAsync(
        CancellationToken cancellationToken)
    {
        using var activity =
            ActivitySource.StartActivity("ValidateOrder");

        cancellationToken.ThrowIfCancellationRequested();

        // Perform order validation here.
        await Task.CompletedTask;
    }
}

Register the activity source with the tracing configuration:

.WithTracing(tracing =>
{
    tracing
        .AddSource("OrderApi.OrderProcessing")
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddOtlpExporter();
});

The custom activity becomes a span when a listener is configured to collect it. When the operation runs inside an existing trace, it can be connected to the current activity automatically.

Use custom spans for operations that help explain application behavior. Avoid creating spans for every trivial method call, because excessive instrumentation increases telemetry volume and makes traces harder to read.

Common Problems and Troubleshooting

Distributed tracing depends on instrumentation, context propagation, sampling, and successful export. If a trace looks incomplete, investigate each part of that pipeline.

Check Context Propagation

If a service creates a new HTTP request without the expected trace context, the downstream operation may not join the original trace. Review custom HTTP handlers, middleware, proxies, and service infrastructure for components that remove or replace tracing headers.

Avoid manually copying trace IDs into arbitrary headers as a substitute for proper context propagation. Use supported instrumentation and standard trace context unless you have a specific integration requirement.

Check Sampling

Sampling controls which traces are recorded or exported. A trace that is not present in the backend may have been excluded by the sampling policy rather than lost during export.

During local debugging, a development sampling configuration can help make traces easier to inspect. In production, select a policy that balances diagnostic value, telemetry cost, and system overhead. Confirm the implications of the chosen sampling strategy before relying on traces for incident investigation.

Check Exporter Configuration

If spans are created but never reach the tracing backend, verify that the application can connect to the OTLP receiver and that the configured protocol matches the receiver's settings.

A collector can also help centralize telemetry processing, routing, and export. When using one, check both the application's exporter configuration and the collector's receiver and exporter pipelines.

Production Best Practices

Use Consistent Service Names

Give every service a stable, meaningful service name, such as order-api, payment-api, or inventory-api. Configure the service name through OpenTelemetry resource attributes so the backend can group telemetry by service.

Avoid assigning a different service name to every request. A service name should identify the application component, not an individual operation.

Keep Trace Context Separate from Business Data

Trace IDs are diagnostic identifiers, not authentication credentials or authorization tokens. Do not use them to make access-control decisions.

Likewise, avoid recording sensitive payment details, passwords, access tokens, or personal data in span attributes, events, or baggage. Baggage can propagate across service boundaries, so any values placed in it should be selected carefully.

Instrument Meaningful Operations

Start with inbound requests, outbound HTTP calls, and important dependencies. Add custom spans when they answer a concrete operational question.

For example, a span around inventory reservation can help distinguish a slow inventory operation from time spent elsewhere in order processing. Include only attributes that help with filtering or diagnosis, and avoid high-cardinality values that can increase storage and query costs.

Connect Traces with Logs and Metrics

Tracing is most useful when it complements other telemetry:

• Traces show the path of an individual request and the timing of its operations.

• Logs provide detailed diagnostic messages and error context.

• Metrics reveal aggregate behavior, such as request rates, error rates, and latency distributions.

Use compatible logging and telemetry configuration so engineers can move between a failed request, its logs, and the relevant traces. Do not rely on trace data alone to establish the cause of an incident.

Test Propagation Before Deployment

Create a test that sends a request through at least two instrumented services. Verify that the downstream span belongs to the expected trace and that parent-child relationships are correct.

Also test failure cases, cancellation, and requests that run without an existing activity. These checks can reveal broken propagation or missing instrumentation before the application reaches production.

Summary

OpenTelemetry distributed tracing makes it possible to follow a request across multiple ASP.NET Core services. A trace ID identifies the overall request journey, while span IDs identify individual operations and help represent their relationships.

By configuring ASP.NET Core and HTTP client instrumentation, using IHttpClientFactory, exporting traces through OTLP, and adding custom spans where necessary, developers can investigate slow requests and failures with better context.

For reliable results, use consistent service names, verify trace context propagation, configure sampling deliberately, and connect traces with logs and metrics. These practices turn distributed tracing into a practical diagnostic tool for understanding how an application behaves across service boundaries.