Manual Instrumentation

Instrumentation is the process of adding observability code to your application. There are two general types of instrumentation - automatic, and manual - and you should be familiar with both in order to effectively instrument your software.

Getting a Tracer

To create spans, you'll need to acquire or initialize a tracer first.

Initializing a new tracer

Ensure you have the right packages installed:

go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/trace \
  go.opentelemetry.io/otel/sdk \

Then initialize an exporter, resources, tracer provider, and finally a tracer.

package app

import (
    "context"
    "fmt"
    "log"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.17.0"
    "go.opentelemetry.io/otel/trace"
)

var tracer trace.Tracer

func newExporter(ctx context.Context)  /* (someExporter.Exporter, error) */ {
    // Your preferred exporter: console, jaeger, zipkin, OTLP, etc.
}

func newTraceProvider(exp sdktrace.SpanExporter) *sdktrace.TracerProvider {
    // Ensure default SDK resources and the required service name are set.
    r, err := resource.Merge(
        resource.Default(),
        resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("ExampleService"),
        ),
    )

    if err != nil {
        panic(err)
    }

    return sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
        sdktrace.WithResource(r),
    )
}

func main() {
    ctx := context.Background()

    exp, err := newExporter(ctx)
    if err != nil {
        log.Fatalf("failed to initialize exporter: %v", err)
    }

    // Create a new tracer provider with a batch span processor and the given exporter.
    tp := newTraceProvider(exp)

    // Handle shutdown properly so nothing leaks.
    defer func() { _ = tp.Shutdown(ctx) }()

    otel.SetTracerProvider(tp)

    // Finally, set the tracer that can be used for this package.
    tracer = tp.Tracer("ExampleService")
}

You can now access tracer to manually instrument your code.

Creating Spans

Spans are created by tracers. If you don't have one initialized, you'll need to do that.

To create a span with a tracer, you'll also need a handle on a context.Context instance. These will typically come from things like a request object and may already contain a parent span from an instrumentation library.

func httpHandler(w http.ResponseWriter, r *http.Request) {
    ctx, span := tracer.Start(r.Context(), "hello-span")
    defer span.End()

    // do some work to track with hello-span
}

In Go, the context package is used to store the active span. When you start a span, you'll get a handle on not only the span that's created, but the modified context that contains it.

Once a span has completed, it is immutable and can no longer be modified.

Get the current span

To get the current span, you'll need to pull it out of a context.Context you have a handle on:

// This context needs contain the active span you plan to extract.
ctx := context.TODO()
span := trace.SpanFromContext(ctx)

// Do something with the current span, optionally calling `span.End()` if you want it to end

This can be helpful if you'd like to add information to the current span at a point in time.

Create nested spans

You can create a nested span to track work in a nested operation.

If the current context.Context you have a handle on already contains a span inside of it, creating a new span makes it a nested span. For example:

func parentFunction(ctx context.Context) {
    ctx, parentSpan := tracer.Start(ctx, "parent")
    defer parentSpan.End()

    // call the child function and start a nested span in there
    childFunction(ctx)

    // do more work - when this function ends, parentSpan will complete.
}

func childFunction(ctx context.Context) {
    // Create a span to track `childFunction()` - this is a nested span whose parent is `parentSpan`
    ctx, childSpan := tracer.Start(ctx, "child")
    defer childSpan.End()

    // do work here, when this function returns, childSpan will complete.
}

Once a span has completed, it is immutable and can no longer be modified.

Span Attributes

Attributes are keys and values that are applied as metadata to your spans and are useful for aggregating, filtering, and grouping traces. Attributes can be added at span creation, or at any other time during the lifecycle of a span before it has completed.

// setting attributes at creation...
ctx, span = tracer.Start(ctx, "attributesAtCreation", trace.WithAttributes(attribute.String("hello", "world")))
// ... and after creation
span.SetAttributes(attribute.Bool("isTrue", true), attribute.String("stringAttr", "hi!"))

Attribute keys can be precomputed, as well:

var myKey = attribute.Key("myCoolAttribute")
span.SetAttributes(myKey.String("a value"))

Semantic Attributes

Semantic Attributes are attributes that are defined by the OpenTelemetry Specification in order to provide a shared set of attribute keys across multiple languages, frameworks, and runtimes for common concepts like HTTP methods, status codes, user agents, and more. These attributes are available in the go.opentelemetry.io/otel/semconv/v1.12.0 package.

For details, see Trace semantic conventions.

Events

An event is a human-readable message on a span that represents "something happening" during it's lifetime. For example, imagine a function that requires exclusive access to a resource that is under a mutex. An event could be created at two points - once, when we try to gain access to the resource, and another when we acquire the mutex.

span.AddEvent("Acquiring lock")
mutex.Lock()
span.AddEvent("Got lock, doing work...")
// do stuff
span.AddEvent("Unlocking")
mutex.Unlock()

A useful characteristic of events is that their timestamps are displayed as offsets from the beginning of the span, allowing you to easily see how much time elapsed between them.

Events can also have attributes of their own -

span.AddEvent("Cancelled wait due to external signal", trace.WithAttributes(attribute.Int("pid", 4328), attribute.String("signal", "SIGHUP")))

Set span status

A status can be set on a span, typically used to specify that there was an error in the operation a span is tracking - .Error.

import (
    // ...
    "go.opentelemetry.io/otel/codes"
    // ...
)

// ...

result, err := operationThatCouldFail()
if err != nil {
    span.SetStatus(codes.Error, "operationThatCouldFail failed")
}

By default, the status for all spans is Unset. In rare cases, you may also wish to set the status to Ok. This should generally not be necessary, though.

Record errors

If you have an operation that failed and you wish to capture the error it produced, you can record that error.

import (
    // ...
    "go.opentelemetry.io/otel/codes"
    // ...
)

// ...

result, err := operationThatCouldFail()
if err != nil {
    span.SetStatus(codes.Error, "operationThatCouldFail failed")
    span.RecordError(err)
}

It is highly recommended that you also set a span's status to Error when using RecordError, unless you do not wish to consider the span tracking a failed operation as an error span. The RecordError function does not automatically set a span status when called.

Creating Metrics

The metrics API is currently unstable, documentation TBA.

Propagators and Context

Traces can extend beyond a single process. This requires context propagation, a mechanism where identifiers for a trace are sent to remote processes.

In order to propagate trace context over the wire, a propagator must be registered with the OpenTelemetry API.

import (
  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/propagation"
)
...
otel.SetTextMapPropagator(propagation.TraceContext{})

OpenTelemetry also supports the B3 header format, for compatibility with existing tracing systems (go.opentelemetry.io/contrib/propagators/b3) that do not support the W3C TraceContext standard.

After configuring context propagation, you'll most likely want to use automatic instrumentation to handle the behind-the-scenes work of actually managing serializing the context.