跳转至

Welcome to the OpenTelemetry for Erlang/Elixir getting started guide! This guide will walk you through the basic steps in installing, configuring, and exporting data from OpenTelemetry.

Phoenix

This part of the guide will show you how to get started with OpenTelemetry in the Phoenix Web Framework.

Prerequisites

Ensure that you have erlang, elixir, postgres (or the database of your choice), and phoenix installed locally. The phoenix installation guide will help you get set up with everything you need.

Example Application

The following example uses a basic Phoenix web application. For reference, a complete example of the code you will build can be found here: opentelemetry-erlang-contrib/examples/dice_game. You can git clone that project or just follow along in your browser.

Additional examples can be found here.

Dependencies

We'll need a few other dependencies that Phoenix doesn't come with.

  • opentelemetry_api: contains the interfaces you'll use to instrument your code. Things like Tracer.with_span and Tracer.set_attribute are defined here.
  • opentelemetry: contains the SDK that implements the interfaces defined in the API. Without it, all the functions in the API are no-ops.
  • opentelemetry_exporter: allows you to send your telemetry data to an OpenTelemetry Collector and/or to self-hosted or commercial services.
  • opentelemetry_phoenix: creates OpenTelemetry spans from the Elixir :telemetry events created by Phoenix.
  • opentelemetry_cowboy: creates OpenTelemetry spans from the Elixir :telemetry events created by the Cowboy web server (which is used by Phoenix).
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# mix.exs
def deps do
  [
    {:opentelemetry, "~> 1.3"},
    {:opentelemetry_api, "~> 1.2"},
    {:opentelemetry_exporter, "~> 1.4"},
    {:opentelemetry_phoenix, "~> 1.1"},
    {:opentelemetry_cowboy, "~> 0.2"}
  ]
end

The last two also need to be setup when your application starts:

1
2
3
4
5
6
# application.ex
@impl true
def start(_type, _args) do
  :opentelemetry_cowboy.setup()
  OpentelemetryPhoenix.setup(adapter: :cowboy2)
end

If you're using ecto, you'll also want to add OpentelemetryEcto.setup([:dice_game, :repo]).

We also need to configure the opentelemetry application as temporary by adding a releases section to your project configuration. This will ensure that if it terminates, even abnormally, the dice_game application will be terminated.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# mix.exs
def project do
  [
    app: :dice_game,
    version: "0.1.0",
    elixir: "~> 1.14",
    elixirc_paths: elixirc_paths(Mix.env()),
    start_permanent: Mix.env() == :prod,
    releases: [
      dice_game: [
        applications: [opentelemetry: :temporary]
      ]
    ],
    aliases: aliases(),
    deps: deps()
  ]
end

Now we can use the new mix setup command to install the dependencies, build the assets, and create and migrate the database.

Try It Out

We can ensure everything is working by setting the stdout exporter as opentelemetry's traces_exporter and then starting the app with mix phx.server.

1
2
# config/dev.exs
config :opentelemetry, traces_exporter: {:otel_exporter_stdout, []}

If everything went well, you should be able to visit localhost:4000 in your browser and see quite a few lines that look like this in your terminal.

(Don't worry if the format looks a little unfamiliar. Spans are recorded in the Erlang record data structure. You can find more information about records here, and this file describes the span record structure, and explains what the different fields are.)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
*SPANS FOR DEBUG*
{span,64480120921600870463539706779905870846,11592009751350035697,[],
      undefined,<<"/">>,server,-576460731933544855,-576460731890088522,
      {attributes,128,infinity,0,
                  #{'http.status_code' => 200,
                    'http.client_ip' => <<"127.0.0.1">>,
                    'http.flavor' => '1.1','http.method' => <<"GET">>,
                    'http.scheme' => <<"http">>,'http.target' => <<"/">>,
                    'http.user_agent' =>
                        <<"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36">>,
                    'net.transport' => 'IP.TCP',
                    'net.host.name' => <<"localhost">>,
                    'net.host.port' => 4000,'net.peer.port' => 62839,
                    'net.sock.host.addr' => <<"127.0.0.1">>,
                    'net.sock.peer.addr' => <<"127.0.0.1">>,
                    'http.route' => <<"/">>,'phoenix.action' => home,
                    'phoenix.plug' =>
                        'Elixir.DiceGameWeb.PageController'}},
      {events,128,128,infinity,0,[]},
      {links,128,128,infinity,0,[]},
      undefined,1,false,
      {instrumentation_scope,<<"opentelemetry_phoenix">>,<<"1.1.0">>,
                             undefined}}

These are the raw Erlang records that will get serialized and sent when you configure the exporter for your preferred service.

Rolling The Dice

Now we'll check out the API endpoint that will let us roll the dice and return a random number between 1 and 6.

Before we call our API, let's add our first bit of manual instrumentation. In our DiceController we call a private dice_roll method that generates our random number. This seems like a pretty important operation, so in order to capture it in our trace we'll need to wrap it in a span.

1
2
3
4
5
  defp dice_roll do
    Tracer.with_span("dice_roll") do
      to_string(Enum.random(1..6))
    end
  end

It would also be nice to know what number it generated, so we can extract it as a local variable and add it as an attribute on the span.

1
2
3
4
5
6
7
8
9
  defp dice_roll do
    Tracer.with_span("dice_roll") do
      roll = Enum.random(1..6)

      Tracer.set_attribute(:roll, roll)

      to_string(roll)
    end
  end

Now if you point your browser/curl/etc. to localhost:4000/api/rolldice you should get a random number in response, and 3 spans in your console.

View the full spans 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
*SPANS FOR DEBUG*
{span,224439009126930788594246993907621543552,5581431573601075988,[],
      undefined,<<"/api/rolldice">>,server,-576460729549928500,
      -576460729491912750,
      {attributes,128,infinity,0,
                  #{'http.request_content_length' => 0,
                    'http.response_content_length' => 1,
                    'http.status_code' => 200,
                    'http.client_ip' => <<"127.0.0.1">>,
                    'http.flavor' => '1.1','http.host' => <<"localhost">>,
                    'http.host.port' => 4000,'http.method' => <<"GET">>,
                    'http.scheme' => <<"http">>,
                    'http.target' => <<"/api/rolldice">>,
                    'http.user_agent' =>
                        <<"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36">>,
                    'net.host.ip' => <<"127.0.0.1">>,
                    'net.transport' => 'IP.TCP',
                    'http.route' => <<"/api/rolldice">>,
                    'phoenix.action' => roll,
                    'phoenix.plug' => 'Elixir.DiceGameWeb.DiceController'}},
      {events,128,128,infinity,0,[]},
      {links,128,128,infinity,0,[]},
      undefined,1,false,
      {instrumentation_scope,<<"opentelemetry_cowboy">>,<<"0.2.1">>,
                             undefined}}

{span,237952789931001653450543952469252891760,13016664705250513820,[],
      undefined,<<"HTTP GET">>,server,-576460729422104083,-576460729421433042,
      {attributes,128,infinity,0,
                  #{'http.request_content_length' => 0,
                    'http.response_content_length' => 1258,
                    'http.status_code' => 200,
                    'http.client_ip' => <<"127.0.0.1">>,
                    'http.flavor' => '1.1','http.host' => <<"localhost">>,
                    'http.host.port' => 4000,'http.method' => <<"GET">>,
                    'http.scheme' => <<"http">>,
                    'http.target' => <<"/favicon.ico">>,
                    'http.user_agent' =>
                        <<"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36">>,
                    'net.host.ip' => <<"127.0.0.1">>,
                    'net.transport' => 'IP.TCP'}},
      {events,128,128,infinity,0,[]},
      {links,128,128,infinity,0,[]},
      undefined,1,false,
      {instrumentation_scope,<<"opentelemetry_cowboy">>,<<"0.2.1">>,
                             undefined}}

{span,224439009126930788594246993907621543552,17387612312604368700,[],
      5581431573601075988,<<"dice_roll">>,internal,-576460729494399167,
      -576460729494359917,
      {attributes,128,infinity,0,#{roll => 2}},
      {events,128,128,infinity,0,[]},
      {links,128,128,infinity,0,[]},
      undefined,1,false,
      {instrumentation_scope,<<"dice_game">>,<<"0.1.0">>,undefined}}
<<"/api/rolldice">>

This is the first span in the request, aka the root span. That undefined next to the span name tells you that it doesn't have a parent span. The two very large negative numbers are the start and end time of the span, in the native time unit. If you're curious, you can calculate the duration in milliseconds like so System.convert_time_unit(-576460729491912750 - -576460729549928500, :native, :millisecond). The phoenix.plug and phoenix.action will tell you the controller and function that handled the request. You'll notice however, that the instrumentation_scope is opentelemetry_cowboy. When we told opentelemetry_phoenix's setup function that we want to use the :cowboy2 adapter, that let it know not to create and additional span, but to instead append attributes to the existing cowboy span. This ensures we have more accurate latency data in our traces.

<<"HTTP GET">>

This is the request for the favicon, which you can see in the 'http.target' => <<"/favicon.ico">> attribute. I believe it has a generic name because it does not have an http.route.

<<"dice_roll">>

This is the custom span we added to our private method. You'll notice it only has the one attribute that we set, roll => 2. You should also note that it is part of the same trace as our <<"/api/rolldice">> span, 224439009126930788594246993907621543552 and has a parent span id of 5581431573601075988 which is the span id of the <<"/api/rolldice">> span. That means that this span is a child of that one, and will be shown below it when rendered in your tracing tool of choice.

Next Steps

Enrich your automatically generated instrumentation with manual instrumentation of your own codebase. This allows you to customize the observability data your application emits.

You'll also want to configure an appropriate exporter to export your telemetry data to one or more telemetry backends.

Creating a New Mix/Rebar Project

To get started with this guide, create a new project with rebar3 or mix:

{{< tabpane langEqualsHeader=true >}}

{{< tab Erlang >}} rebar3 new release otel_getting_started {{< /tab >}}

{{< tab Elixir >}} mix new --sup otel_getting_started {{< /tab >}}

{{< /tabpane >}}

Then, in the project you just created, add both opentelemetry_api and opentelemetry as dependencies. We add both because this is a project we will run as a Release and export spans from.

{{< tabpane langEqualsHeader=true >}}

{{< tab Erlang >}} {deps, [{opentelemetry_api, "~> 1.2"}, {opentelemetry, "~> 1.3"}]}. {{< /tab >}}

{{< tab Elixir >}} def deps do [ {:opentelemetry_api, "~> 1.2"}, {:opentelemetry, "~> 1.3"} ] end {{< /tab >}}

{{< /tabpane >}}

In the case of Erlang, the API Application will also need to be added to src/otel_getting_started.app.src and a relx section to rebar.config. In an Elixir project, a releases section needs to be added to mix.exs:

{{< tabpane langEqualsHeader=true >}}

{{< tab Erlang >}} %% src/otel_getting_started.app.src ... {applications, [kernel, stdlib, opentelemetry_api]}, ...

%% rebar.config {relx, [{release, {otel_getting_started, "0.1.0"}, [{opentelemetry, temporary}, otel_getting_started]},

1
   ...]}.

{{< /tab >}}

{{< tab Elixir >}}

mix.exs

releases: [ otel_getting_started: [ version: "0.0.1", applications: [opentelemetry: :temporary, otel_getting_started: :permanent] ] ] {{< /tab >}}

{{< /tabpane >}}

The SDK opentelemetry should be added as early as possible in the Release boot process to ensure it is available before any telemetry is produced. Here it is also set to temporary under the assumption that we prefer to have a running Release not producing telemetry over crashing the entire Release.

In addition to the API and SDK, an exporter for getting data out is needed. The SDK comes with an exporter for debugging purposes that prints to stdout and there are separate packages for exporting over the OpenTelemetry Protocol (OTLP) and the Zipkin protocol.

Initialization and Configuration

Configuration is done through the Application environment or OS Environment Variables. The SDK (opentelemetry Application) uses the configuration to initialize a Tracer Provider, its Span Processors and the Exporter.

Using the Console Exporter

Exporters are packages that allow telemetry data to be emitted somewhere - either to the console (which is what we're doing here), or to a remote system or collector for further analysis and/or enrichment. OpenTelemetry supports a variety of exporters through its ecosystem, including popular open source tools like Jaeger and Zipkin.

To configure OpenTelemetry to use a particular exporter, in this case otel_exporter_stdout, the Application environment for opentelemetry must set the exporter for the span processor otel_batch_processor, a type of span processor that batches up multiple spans over a period of time:

{{< tabpane langEqualsHeader=true >}}

{{< tab Erlang >}} %% config/sys.config.src [ {opentelemetry, [{span_processor, batch}, {traces_exporter, {otel_exporter_stdout, []}}]} ]. {{< /tab >}}

{{< tab Elixir >}}

config/runtime.exs

config :opentelemetry, span_processor: :batch, traces_exporter: {:otel_exporter_stdout, []} {{< /tab >}}

{{< /tabpane >}}

Working with Spans

Now that the dependencies and configuration are set up, we can create a module with a function hello/0 that starts some spans:

{{< tabpane langEqualsHeader=true >}}

{{< tab Erlang >}} %% apps/otel_getting_started/src/otel_getting_started.erl -module(otel_getting_started).

-export([hello/0]).

-include_lib("opentelemetry_api/include/otel_tracer.hrl").

hello() -> %% start an active span and run a local function ?with_span(operation, #{}, fun nice_operation/1).

nice_operation(_SpanCtx) -> ?add_event(<<"Nice operation!">>, [{<<"bogons">>, 100}]), ?set_attributes([{another_key, <<"yes">>}]),

1
2
3
4
5
6
%% start an active span and run an anonymous function
?with_span(<<"Sub operation...">>, #{},
           fun(_ChildSpanCtx) ->
                   ?set_attributes([{lemons_key, <<"five">>}]),
                   ?add_event(<<"Sub span event!">>, [])
           end).

{{< /tab >}}

{{< tab Elixir >}}

lib/otel_getting_started.ex

defmodule OtelGettingStarted do require OpenTelemetry.Tracer, as: Tracer

def hello do Tracer.with_span :operation do Tracer.add_event("Nice operation!", [{"bogons", 100}]) Tracer.set_attributes([{:another_key, "yes"}])

1
2
3
4
5
  Tracer.with_span "Sub operation..." do
    Tracer.set_attributes([{:lemons_key, "five"}])
    Tracer.add_event("Sub span event!", [])
  end
end

end end {{< /tab >}}

{{< /tabpane >}}

In this example, we're using macros that use the process dictionary for context propagation and for getting the tracer.

Inside our function, we're creating a new span named operation with the with_span macro. The macro sets the new span as active in the current context -- stored in the process dictionary, since we aren't passing a context as a variable.

Spans can have attributes and events, which are metadata and log statements that help you interpret traces after-the-fact. The first span has an event Nice operation!, with attributes on the event, as well as an attribute set on the span itself.

Finally, in this code snippet, we can see an example of creating a child span of the currently-active span. When the with_span macro starts a new span, it uses the active span of the current context as the parent. So when you run this program, you'll see that the Sub operation... span has been created as a child of the operation span.

To test out this project and see the spans created, you can run with rebar3 shell or iex -S mix, each will pick up the corresponding configuration for the release, resulting in the tracer and exporter to started.

{{< tabpane langEqualsHeader=true >}}

{{< tab Erlang >}} $ rebar3 shell ===> Compiling otel_getting_started Erlang/OTP 23 [erts-11.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Eshell V11.1 (abort with ^G) 1> 1> otel_getting_started:hello(). true SPANS FOR DEBUG {span,177312096541376795265675405126880478701,5706454085098543673,undefined, 13736713257910636645,<<"Sub operation...">>,internal, -576460750077844044,-576460750077773674, [{lemons_key,<<"five">>}], [{event,-576460750077786044,<<"Sub span event!">>,[]}], [],undefined,1,false,undefined} {span,177312096541376795265675405126880478701,13736713257910636645,undefined, undefined,operation,internal,-576460750086570890, -576460750077752627, [{another_key,<<"yes">>}], [{event,-576460750077877345,<<"Nice operation!">>,[{<<"bogons">>,100}]}], [],undefined,1,false,undefined} {{< /tab >}}

{{< tab Elixir >}} $ iex -S mix Erlang/OTP 23 [erts-11.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Compiling 1 file (.ex) Interactive Elixir (1.11.0) - press Ctrl+C to exit (type h() ENTER for help) iex(1)> OtelGettingStarted.hello() true iex(2)> SPANS FOR DEBUG {span,180094370450826032544967824850795294459,5969980227405956772,undefined, 14276444653144535440,<<"Sub operation...">>,'INTERNAL', -576460741349434100,-576460741349408901, [{lemons_key,<<"five">>}], [{event,-576460741349414157,<<"Sub span event!">>,[]}], [],undefined,1,false,undefined} {span,180094370450826032544967824850795294459,14276444653144535440,undefined, undefined,:operation,'INTERNAL',-576460741353342627, -576460741349400034, [{another_key,<<"yes">>}], [{event,-576460741349446725,<<"Nice operation!">>,[{<<"bogons">>,100}]}], [],undefined,1,false,undefined} {{< /tab >}}

{{< /tabpane >}}

Exporting to the OpenTelemetry Collector

The Collector provides a vendor agnostic way to receive, process and export telemetry data. The package opentelemetry_exporter provides support for both exporting over both HTTP (the default) and gRPC to the collector, which can then export Spans to a self-hosted service like Zipkin or Jaeger, as well as commercial services. For a full list of available exporters, see the registry.

For testing purposes the opentelemetry-erlang repo has a Collector configuration, config/otel-collector-config.yaml that can be used as a starting point. This configuration is used in docker-compose.yml to start the Collector with receivers for both HTTP and gRPC that then export to Zipkin also run by docker-compose.

To export to the running Collector the opentelemetry_exporter package must be added to the project's dependencies:

{{< tabpane langEqualsHeader=true >}}

{{< tab Erlang >}} {deps, [{opentelemetry_api, "~> 1.3"}, {opentelemetry, "~> 1.3"}, {opentelemetry_exporter, "~> 1.4"}]}. {{< /tab >}}

{{< tab Elixir >}} def deps do [ {:opentelemetry_api, "~> 1.3"}, {:opentelemetry, "~> 1.3"}, {:opentelemetry_exporter, "~> 1.4"} ] end {{< /tab >}}

{{< /tabpane >}}

It should then be added to the configuration of the Release before the SDK Application to ensure the exporter's dependencies are started before the SDK attempts to initialize and use the exporter.

Example of Release configuration in rebar.config and for mix's Release task:

{{< tabpane langEqualsHeader=true >}}

{{< tab Erlang >}} %% rebar.config {relx, [{release, {my_instrumented_release, "0.1.0"}, [opentelemetry_exporter, {opentelemetry, temporary}, my_instrumented_app]},

1
   ...]}.

{{< /tab >}}

{{< tab Elixir >}}

mix.exs

def project do [ releases: [ my_instrumented_release: [ applications: [opentelemetry_exporter: :permanent, opentelemetry: :temporary] ],

1
2
  ...
]

] end {{< /tab >}}

{{< /tabpane >}}

Finally, the runtime configuration of the opentelemetry and opentelemetry_exporter Applications are set to export to the Collector. The configurations below show the defaults that are used if none are set, which are the HTTP protocol with endpoint of localhost on port 4318. If using grpc for the otlp_protocol the endpoint should be changed to http://localhost:4317.

{{< tabpane langEqualsHeader=true >}}

{{< tab Erlang >}} %% config/sys.config.src [ {opentelemetry, [{span_processor, batch}, {traces_exporter, otlp}]},

{opentelemetry_exporter, [{otlp_protocol, http_protobuf}, {otlp_endpoint, "http://localhost:4318"}]}]} ]. {{< /tab >}}

{{< tab Elixir >}}

config/runtime.exs

config :opentelemetry, span_processor: :batch, traces_exporter: :otlp

config :opentelemetry_exporter, otlp_protocol: :http_protobuf, otlp_endpoint: "http://localhost:4318" {{< /tab >}}

{{< /tabpane >}}