Support per-process declarative configuration #923
Description
Problem Statement
The OpenTelemetry Operator is planning to rework their Instrumentation resource to support the new OpenTelemetry declarative config file format as part of their v1beta1 milestone. It would be really nice if I was able to use OBI as the implementation of the Instrumentation Kubernetes resources in the cluster the same way that they can be used with auto-instrumentation agents today. That would enable some pretty neat consumption experiences, like changing the sampling rate while an application is running, or changing which attributes are aggregated away with a View.
From a broader perspective, I want to be able to use OBI as a drop-in replacement for manual instrumentation. Ideally, this means I get the same configuration capabilities (i.e. declarative configuration) at the same granularity (per-process).
Many of the pieces are already there: pod-label and k8s-namespace based service discovery, (global) exporter configuration, sampling config, aggregations, etc.
Proposal
Overall, I propose starting with a subset of the declarative configuration, and adding support for more of it over time.
To start, migrate
and to be structured the same way as the declarative configuration is structured. Then, add fields over time as.Potential issues and concerns (i'm sure there are many):
- Many exporters will be more resource-intensive than a single exporter, and lose some of the benefits of batching. This might be a bit of a footgun for users.
- Some parts of the declarative configuration might be impossible for us to support in a way that is exactly the same. For example, OBI can't (can it?) open a port in the containers' namespace to serve a Prometheus exporter /metrics endpoint. Telemetry sent from the exporter also won't have the source IP of the pod, which means it won't interact with the k8sattributes processor the same way.
- Some parts of the declarative config won't apply: We (presumably) won't implement a "simple span processor" that blocks while the span is exported.
We need to think about how we will handle cases where the declarative configuration can't be implemented, or doesn't make sense. It would be nice if we could ignore unsupported config fields, and log a warning, rather than crashing.
Alternative considered
We could instead keep our own configuration structure, and add features similar to how sampling is supported: #345. The operator could map portions of the instrumentation resource to OBI configuration based on what is or isn't supported.
This would mean we can evolve independently of declarative config, and don't need to try to support multiple exporters. But it also means we don't get the benefits of supporting a config surface users are already familiar with.