Add system properties bridge #15339

zeitlinger · 2025-11-17T17:09:09Z

to bridge system properties / env vars to declarative config

Fixes #14192

Alternative implementation for #14767

Breaking Changes:

otel.experimental.javascript-snippet becomes otel.instrumentation.servlet.experimental.javascript-snippet

zeitlinger · 2025-11-17T17:44:45Z

@trask I really like this new bridge based on your idea 😄

...on-api/src/main/java/io/opentelemetry/instrumentation/api/internal/ConfigPropertiesUtil.java

trask · 2025-11-24T22:53:31Z

...on-api/src/main/java/io/opentelemetry/instrumentation/api/internal/ConfigPropertiesUtil.java

+  /** Returns true if the given OpenTelemetry instance supports Declarative Config. */
+  public static boolean isDeclarativeConfig(OpenTelemetry openTelemetry) {
+    return isIncubator && openTelemetry instanceof ExtendedOpenTelemetry;
+  }
+


is isIncubator needed, since instrumentation-api (unfortunately) already depends on the api-incubator?

only for the test (see last comment)

zeitlinger · 2025-11-25T14:56:48Z

I have no idea what could have caused this failure:

11:31:59.660 [Test worker] INFO  t.g.i.19421579350 - Pulling docker image: ghcr.io/open-telemetry/opentelemetry-java-instrumentation/smoke-test-zulu-openjdk-8u31:20251117.19421579350. Please be patient; this may take some time but only needs to be done once.
11:32:00.526 [docker-java-stream--1576344563] INFO  t.g.i.19421579350 - Starting to pull image
11:32:00.527 [docker-java-stream--1576344563] INFO  t.g.i.19421579350 - Pulling image layers:  0 pending,  0 downloaded,  0 extracted, (0 bytes/0 bytes)
11:32:00.983 [docker-java-stream--1576344563] INFO  t.g.i.19421579350 - Pulling image layers:  1 pending,  1 downloaded,  0 extracted, (13 MB/? MB)
11:32:01.834 [docker-java-stream--1576344563] INFO  t.g.i.19421579350 - Pulling image layers:  1 pending,  1 downloaded,  1 extracted, (129 MB/? MB)
11:32:01.972 [docker-java-stream--1576344563] INFO  t.g.i.19421579350 - Pulling image layers:  0 pending,  2 downloaded,  1 extracted, (143 MB/148 MB)
11:32:03.747 [docker-java-stream--1576344563] INFO  t.g.i.19421579350 - Pulling image layers:  0 pending,  2 downloaded,  2 extracted, (148 MB/148 MB)
11:32:03.751 [Test worker] INFO  t.g.i.19421579350 - Image ghcr.io/open-telemetry/opentelemetry-java-instrumentation/smoke-test-zulu-openjdk-8u31:20251117.19421579350 pull took PT4.090551S
11:32:03.751 [docker-java-stream--1576344563] INFO  t.g.i.19421579350 - Pull complete. 2 layers, pulled in 3s (downloaded 148 MB at 49 MB/s)
11:32:03.753 [Test worker] INFO  t.g.i.19421579350 - Creating container for image: ghcr.io/open-telemetry/opentelemetry-java-instrumentation/smoke-test-zulu-openjdk-8u31:20251117.19421579350
11:32:04.376 [Test worker] INFO  t.g.i.19421579350 - Container ghcr.io/open-telemetry/opentelemetry-java-instrumentation/smoke-test-zulu-openjdk-8u31:20251117.19421579350 is starting: d09147ceccc4abd46f5c923ad2025cb51e2f7d27659e9b20370bf9c4d5e45420
11:32:04.459 [docker-java-stream-822006274] INFO  i.o.smoketest.CrashEarlyJdk8Test - STDOUT: started
11:32:04.467 [Test worker] INFO  t.g.i.19421579350 - Container ghcr.io/open-telemetry/opentelemetry-java-instrumentation/smoke-test-zulu-openjdk-8u31:20251117.19421579350 started in PT0.713852S
Container.ExecResult(exitCode=134, stdout=compiling
finish compiling
executing
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000000076e60fe08, pid=31, tid=140468591523520
#
# JRE version: OpenJDK Runtime Environment (8.0_31-b13) (build 1.8.0_31-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.31-b07 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  0x000000076e60fe08
#
# Core dump written. Default location: //core or core.31
#
# An error report file with more information is saved as:
# //hs_err_pid31.log
Compiled method (c1)     807 1777       3       java.util.stream.AbstractPipeline::wrapSink (37 bytes)
 total in heap  [0x00007fc150965350,0x00007fc150965a78] = 1832
 relocation     [0x00007fc150965478,0x00007fc1509654e0] = 104
 main code      [0x00007fc1509654e0,0x00007fc150965840] = 864
 stub code      [0x00007fc150965840,0x00007fc1509658f8] = 184
 metadata       [0x00007fc1509658f8,0x00007fc150965908] = 16
 scopes data    [0x00007fc150965908,0x00007fc1509659a8] = 160
 scopes pcs     [0x00007fc1509659a8,0x00007fc150965a58] = 176
 dependencies   [0x00007fc150965a58,0x00007fc150965a60] = 8
 nul chk table  [0x00007fc150965a60,0x00007fc150965a78] = 24
Compiled method (c1)     807 1782       3       java.util.stream.StreamSpliterators$WrappingSpliterator$$Lambda$254/1777443462::get$Lambda (9 bytes)
 total in heap  [0x00007fc15096f350,0x00007fc15096f7e8] = 1176
 relocation     [0x00007fc15096f478,0x00007fc15096f4b8] = 64
 main code      [0x00007fc15096f4c0,0x00007fc15096f680] = 448
 stub code      [0x00007fc15096f680,0x00007fc15096f710] = 144
 oops           [0x00007fc15096f710,0x00007fc15096f718] = 8
 metadata       [0x00007fc15096f718,0x00007fc15096f730] = 24
 scopes data    [0x00007fc15096f730,0x00007fc15096f780] = 80
 scopes pcs     [0x00007fc15096f780,0x00007fc15096f7e0] = 96
 dependencies   [0x00007fc15096f7e0,0x00007fc15096f7e8] = 8
#
# If you would like to submit a bug report, please visit:
#   http://www.azulsystems.com/support/
#
, stderr=[otel.javaagent 2025-11-25 11:32:05:162 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.23.0-SNAPSHOT
/test.sh: line 7:    31 Aborted                 (core dumped) java -javaagent:opentelemetry-javaagent.jar CrashEarlyJdk8
)

laurit · 2025-11-25T15:21:59Z

I have no idea what could have caused this failure:

CrashEarlyJdk8 fails when you use lambdas too early in the agent initalization.

zeitlinger · 2025-11-25T15:54:56Z

I have no idea what could have caused this failure:

CrashEarlyJdk8 fails when you use lambdas too early in the agent initalization.

thank you!

zeitlinger · 2025-11-26T11:16:41Z

@trask please check again

...va/io/opentelemetry/instrumentation/awssdk/v2_2/internal/AbstractAwsSdkTelemetryFactory.java

zeitlinger · 2025-12-02T10:41:40Z

@trask I've split the PR - please have another look

...ap/src/main/java/io/opentelemetry/javaagent/bootstrap/servlet/ExperimentalSnippetHolder.java

...sting/src/main/java/io/opentelemetry/instrumentation/awssdk/v2_2/AbstractAws2ClientTest.java

...va/io/opentelemetry/instrumentation/awssdk/v2_2/internal/AbstractAwsSdkTelemetryFactory.java

trask · 2025-12-02T23:45:51Z

.../java/io/opentelemetry/instrumentation/awssdk/v1_11/autoconfigure/TracingRequestHandler.java

+                .orElse(false))
+        .setMessagingReceiveInstrumentationEnabled(
+            ConfigPropertiesUtil.getBoolean(
+                    openTelemetry, "messaging", "experimental", "receive_telemetry", "enabled")


and here

"messaging", "receive_telemetry/development", "enabled"

yes, I think so.

We'd also need to add this translation to the declarative config bridge - but we can first add it here

We have these cases - and both are currently used about equally as much:

experimental-foo -> foo/development

experimental.foo -> foo/development

We can't translate back from foo/development - unless we try multiple properties - and that gets harder to troubleshoot.

(this is why the tests are failing)

just realized that we can just map all system properties with "otel." prefix to the corresponding declarative config value - and thus avoid the problem.

I think this is similar to what you suggested @trask

No, we can treat - like ., because config properties treat them the same - so it should be simple:

https://github.com/open-telemetry/opentelemetry-java/blob/d9fce84689031883155ad0959948fa7fda7295f4/api/all/src/main/java/io/opentelemetry/api/internal/ConfigUtil.java#L74-L76

found a solution now

…tal-foo" and "experimental.foo" that both translate to "foo/development"

zeitlinger · 2025-12-05T13:05:38Z

@trask please check again

trask

nice

trask · 2025-12-05T22:45:46Z

...va/io/opentelemetry/instrumentation/awssdk/v2_2/internal/AbstractAwsSdkTelemetryFactory.java

  }

-  protected abstract boolean getBoolean(String name, boolean defaultValue);
+  protected abstract boolean getBoolean(boolean defaultValue, String... name);


do we need this abstract method now that (I think) we have unified access between agent and library instrumentation?

We have almost - but you can only use property customizers (for system properties) in the agent.

I'm not following why that's a problem, IIUC that happens at startup, then declarative config is backed by the result?

This issue exists when NOT using declarative config:

Agent: You can set the property using a customizer, e.g. via default properties

Library: property is directly read from system properties or environment variables

trask · 2025-12-05T22:48:28Z

...ap/src/main/java/io/opentelemetry/javaagent/bootstrap/servlet/ExperimentalSnippetHolder.java

-    String result = ConfigPropertiesUtil.getString("otel.experimental.javascript-snippet");
-    return result == null ? "" : result;
+    return ConfigPropertiesUtil.getString(
+            GlobalOpenTelemetry.get(), "servlet", "javascript-snippet/development")


can you add breaking label and add this to the CHANGELOG so I don't forget?

trask · 2025-12-05T22:51:30Z

...api/src/main/java/io/opentelemetry/instrumentation/api/instrumenter/InstrumenterBuilder.java

+        ConfigPropertiesUtil.isDeclarativeConfig(openTelemetry)
+            ? ConfigPropertiesUtil.getString(
+                    openTelemetry, "common", "span_suppression_strategy/development")
+                .orElse(null)
+            : ConfigPropertiesUtil.getString(
+                "otel.instrumentation.experimental.span-suppression-strategy");


can this be unified now?

We can if we accept another breaking change (the common part)

ah, I missed that, this one may be more used, can you open an issue to change it in next major version bump (3.0)?

zeitlinger self-assigned this Nov 17, 2025

zeitlinger requested a review from a team as a code owner November 17, 2025 17:09

github-actions bot added the test native This label can be applied to PRs to trigger them to run native tests label Nov 17, 2025

zeitlinger mentioned this pull request Nov 17, 2025

Jdbc declarative config params #15199

Closed

zeitlinger added this to Declarative Configuration: Java Nov 17, 2025

zeitlinger moved this to Awaiting Review in Declarative Configuration: Java Nov 17, 2025

zeitlinger changed the title ~~Add system properties bridge~~ Add system properties bridge, part 1 Nov 17, 2025

zeitlinger force-pushed the system-properties-bridge branch from 3a9e285 to 9ae95f1 Compare November 18, 2025 09:54

zeitlinger changed the title ~~Add system properties bridge, part 1~~ Add system properties bridge Nov 18, 2025

trask reviewed Nov 24, 2025

View reviewed changes

zeitlinger force-pushed the system-properties-bridge branch from 0138d4f to 16e62ea Compare November 25, 2025 08:15

zeitlinger force-pushed the system-properties-bridge branch from cf18ce0 to 18d719d Compare November 27, 2025 13:15

zeitlinger mentioned this pull request Nov 27, 2025

clean up ConfigPropertiesUtil #15479

Draft

zeitlinger commented Nov 27, 2025

View reviewed changes

...va/io/opentelemetry/instrumentation/awssdk/v2_2/internal/AbstractAwsSdkTelemetryFactory.java Outdated Show resolved Hide resolved

trask reviewed Dec 2, 2025

View reviewed changes

zeitlinger force-pushed the system-properties-bridge branch from 55146b8 to deddaa1 Compare December 3, 2025 10:59

zeitlinger added 8 commits December 4, 2025 17:53

add system properties bridge

51a532f

add system properties bridge

68d64c8

add tests

2acda70

add list

edc6d1c

refactor

a78bf62

refactor

ac66918

string

91b03f3

string

6aa96f1

zeitlinger added 21 commits December 4, 2025 17:53

pr review

501a769

use optional instead of providing default for boolean

5c30b44

use optional instead of providing default for int

c5c34ed

use optional instead of providing default for list

7ca08d7

use optional instead of providing default for string

1d72e01

fix

669236a

fix

e7072ba

fix

d921bab

split pr

6724cb7

split pr

d1a9784

fix

41aac72

fix

9245a86

pr review

278259e

revert aws test

a3fbf23

use "/development" instead of "experimental" in declarative config

93adb59

use "/development" instead of "experimental" in declarative config

7c81df5

fix

7bc8ed9

docs

91b0050

avoid double bridge

d9f56ba

normalize system props before lookup to avoid ambiguity of "experimen…

b542ea1

…tal-foo" and "experimental.foo" that both translate to "foo/development"

normalize system props before lookup to avoid ambiguity of "experimen…

6c889a7

…tal-foo" and "experimental.foo" that both translate to "foo/development"

zeitlinger force-pushed the system-properties-bridge branch from 2df0753 to 6c889a7 Compare December 4, 2025 16:54

zeitlinger added 5 commits December 5, 2025 07:59

normalize system props before lookup to avoid ambiguity of "experimen…

2cc9cf0

…tal-foo" and "experimental.foo" that both translate to "foo/development"

normalize system props before lookup to avoid ambiguity of "experimen…

fe2886b

…tal-foo" and "experimental.foo" that both translate to "foo/development"

Merge branch 'main' into system-properties-bridge

c9d0e19

rename js snippet property

d7d0ba8

normalize system props before lookup to avoid ambiguity of "experimen…

d2f1b6b

…tal-foo" and "experimental.foo" that both translate to "foo/development"

zeitlinger added this to the v2.23.0 milestone Dec 5, 2025

trask reviewed Dec 5, 2025

View reviewed changes

Add system properties bridge #15339

Are you sure you want to change the base?

Add system properties bridge #15339

Uh oh!

Conversation

zeitlinger commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zeitlinger commented Nov 17, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zeitlinger commented Nov 25, 2025

Uh oh!

laurit commented Nov 25, 2025

Uh oh!

zeitlinger commented Nov 25, 2025

Uh oh!

zeitlinger commented Nov 26, 2025

Uh oh!

Uh oh!

zeitlinger commented Dec 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zeitlinger commented Dec 5, 2025

Uh oh!

trask left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zeitlinger commented Nov 17, 2025 •

edited

Loading