Query by certain tags not working and I can't figure out why

[tomkcook]

I have Jaeger v2 and OpenSearch running in a docker compose stack. My compose services and Jaeger config.yaml are shown below.

I have a service that is reporting spans using the OTel SDK. The top-level span in each trace has a tag called vtba.ap_group_uuid while a subordinate span in each trace has both that tag and another called vtba.sta_mac. These are submitted as attributes on each span. The first contains a UUID represented as a string in the usual format (eg a15068f5-c702-4ad9-a0b4-ccf60bd2ddb9) while the second contains a MAC address in colon-separated notation (eg 9e:88:18:3c:e3:80).

I'm trying to query for these, embedding the traces display in an iframe in another website. I'm using a query string such as this:

end=1764081585029000&start=1764070785029000&lookback=3h&limit=100&service=vTBA+Gateway&operation=PSK+datagram+receiver&uiEmbed=v0&tags=%7B%22vtba.ap_group_uuid%22%3A%2213508881-a78f-4dc2-89c6-7af57776c0d3%22%2C%22vtba.sta_mac%22%3A%227c%3A3d%3A09%3A50%3A18%3A49%22%7D

After undoing the URL-encoding, the tags parameter looks like this:

{"vtba.ap_group_uuid":"13508881-a78f-4dc2-89c6-7af57776c0d3","vtba.sta_mac":"7c:3d:09:50:18:49"}

If I only query by vtba.ap_group_uuid then everything works as expected. If I add the vtba.sta_mac parameter as above, I always get no results. I have confirmed repeatedly that there really are traces with those tag values.

I've checked the OpenSearch index mapping and it includes this:

{
  "jaeger-span-2025-11-25": {
    "mappings": {
      ...
      "properties": {
        ...
        "tag": {
            "properties": {
                "vtba@ap_group_uuid": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "vtba@sta_mac": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
                ...

As far as I can tell, this is correct. Gemini suggests that tag values containing colons might be the issue but I have a fairly low confidence in this - the config fragment it suggests to fix it is nonsensical.

Am I doing something obviously wrong here?

  jaeger:
    image: jaegertracing/jaeger:latest
    environment:
      - SPAN_STORAGE_TYPE=opensearch
      - ES_SERVER_URLS=http://opensearch:9200
      - ES_TAGS_AS_FIELDS_ALL=true
      - ES_INDEX_PREFIX=jaeger
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - "16686:16686"
      - "4319:4317"
      - "4320:4318"
    depends_on:
      opensearch:
        condition: service_healthy
    command: --config /config.yaml
    volumes:
      - ./jaeger/config.yaml:/config.yaml

  opensearch:
    image: opensearchproject/opensearch:latest
    container_name: opensearch
    ports:
      - "9200:9200"
      - "9600:9600"
    environment:
      - discovery.type=single-node
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=ThePassword1234!
      - DISABLE_SECURITY_PLUGIN=true
    healthcheck:
      test: ["CMD", "curl", "-s", "-f", "http://localhost:9200/_cat/health"]
      interval: 10s
      start_period: 10s
      timeout: 5s
      retries: 3
    volumes:
      - opensearch-data:/usr/share/opensearch/data

service:
  extensions:
    [
      jaeger_storage,
      jaeger_query,
      remote_sampling,
      healthcheckv2,
      expvar,
      zpages,
    ]
  pipelines:
    traces:
      receivers: [otlp, jaeger, zipkin]
      processors: [batch]
      exporters: [jaeger_storage_exporter]
  telemetry:
    resource:
      service.name: jaeger
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: "${env:JAEGER_LISTEN_HOST:-localhost}"
                port: 8888
    logs:
      level: info
    # TODO Initialize telemetry tracer once OTEL released new feature.
    # https://github.com/open-telemetry/opentelemetry-collector/issues/10663

extensions:
  jaeger_query:
    storage:
      traces: some_storage
    base_path: /v2/connections

  jaeger_storage:
    backends:
      some_storage:
        elasticsearch:
          server_urls:
            - http://opensearch:9200
          tags_as_fields:
            all: true
            include: "vtba.ap_group_uuid,vtba.sta_mac,veea.serial"

  remote_sampling:
    # We can either use file or adaptive sampling strategy in remote_sampling
    file:
      path:
      default_sampling_probability: 1
      reload_interval: 1s
    # adaptive:
    #   sampling_store: some_store
    #   initial_sampling_probability: 0.1
    http:
      endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:5778"
    grpc:
      endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:5779"

  healthcheckv2:
    use_v2: true
    http:
      endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:13133"
    grpc:

  expvar:
    endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:27777"

  zpages:
    # for some reason the official extension listens on ephemeral port 55679
    # so we override it with a normal port
    endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:27778"

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:4317"
      http:
        endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:4318"

  jaeger:
    protocols:
      grpc:
        endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:14250"
      thrift_http:
        endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:14268"
      thrift_binary:
        endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:6832"
      thrift_compact:
        endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:6831"

  zipkin:
    endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:9411"

processors:
  batch:

exporters:
  jaeger_storage_exporter:
    trace_storage: some_storage

[dosubot[bot]]

Hi @tomkcook! I'm Dosu and I’m helping the jaeger team.

This is a subtle but known issue: Jaeger interprets tag values in queries as regular expressions, not as literal strings, and does not escape special characters like colons for you. So when you query for a MAC address (which contains colons), the value is treated as a regex pattern, which can cause the query to fail to match as expected if the value isn't escaped. This is why querying by just the UUID works, but adding the MAC address returns no results—even though the data is present.

To fix this, try escaping the colons in your MAC address value with a backslash (e.g., 7c\:3d\:09\:50\:18\:49) in your query. This tells the regex engine to treat the colons as literal characters rather than regex operators. You can test this by querying with only the vtba.sta_mac tag (with and without escaping) to confirm the behavior in isolation.

If you still get no results, it’s worth querying OpenSearch directly to verify how the tag values are stored and indexed, just to rule out any data ingestion issues. But based on the Jaeger code and how queries are constructed, escaping regex special characters in tag values is the correct approach here.
Reference: Jaeger uses RegexpQuery for tag values, which are not escaped automatically

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[yurishkuro]

I don't have high confidence in this answer.

[tomkcook]

@yurishkuro you would be right to be sceptical. After chasing down many, many rabbit holes I realised it was because I was including the operation name in the query and this didn't apply to the span where the tags all matched.