cannot integrate with tempo+TLS

[PietroPasotti]

Bug Description

charm goes to error status on tracing-relation-changed as soon as you integrate tempo and traefik with self-signed-cer

To Reproduce

• juju deploy cos-lite --channel edge --trust
• juju deploy tempo-k8s --channel edge tempo
• jhack imatrix fill
• juju deploy self-signed-certificates --channel edge ssc
• jhack imatrix fill

Environment

┌──────────────┬───────────────────────────────────┐ 
│ jhack        │ 0.4.3.3.1                         │ 
│ python       │ 3.10.12 (/bin/python3)            │ 
│ juju-* snaps │  juju │ 3.5.3 - 28060 (3/stable)  │ 
│ microk8s     │ MicroK8s v1.31.0 revision 7175    │ 
│ lxd          │ 6.1                               │ 
│ multipass    │ 1.14.0                            │ 
│ multipassd   │ 1.14.0                            │ 
│ os           │ Ubuntu 22.04.5 LTS                │ 
│ kernel       │ Linux 6.8.0-40-generic x86_64     │ 
└──────────────┴───────────────────────────────────┘ 

Relevant log output

File "/var/lib/juju/agents/unit-grafana-0/charm/venv/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.64.140.43', port=4318): Max retries exceeded with url: /v1/traces (Caused by SSLError(SSLCertVerifica
tionError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '10.64.140.43'. (_ssl.c:1131)")))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/opentelemetry/sdk/trace/export/__init__.py", line 367, in _export_batch
    self.span_exporter.export(self.spans_list[:idx])  # type: ignore
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 191, in export
    return self._export_serialized_spans(serialized_data)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 161, in _export_serialized_spans
    resp = self._export(serialized_data)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 135, in _export
    return self._session.post(
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='10.64.140.43', port=4318): Max retries exceeded with url: /v1/traces (Caused by SSLError(SSLCertVerification
Error(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '10.64.140.43'. (_ssl.c:1131)")))
unit-grafana-0: 10:01:01 INFO juju.worker.uniter.operation ran "grafana-relation-changed" hook (via hook dispatching script: dispatch)

Additional context

No response

[sed-i]

Several things to look into:

1. Shouldn't throw a bare exception like that in general.
2. Why certificate verify failed: IP address mismatch, certificate is not valid for '10.64.140.43'? Having an IP is fine, should have gone into "san ip".
3. Is imatrix fill doing something unexpected?

Once better triaged, we should move the issue to the appropriate repo.