Allow configuring retryable status codes

[CmdrSharp]

Component(s)

exporter/otlphttp

Is your feature request related to a problem? Please describe.

The current logic (https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlphttpexporter/otlp.go#L225-L227) makes sense for an OTel Collector run as an actual collector, however many run OTel Collector in a "gateway" mode to perhaps do certain processing before exporting to the backend.

In scenarios like these, a response like 429 (which for Loki would mean tenant limits are exceeded) are often not a retryable event, and should not be enqueued. There is currently no way to configure this however - a 429 will always (as long as retry_on_failure is enabled) cause a retry.

Describe the solution you'd like

I suggest adding the ability to configure either:

• A list of status codes that should cause a retry
• A list of status codes that should NOT cause a retry

Describe alternatives you've considered

No response

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[dmathieu]

A 429 can be retried, given enough time to clear the rate limiting.

[CmdrSharp]

A 429 can be retried, given enough time to clear the rate limiting.

It can be and that's currently forced behavior (if retries are enabled). In our case, we'd much rather return a 429 and have the sender do the retries in that case. That is currently not supported behavior.

[vignesh-codes]

I guess adding A list of status codes that should NOT cause a retry makes more sense.

Proposed solution

Introduce an optional config

retry_on_failure:
  enabled: true
  non_retry_status: [429]

The exporter checks these list before enqueuing retries

Share thoughts.
I am happy to contribute here

[CmdrSharp]

I feel like this is the most user-friendly option of the two.

[axw]

I'm generally in favour of assuming everything is retryable and special-casing permanent errors - that proposed config LGTM.

Related: #14174

[farhan-pasha]

Hi @vignesh-codes I'm also interested in working on this if you're not actively working on it. It looks like a good first issue for me. Let me know! Thanks :)

[vignesh-codes]

@farhan-pasha i have'nt started implementing em. Feel free to get assigned and raise a pr