Introduction of back pressure and circuit breaker

[sdegroot]

Thema / Theme

Objecten API

Omschrijving / Description

Several ZGW-services communicate with each other. The most common (and potentially problematic) example: open-zaak / objecten-api calling open-notificaties.

I've seen several times now that when there is a misconfiguration or when there is a semi-large load, things start to break down. Response times start to increase, rest calls failing and database usage is maxing out.

Recently, I experienced a misconfiguration of open-zaak where the authentication for open-notifications was incorrect. Thus, resulting in a HTTP 403 for every notification being sent. Open-zaak kept trying to send notifications with about 10K calls per minute. In turn, this lead to the database use being 100% continuously (not sure why, probably for each call there is some sort of database check?). All systems started to degrade in performance and functionality.

I suggest that all ZGW components introduce some kind of back pressure and ideally even a circuit breaker to prevent the overload of components. Having a circuit breaker will also help with alerting on problems before the impact becomes so large that end-users start to complain.

Toegevoegde waarde / Added value

• better monitoring capabilities
• preventing cascading failures after a component fails

Aanvullende opmerkingen / Additional context

No response

[alextreme]

Discussed, and as we're talking about communicating performance between 4 processes (oz-uwsgi -> oz-worker -> on-uwsgi -> on-worker) and you want to know on the side of oz-uwsgi if on-worker is able to keep up, this requires quite a bit of coordination between the various services in order to deal with memory exhaustion at the various steps

We could communicate this via a HTTP header from ON to OZ (and Objects API). The alternative is that OZ throttles incoming calls if it detects that multiple notifications to ON are throwing errors.

Steven mentions that retrying on a HTTP 403 as mentioned by Sander doesn't make sense as a Forbidden won't work if you try it multiple times, in that case authentication needs to be fixed

We'll keep this in triage and come up with a few various ideas, maybe a PoC to see what would work in which situations. There isn't an easy fix for this in my eyes

[alextreme]

Discussed further with Team Bron and Joeri

A health endpoint could be added to Open Notificaties, Objects API and Open Zaak which would expose the general health (number of celery tasks in the queue, redis memory?) of the component. And environment variable can then be set to determine from which number the health endpoint becomes yellow instead of green. This status can then be picked up by the Utrecht infrastructure, but how you would act on this is then installation-specific and outside the components. Estimated at 3-4 days of work, @sdegroot let us know if this would work for Utrecht

An alternative direction would be to push this via OTel, but we currently don't have OTel in the components.

An alternative direction would be to communicate when creating objects or cases if notifications should be sent at all.

Communicating the health back to Open Zaak and Objects API is seen as a more complicated solution, especially if those components need to act on the status. So we don't recommend this.

[Pauline-fossgov]

@alextreme OTel is being implemented in OF open-formulieren/open-forms#4966

[alextreme]

Health endpoint for monitoring task queue size / redis which could be monitored via prometheus would also work for @JanBrek

[alextreme]

Besproken: wat zijn de criterea die nuttig zijn om op te monitoren?

• Oplopende queue qua notificaties/taken
• Snelheid van API requests
• Database query times die oplopen
• Database verbindingen die niet gelegd kunnen worden
• Notificaties die niet afgeleverd kunnen worden ondanks retryen van notificaties

Wens is wel af te wachten totdat OTel breder is geimplementeerd in Team Bron componenten