Proposal: Nagios-compatible monitoring Semantic Conventions

[martialblog]

Area(s)

area:new

Propose new conventions

Hello,

I'd like to propose the inclusion of Semantic Conventions for Nagios-compatible monitoring data.

Nagios-compatible monitoring applications are tools that use "monitoring plugin" that conform to the
"Plugin Output for Nagios" [1]. Example applications are Nagios, CheckMK, Icinga, Naemon, Shinken, Sensu.

These tools use "plugins" that are executed in intervals to determine the "state" of a host/service/component. There is vast ecosystem of these plugins [2] [3].

With this "state" - meaning the POSIX return code -the plugins can also return optional metrics (so called "performance data"). For example:

PING ok - Packet loss = 0%, RTA = 0.80 ms | percent_packet_loss=0, rta=0.80

DISK CRITICAL - free space: / 13969 MB (77% inode=99%); /boot 2 MB (0% inode=96%); 
| /=4151MB;14496;16308;0;18121 /boot=294MB;236;266;0;296

The performance data is provided in the following format [1] [4] [5]:

'label'=value[UOM];[warn];[crit];[min];[max]

The optional “warn” and “crit” denote thresholds for the value (meaning the threshold when they change from the OK return code to the Warning or Critical return code).

Conventions could include how these metrics and their thresholds should be provided, which attributes/resources they use or how a plugin execution is provided as an Event.

If this is a sensible proposal, I'd love to contribute.

References:

1. https://nagios-plugins.org/doc/guidelines.html#AEN200
2. https://www.monitoring-plugins.org/
3. https://github.com/Linuxfabrik/monitoring-plugins
4. https://docs.checkmk.com/latest/en/localchecks.html#metrics
5. https://icinga.com/docs/icinga-2/latest/doc/05-service-monitoring/#performance-data-metrics

Slack Threat: https://cloud-native.slack.com/archives/C07UWG2ATQB/p1764329658484799

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[thompson-tomo]

So my initial take on this would be that:

• An open telemetry collector reciever/processor is written to recieve the nagios metrics and transform them to otel equivalents. This transformation would
  • take the label and resolve the otel metric name, description, type & fixed attributes.
  • if min/max are specified and non 0, additional complementary metrics are created to capture that just like we do with memory etc
  • if warn/crit are specified and non 0, a complementary status metric is used to capture the state
• we capture the nagios check as a health_check event as discussed in Semantic Conventions for Health Check events #1106.

[martialblog]

An open telemetry collector reciever/processor is written to recieve the nagios metrics and transform them to otel equivalents.

Yes I think that makes sense. Tools like Telegraf already do something similar. https://github.com/influxdata/telegraf/tree/release-1.36/plugins/parsers/nagios

we capture the nagios check as a health_check event

Interesting, I'll have a look at that proposal. I did search for similar open issues but the terminology used in the "Nagios-compatible monitoring realm" is sometimes tricky, since every tool or vendor calls the same thing differently (checks, status checks, health checks, and so on).

There's not even a good term for "Nagios-compatible monitoring tool". I think, Nagios was just the first tool that used return codes and the output format and other tools are either forks or inspired by it. Unfortunately there is no actual standard for this either.

[thompson-tomo]

Agree hard part is coming up with naming when you don't have a standard to follow.

[martialblog]

Hi,

I've been thinking about this some more.

Since each of these results contains multiple metrics and thus it should be split up in an
expected manor. For example:

DISK CRITICAL - free space: / 13969 MB (77% inode=99%); /boot 2 MB (0% inode=96%); | /=4151MB;14496;16308;0;18121 /boot=294MB;236;266;0;296

The actual result of the check:

• result - 0,1,2,3 (for OK, WARNING, CRITICAL, UNKNOWN and so on)

However, maybe this could also be expressed as a state metric: https://opentelemetry.io/docs/specs/semconv/how-to-write-conventions/status-metrics/

I'm not sure about this yet.

For each "performance data" bundle, a separate metric for each element is created
each of which MUST contain a "label" attribute for the respective label.

• performance_data
• performance_data_warn
• performance_data_crit
• performance_data_min
• performance_data_max

While the warn,crit,min,max values could also just be attributes on a single metric,
which would keep the "bundle" together, I think it makes more sense to have them be their own metric.

Mainly because so they can be queried (and represented in a chart) but also because of their cardinality it's probably not a good idea to have them be attributes.

For the above example that would be (using Prometheus notation just because I'm familiar with it):

• performance_data{label="/", unit="MB"} 4151
• performance_data_warn{label="/"} 14496
• performance_data_crit{label="/"} 16308
• performance_data_min{label="/"} 0
• performance_data_max{label="/"} 18121
• performance_data{label="/boot", unit="MB"} 294
• performance_data_warn{label="/boot"} 236
• performance_data_crit{label="/boot"} 266
• performance_data_min{label="/boot"} 0
• performance_data_max{label="/boot"} 296

Together with the standard attributes for resources (e.g. service.name, host.name, etc.) this should make it possible to store and query the specific results for any component.

Inspiration was taken from this prototype: https://codeberg.org/oxzi/icinga-prometheus-remote-writer

I think, the "health_check" or "state_check" proposal captures some of this,
but these "performance data" bundles are very specific to the "Nagios-compatible monitoring realm" and might need special treatment.

But I'm open to corrections and ideas.

[thompson-tomo]

So if I disect the below example

DISK CRITICAL - free space: / 13969 MB (77% inode=99%); /boot 2 MB (0% inode=96%); | /=4151MB;14496;16308;0;18121 /boot=294MB;236;266;0;296

I would extract the following

• DISK CRITICAL - free space: / 13969 MB (77% inode=99%); /boot 2 MB (0% inode=96%); check details
• /=4151MB;14496;16308;0;18121 performance data # 1
• /boot=294MB;236;266;0;296 performance data # 2

The check details are used in the health/state check metric as I assume there is a common pattern.

We use the data from a performance data as following:

• 4151: data point value for system.filesystem. utilization with system.filesystem.state set to free
• 14496: potentially not used
• 16308: potentially not used
• 0: not used in this case
• 18121: data point value for system.filesystem.limit

All of these data points would have the label as an attribute.

I am still thinking as to the best way to capture the threshold but this could be attributes on the limit metric as done in https://opentelemetry.io/docs/specs/semconv/hardware/battery/#metric-hwbatterychargelimit but open to a nicer solution.

[martialblog]

Hi James, thanks for the feedback. You're dissection of the output is correct.

In an ideal world I agree that it would be best if each "performance data" would use a proper name for the metric name (e.g. system.filesystem. utilization as you described). However, I don't think that is feasible unfortunately.

There's just a vast library of these "monitoring plugins" out there, written in every programming language conceivable. Some of these plugins have been in use for decades.

That means, that the tools/vendors using the plugins would have to insert the correct metric names, but they don't what plugins the users has applied. The "Nagios-compatible plugin interface" is the only agreement between the tools and their users. Also each tool/vendor would have to agree on which metric name to use for a given plugin.

This is why I currently think the best way to capture this ecosystem is to have a single (or set of) "performance data" metric that all vendors can agree on.

[thompson-tomo]

I agree on not wanting to change existing plugins however that is where the collector plugin would come in as it would transform the incoming data including changing the name and attributes to fit otel conventions. Hence avoids any need for changes in plugins.

You would still have a generic health/state check plugin for high level as well as the event.

[dgoetz]

A challenge for the collector plugin trying to translate would be that the ecosystem is very wild and you can find monitoring plugins for very specific components including multiple modes with different performance data which might also vary. This is increased by the option to create your own plugins and I have seen this in most environments I have been involved as a consultant.
Also the quality is varying so you will find some very loose interpretations of the standard.

Just to give an example: The standard collection of plugins be it monitoring or nagios plugins does not contain a plugin to monitor memory usage on Linux which is a very basic metric you will want to monitor. So there are many attempts to fill this gap. One very old and common one is check_mem.pl which allows to query free or used memory, has an option to consider caches as free and all this changes the output including performance data.
Because of this different vendors developed other plugins or included a mode in a plugin which does all the system monitoring. Just to mention two NETWAYS has check_system_basics with mode memory which is Linux only and Linux Fabrik has memory_usage which covers Linux and Windows.

So having an ever growing mapping for translation which needs to be kept up to date and in sync in your environment while allowing your own expansion for self-written plugins, sounds at least for me not manageable. So a more simple generic approach sound more manageable, but of course I have to agree also like less quality.

[dgoetz]

And while thinking about I found another challenge as there is also an extension of the format that is not uncommon which is called multi format or check_multi format.

It originates from check_multi and is for example also used in Icinga for Windows.

This format is used to combine multiple checks in one execution and then one output, so it needs also to combine the performance data.

When looking at the original format consisting of "service_description::plugin_name::performance_data" and how it is used by Icinga for Windows in the linked example with "database_name::plugin_name::performance_data" I am not even sure how wild this one can get.

(Sorry, for bringing up all of these challenges. But please do not shoot the messenger!)