flatten json log messages inside log record body into individual fields

[nikhilsinhaparseable]

if a log event is json, with otel-collector, log records's body will contain the entire log event
in order to make it usable and queryable, we flatten the json into individual columns
original body field is also pertained for backward compatibility

if log event is unstructured (not a json), it remains as-is in body field

Summary by CodeRabbit

• New Features
  • Enhanced log processing to automatically flatten JSON objects found inside string-valued log fields into new top-level fields prefixed with the original field name.
  • The original field value is preserved; flattening only applies when the string parses as a JSON object and does not change other log behavior.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

Added JSON-string parsing and flattening to OTEL LogRecord body processing: when a body field is a string that parses to a JSON object, the inner object is flattened and its keys are added as top-level attributes prefixed by "{original_key}_{inner_key}", while retaining the original body field.

Changes

Cohort / File(s)	Summary
OTEL Logs JSON Flattening src/otel/logs.rs	Imported generic_flattening and updated flatten_log_record to iterate body JSON key-value pairs, always preserve the original key, detect string values that parse as JSON objects, apply generic_flattening, and inject flattened inner fields as top-level attributes with prefixed keys.

Sequence Diagram(s)

(Skipped — change is localized to a single function and does not introduce multi-component sequential flows.)

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• fix: stringify otel attributes of array types #1298 — Touches the same flatten_log_record function to collect/serialize other_attributes; may conflict or overlap with the new flattening behavior.
• refactor: specialized flatten happens at the same semantic location #1177 — Refactors OTEL flattening/ingestion paths and may affect where or how this new JSON-string flattening should be integrated.

Poem

🐰 I found a string that hid a nest,

I parsed it out and did my best,
Each inner hop now wears a name,
Prefixed bright — not quite the same,
A flattened field parade, hooray! 🥕

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Description check	❓ Inconclusive	The description explains the goal and rationale, but is missing the required template structure and checklist items for testing, comments, and documentation.	Use the repository's description template and complete the testing and documentation checklist items to clarify review requirements.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately and concisely describes the main change: flattening JSON log messages in the log record body into individual fields.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@src/otel/logs.rs`:
- Around line 101-112: The current loop that handles Value::String -> parsed ->
generic_flattening stores flattened objects into the same map (log_record_json)
so array elements overwrite each other; update the logic in the block
referencing generic_flattening, log_record_json, prefixed_key and
flattened_value to either skip flattening when generic_flattening returns
multiple Value::Object items (preserve the original body field) or, preferably,
enumerate array elements and inject keys with indexed prefixes (e.g.,
format!("{key}_{index}_{inner_key}") for each flattened_value) so fields from
all array elements are preserved; if the overwrite behavior is intentional, add
a clear comment near generic_flattening usage documenting that only the last
array element is kept.

🧹 Nitpick comments (1)

src/otel/logs.rs (1)

96-114: Potential key collision between original and flattened fields.

If body_json contains a key that matches a generated prefixed key, the flattened value will silently overwrite the original. For example, if body_json has both ("body_foo", ...) and ("body", "{\"foo\": ...}"), the flattened body_foo will replace the original.

Consider checking for key existence before inserting flattened fields to avoid silent overwrites:

💡 Suggested approach

                     for (inner_key, inner_value) in flattened_obj {
                         let prefixed_key = format!("{key}_{inner_key}");
+                        // Avoid overwriting existing keys from original body
+                        if !log_record_json.contains_key(&prefixed_key) {
                             log_record_json.insert(prefixed_key, inner_value);
+                        }
                     }