Nested objects in array lose information on toonification

[nomenclature95]

Summary

When toonifying nested JSON structures, certain properties (such as arrays of objects or strings) are lost in the output, even though token counts are reduced. This results in incomplete toonified data.

---

Steps to Reproduce

1. Use the following JSON input (inline below).
   The data includes a nested array of objects under offset and an array of strings under hierarchy.

   {
     "categorization": [
       {
         "id": "01.04.04.01.",
         "label": "Aspetti generali",
         "hierarchy": [
           "Prodotti",
           "Organizzazione altro e Sito Internet",
           "Aspetti generali",
           "Aspetti generali"
         ],
         "score": 900,
         "winner": true,
         "namespace": "$namespace",
         "frequency": 0,
         "offset": [
           { "start": 511, "end": 520 },
           { "start": 524, "end": 527 },
           { "start": 528, "end": 543 }
         ]
       }
     ]
   }

2. Run toonify on this JSON.

3. Inspect the toonified output.

---

Observed Behavior

• The resulting toonified JSON omits both the offset (array of objects) and hierarchy (array of strings) properties.
• Token count is indeed reduced, but this reduction comes from the loss of meaningful structure and data.
• See attached screenshot for reference:

---

Expected Behavior

Toonified output should preserve all properties (including nested arrays and objects), ensuring structural and semantic fidelity while still optimizing token usage.

---

Environment

• package version: toonify-1.4.0
• IDE: PyCharm (Notebook mode)
• Experiment Context: Quick local test to measure token impact

---

Additional Context

I understand that this type of JSON doesn't fit neatly into traditional tabular data structures. However, supporting more complex, nested formats would significantly improve toonify's robustness, especially for real-world datasets used with LLMs where hierarchical or relational structures are common.

---

Suggested Improvement

Consider extending toonify's serialization logic to:

• Preserve nested arrays and object properties.
• Optionally flatten or represent them symbolically (e.g., offset[start:end] shorthand) without dropping information.
• Provide a fallback or warning when certain structures can't be safely toonified.

[VinciGit00]

Hi @nomenclature95! 👋

Thank you for reporting this issue with such a detailed description and example. You're absolutely right - the nested arrays and objects were being lost during toonification.

What was the problem?

The encoder was using a compact tabular format for arrays of objects, but this format only preserved primitive fields (strings, numbers, booleans). When objects contained nested arrays like hierarchy or offset, these fields were silently dropped.

The Fix

I've modified the is_uniform_array_of_objects() function to detect when objects contain non-primitive fields (nested arrays or objects). When detected, the encoder now uses the list array format instead, which properly preserves all nested structures.

Your example data now correctly roundtrips with all fields preserved:

• ✅ hierarchy array of strings is preserved
• ✅ offset array of objects is preserved
• ✅ All primitive fields are preserved
• ✅ Full structural and semantic fidelity maintained

Testing

I've added comprehensive tests covering:

• Arrays with nested arrays (your exact use case)
• Arrays with nested objects
• Mixed primitive and nested fields
• Complex roundtrip scenarios
• Partial nested fields

All 107 existing tests pass without regression.

Pull Request

The fix is ready for review in PR #7: #7

Thank you again for the excellent bug report! 🎉

[github-actions]

🎉 This issue has been resolved in version 1.5.0 🎉

The release is available on:

• v1.5.0
• GitHub release

Your semantic-release bot 📦🚀