Quarantining or identifying rows for failed checks

[marrov]

This is a question that I have seen repeatedly from DEs working with cuallee: can we use checks in the cuallee to enable a quarantining feature?

With cuallee you get the number of rows that fail from your checks but not the rows themselves, AFAIK. The use case here is instead of just having that as a statistic collected in a table, users would like to be able to use the defined checks to split the incoming df into one that passes the tests and one that does not. This way these bad records can be investigated and re-processed into the clean table or dropped entirely. Any thoughts on this? Or maybe this idea is an anti-pattern altogether? Would love to hear your thoughts about this.

[canimus]

Hi @marrov,

Quarantine records are definitely an interesting topic in data engineering. While PySpark's compute engine has a placeholder for such functionality, it was never fully implemented. In fact, I recall that in the early branches of this repository, a commit by @vestalisvirginis introduced a sample method that returned a subset of bad records when a check failed.

The concept is straightforward, and I agree that it would be a valuable addition to cuallee's functionality. However, based on my experience implementing quarantine records—particularly in use cases like an Asset Manager receiving data from well-known vendors like FactSet or Bloomberg—I’ve found that quarantine mechanisms may not always be worth the added complexity. Here's why:

Dataset Management Overhead
Quarantine requires explicitly nominating and managing a separate quarantine file within your dataset. This adds complexity to your pipelines, as you'll need to ensure that only quarantined records are processed, diverging from the idempotent principle.

Increased Complexity in Automation
While quarantining can be automated, critical decisions about quarantined records often depend on human intervention. This introduces a "human-in-the-loop" dependency, which can conflict with modern goals of automation and agent-driven workflows.

Schema Maintenance Challenges
You'll need separate schemas for clean and quarantined records. Managing these dual schemas can be cumbersome, especially if schema changes occur frequently, as updates must be handled separately for clean and malformed data.

Philosophical Conflict
Accepting malformed records for later resolution can inadvertently normalize poor data practices. It’s often more efficient—both statistically and heuristically—to address data quality issues at the source instead of handling them downstream.

That said, adding a quarantine feature could enhance the value of validation checks. The challenge lies in exposing the relevant predicates from the underlying compute engine (e.g., PySpark) through the user-facing interface, such as cuallee's Check.

Let me think through some potential approaches for implementing this. In the meantime, I'd love to hear your thoughts and ideas as well. I hope this perspective is helpful!