feat(experimental): Add schema change support for BigQuery

[iambriccardo]

Schema Evolution Support for BigQuery Destinations

This PR introduces end-to-end schema evolution support for BigQuery destinations, enabling replication pipelines to detect, track, and apply schema changes from source PostgreSQL tables.

Design Overview

DDL Event Trigger

Schema changes are captured transactionally using a PostgreSQL DDL event trigger. When an ALTER TABLE executes, the trigger fires within the same transaction, emitting a logical replication message before the transaction commits. This guarantees schema change events are ordered consistently with data changes in the replication stream.

Schema Versioning

Schema versions are tracked using snapshot_id:

• Initial schema load creates a table with snapshot_id=0 (base schema)
• Each DDL event generates a new schema version with snapshot_id=start_lsn of the DDL logical message, providing total ordering and stable identifiers

On restart, the system loads the schema version with the largest snapshot_id <= confirmed_flush_lsn.

Replication Masks

A replication mask is a bitmask determining which columns of a TableSchema are actively replicated. This decouples column-level filtering from schema loading by:

• Loading the complete table schema independently
• Using Relation messages to determine which columns to replicate
• Combining both in ReplicatedTableSchema, a wrapper exposing only active columns

This allows columns to be added/removed from a publication without breaking the pipeline.

Destination Table Metadata

Introduces DestinationTableMetadata to store destination-side table metadata, replacing previous table mappings with a more extensible structure. The system tracks applied schemas in its own store rather than querying the destination catalog, storing:

• snapshot_id: last successfully applied schema
• replication_mask: column mask for the applied schema
• previous_snapshot_id: schema before current (used for recovery)

Transactional Safety

Schema change atomicity varies by destination, so we use a two-phase state approach:

• Applied → schema is consistent
• Applying → schema change in progress

When a schema change begins, we transition to Applying before modifying the destination. If failure occurs mid-change, this state signals corruption and enables recovery flows. Recovery uses previous_snapshot_id to identify the last known-good schema.

Note: BigQuery lacks atomic DDL for multiple statements, so failed schema changes cannot safely roll back and require manual intervention.

Schema Diffing

When a Relation event arrives, the destination receives a ReplicatedTableSchema. Diffing is performed at the destination layer, giving each destination full control over handling schema changes. The ReplicatedTableSchema::diff() method provides a simple changeset, but destinations can implement custom logic.

How It Works

1. DDL event trigger fires on ALTER TABLE
2. Trigger emits schema change in the replication stream within the same transaction
3. PostgreSQL sends a Relation message with updated column information
4. System emits a RelationEvent containing the new ReplicatedTableSchema
5. Destination detects changes via diffing and applies them according to its capabilities

Other Changes

• Consistent schema loading: New describe_table_schema PostgreSQL function returns schema data in a consistent structure for both initial copy and DDL events
• Composite primary keys: Schema query now loads ordinal positions of primary keys
• Compile-time invariants: ReplicatedTableSchema is supplied with each event, eliminating schema loading in destinations and enforcing invariants via the type system

Future Work

• Cleanup of old schema versions (planned for follow-up PR)
• The new schema is included with each event after DDL for better DX, though technically only the RelationEvent is required

[depthfirst-app]

🟠 Severity: HIGH

Supply Chain Risk: Unverified Git Dependencies

Two critical PostgreSQL driver dependencies (postgres-replication and tokio-postgres) were switched from the trusted Materialize fork to a personal GitHub fork. These handle authentication and replication stream parsing. Unlike crates.io packages, git dependencies lack security vetting, can be force-pushed or deleted, and create single points of failure. The same risk applies to line 77 (tokio-postgres).
Helpful? Add 👍 / 👎

💡 Fix Suggestion

Suggestion: Replace the personal GitHub fork (https://github.com/iambriccardo/rust-postgres) with a trusted source. Options: (1) If possible, use the official crates.io versions of postgres-replication and tokio-postgres; (2) If custom patches are required, fork to your organization's GitHub account with proper access controls and security review processes; (3) If changes are ready, publish to crates.io after security review; (4) Document the specific changes required from the fork and track upstream merge status. This affects both line 60 (postgres-replication) and line 77 (tokio-postgres). Before making changes, identify what patches or modifications exist in the personal fork that differ from upstream to ensure functionality isn't broken.