Feature | Add SqlBulkCopyOptions.CacheMetadata flag

[samsharma2700]

Description

Add SqlBulkCopyOptions.CacheMetadata to skip redundant metadata queries on repeated WriteToServer calls.

Issue

SqlBulkCopy executes a metadata discovery query against the destination table on every WriteToServer call, even when the same instance is used repeatedly for the same table. This query (which retrieves column types, collations, and encryption metadata) adds a network roundtrip that's unnecessary when the schema hasn't changed.

Related issue: #3627

Solution

Add a new opt-in flag SqlBulkCopyOptions.CacheMetadata that caches the metadata result after the first query and reuses it for subsequent WriteToServer calls to the same table.

Usage

	using var bulkCopy = new SqlBulkCopy(connection, SqlBulkCopyOptions.CacheMetadata, null);
  	bulkCopy.DestinationTableName = "MyTable";

  	bulkCopy.WriteToServer(batch1);  // executes metadata query, caches result
  	bulkCopy.WriteToServer(batch2);  // reuses cache, no query
  	bulkCopy.WriteToServer(batch3);  // reuses cache, no query

  	// If schema changes, manually invalidate:
  	bulkCopy.ClearCachedMetadata();

Behavior

• Cache is scoped to the SqlBulkCopy instance (not per-connection or global)
• Cache is automatically invalidated when DestinationTableName changes
• Cache is cleared on Dispose()/Close()
• New public method ClearCachedMetadata() allows manual invalidation

Why Opt-In

The metadata query provides schema information used by the TDS protocol. Caching assumes the destination table schema won't change between calls. Making this opt-in ensures
backwards compatibility and puts the responsibility for schema stability on the caller.

Changes

• SqlBulkCopyOptions.cs : Added CacheMetadata = 1 << 7
• SqlBulkCopy.cs : Cache logic in CreateAndExecuteInitialQueryAsync(), auto-invalidation in DestinationTableName setter, ClearCachedMetadata() method
• XML documentation for new API surface
• Manual/integration tests (16 tests)

Testing

Build/Tests are passing.

[Copilot]

Pull request overview

This PR adds a new SqlBulkCopyOptions.CacheMetadata flag to optimize repeated bulk copy operations to the same destination table by caching metadata after the first query, thus eliminating redundant metadata discovery queries on subsequent WriteToServer calls. This addresses issue #3627 where metadata queries were executed multiple times per operation, causing significant performance degradation in high-frequency bulk insert scenarios.

Changes:

• Adds new CacheMetadata enum value to SqlBulkCopyOptions (bit 7)
• Implements metadata caching logic in SqlBulkCopy with automatic invalidation on table name changes
• Adds public InvalidateMetadataCache() method for manual cache invalidation
• Includes comprehensive test coverage in both functional and manual test suites
• Provides XML documentation for the new API members

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlBulkCopyOptions.cs	Adds CacheMetadata enum value (1 << 7)
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlBulkCopy.cs	Implements caching logic with fields, cache check, cache storage, invalidation method, and dispose cleanup
src/Microsoft.Data.SqlClient/tests/FunctionalTests/SqlBulkCopyCacheMetadataTest.cs	Adds unit tests for flag values, combinations, and API methods
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/SqlBulkCopyTest/CacheMetadata.cs	Adds integration tests for caching behavior, invalidation, destination changes, DataTable support, and option combinations
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/SqlBulkCopyTest/SqlBulkCopyTest.cs	Wires up the new manual tests
src/Microsoft.Data.SqlClient/tests/ManualTests/Microsoft.Data.SqlClient.ManualTesting.Tests.csproj	Adds CacheMetadata.cs to project and introduces an unrelated project reference change
doc/snippets/Microsoft.Data.SqlClient/SqlBulkCopyOptions.xml	Documents CacheMetadata option with warnings about schema changes
doc/snippets/Microsoft.Data.SqlClient/SqlBulkCopy.xml	Documents InvalidateMetadataCache method

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlBulkCopy.cs:3262

• ResetWriteToServerGlobalVariables sets _sqlDataReaderRowSource = null; twice. This looks like an accidental duplicate assignment and may be masking a missing reset of a different field. Remove the duplicate (or reset the intended field) to keep the state reset logic clear.

            _rowSourceType = ValueSourceType.Unspecified;
            _sqlDataReaderRowSource = null;
            _sqlDataReaderRowSource = null;
        }

[paulmedynski]

I've reviewed the code and some of the tests. We can discuss my comments before I review the remainder of the tests.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.56%. Comparing base (a68e00f) to head (58cf942).
⚠️ Report is 19 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (a68e00f) and HEAD (58cf942). Click for more details.

HEAD has 6 uploads less than BASE

Flag	BASE (a68e00f)	HEAD (58cf942)
netfx	2	0
netcore	2	0
addons	2	0

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #3939       +/-   ##
===========================================
- Coverage   75.22%   64.56%   -10.66%     
===========================================
  Files         266      282       +16     
  Lines       42932    65985    +23053     
===========================================
+ Hits        32294    42605    +10311     
- Misses      10638    23380    +12742     

Flag	Coverage Δ
PR-SqlClient-Project	64.56% <100.00%> (?)
addons	?
netcore	?
netfx	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.