Conversation
There was a problem hiding this comment.
Pull request overview
This PR consolidates the various reference-assembly source files into a shared src/Microsoft.Data.SqlClient/ref/ set (organized roughly by namespace), updates the existing netcore/ref and netfx/ref projects to compile those shared files, and adjusts the solution layout to include the new common ref project—aiming to unblock packing the common project without removing ref projects.
Changes:
- Added a new common ref project (
src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.csproj) and split ref sources into multiple shared files undersrc/Microsoft.Data.SqlClient/ref/. - Updated
src/Microsoft.Data.SqlClient/netcore/refandsrc/Microsoft.Data.SqlClient/netfx/refcsproj files to compile the shared ref sources from../../ref/. - Updated the solution to add a
refsolution folder and include the new common ref project (and re-add the common src project).
Reviewed changes
Copilot reviewed 13 out of 16 changed files in this pull request and generated 21 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Microsoft.Data.SqlClient/ref/Microsoft.Data.cs | Adds ref stubs for Microsoft.Data surface (currently has namespace mismatch issues). |
| src/Microsoft.Data.SqlClient/ref/Microsoft.Data.Sql.cs | Adds ref stubs for Microsoft.Data.Sql surface (contains an auto-property stub issue). |
| src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.csproj | New common multi-targeted ref project for shared ref sources. |
| src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.Server.cs | Adds server namespace ref stubs (contains incorrect <include> pathing). |
| src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.Diagnostics.cs | Adds diagnostics ref stubs (contains namespace mismatch + malformed XML doc comments + auto-property stub issue). |
| src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.DataClassification.cs | Adds DataClassification ref stubs. |
| src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlTypes.cs | Adds SqlTypes ref stubs (SqlJson/SqlVector/SqlFileStream). |
| src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.Batch.cs | Removes old batch ref file (now expected to be covered by merged ref sources). |
| src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.Batch.NetCoreApp.cs | Removes netcoreapp-only batch ref file (now expected to be covered by merged ref sources). |
| src/Microsoft.Data.SqlClient/netfx/ref/Microsoft.Data.SqlClient.csproj | Points netfx ref project at shared ../../ref/* files (currently missing Diagnostics ref file). |
| src/Microsoft.Data.SqlClient/netcore/ref/Microsoft.Data.SqlClient.csproj | Points netcore ref project at shared ../../ref/* files (currently missing Diagnostics ref file). |
| src/Microsoft.Data.SqlClient/netcore/ref/Microsoft.Data.SqlClient.Manual.cs | Removes old manual ref file (expected to be covered by merged ref sources). |
| src/Microsoft.Data.SqlClient.sln | Adds a ref folder and the new common ref project; adjusts project entries (project type GUID inconsistency). |
src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.Diagnostics.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.Diagnostics.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.Data.SqlClient/netcore/ref/Microsoft.Data.SqlClient.csproj
Show resolved
Hide resolved
src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.Diagnostics.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.Diagnostics.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.Diagnostics.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.Diagnostics.cs
Outdated
Show resolved
Hide resolved
paulmedynski
left a comment
There was a problem hiding this comment.
I have questions about internal/private items being declared, and a lack of access modifiers.
I'm also wondering if it's worth adding a check to the pipelines to ensure that the APIs declared in the ref project are identical to those declared in the implementation. Or perhaps a build target to catch problems pre-commit. It's critical that the APIs are identical, and we've released packages with discrepancies before.
| #endif | ||
| public sealed class OperationAbortedException : System.SystemException | ||
| { | ||
| internal OperationAbortedException() { } |
There was a problem hiding this comment.
Why would the ref APIs declare any internal or private items? Same question across other files here as well.
There was a problem hiding this comment.
As per the magic matricies:
Yes, there is a valid reason in this case.
src/Microsoft.Data.SqlClient/ref/Microsoft.Data.cs:13 includes a non-public constructor so the compiler does not synthesize a public parameterless .ctor for OperationAbortedException. If that constructor were absent in the ref stub, the type would accidentally appear publicly constructible in the reference assembly, which would be an API contract bug.
So while ref files are for public API contract, they sometimes include non-public members to preserve correct type shape semantics. This pattern shows up in other ref types too (for example SqlException in src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.cs:1642).
Short version: those members are not there to expose internal API, they are there to prevent exposing the wrong public API.
There was a problem hiding this comment.
Sounds like a great code comment so we don't ask this question again in a few months!
src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.Diagnostics.cs
Show resolved
Hide resolved
mdaigle
left a comment
There was a problem hiding this comment.
I would also like to find a way to more easily compare before and after. Maybe we can leverage this? https://learn.microsoft.com/en-us/dotnet/standard/assembly/inspect-contents-using-metadataloadcontext
src/Microsoft.Data.SqlClient/ref/Microsoft.Data.SqlClient.Batch.NetCoreApp.cs
Show resolved
Hide resolved
|
I think there's already tooling to compare ref assemblies to implementation assemblies - the ApiCompat packages may be useful here. There are also other similar tools located here (although netstandard2.0 ref assemblies make GenAPI very difficult.) An earlier version of GenAPI is already in the SqlClient source tree. |
paulmedynski
left a comment
There was a problem hiding this comment.
This will be an interesting experiment to see how well the AI can be instructed to produce precisely the code that we want.
There was a problem hiding this comment.
I'm going to review whatever the AI/agents create from this file, rather than this file itself.
| <!-- ================================================================== --> | ||
| <!-- _SetApiCompatProperties --> | ||
| <!-- --> | ||
| <!-- All path properties are set inside a target so that they are --> |
There was a problem hiding this comment.
Extra padding at the end of these 3 lines.
There was a problem hiding this comment.
This target is specific to the MDS package, so maybe CompareMdsRefAssemblies.targets would be a better name.
| Build target that compares locally-built ref assemblies against those published | ||
| in a specified NuGet package version of Microsoft.Data.SqlClient. Uses | ||
| Microsoft.DotNet.ApiCompat.Tool in strict mode to detect any API surface | ||
| differences (additions or removals). |
There was a problem hiding this comment.
Does that include changes, like function arguments whose order has changed?
There was a problem hiding this comment.
Yes, it does. But interestingly, it doesn't care about "source breaking" changes: https://learn.microsoft.com/en-us/dotnet/standard/library-guidance/breaking-changes#source-breaking-change
So, for example, it won't error if you change an optional parameter to required.
| differences (additions or removals). | ||
| Usage: | ||
| dotnet msbuild build.proj /t:CompareRefAssemblies /p:BaselinePackageVersion=6.1.4 |
There was a problem hiding this comment.
dotnet build is the typical way to build. We're not using any special msbuild features here.
There was a problem hiding this comment.
I will make some of these changes, but don't want to invest too much time into polishing this. Once we have our pack process improved, we'll be able to use the EnablePackageValidation property in our csproj file to get this for free: https://learn.microsoft.com/en-us/dotnet/fundamentals/apicompat/package-validation/baseline-version-validator
| Text="Baseline package $(BaselinePackageVersion) already extracted at $(BaselineExtractDir). Skipping download." /> | ||
|
|
||
| <MakeDir | ||
| Condition="!Exists('$(BaselineExtractDir)ref\')" |
There was a problem hiding this comment.
Can we avoid repeating this condition by putting these actions in an <ItemGroup> ?
There was a problem hiding this comment.
I would need to separate them out into another target. ItemGroups can't contain tasks.
|
|
||
| <Message | ||
| Importance="high" | ||
| Text="Baseline ref assemblies available at $(BaselineExtractDir)ref\" /> |
There was a problem hiding this comment.
This is a duplicate message when the baseline package already exists.
| <!-- _RestoreApiCompatTool --> | ||
| <!-- ================================================================== --> | ||
| <Target Name="_RestoreApiCompatTool" DependsOnTargets="_SetApiCompatProperties"> | ||
| <Exec Command="dotnet tool restore" WorkingDirectory="$(RepoRoot)" /> |
There was a problem hiding this comment.
This restores all of the tools, which contradicts the name of the target.
| <Message Importance="high" Text="Building legacy netfx ref project (net462)..." /> | ||
| <MSBuild | ||
| Projects="$(LegacyNetFxRefProject)" | ||
| Targets="restore" |
There was a problem hiding this comment.
Why separate out the Restore target? You can specify multiple targets here:
Targets="Restore;Build"
It's fine to pass more properties to the Restore target than it needs.
| <!-- ================================================================== --> | ||
| <Target Name="_BuildNewRefProject" DependsOnTargets="_SetApiCompatProperties"> | ||
| <Message Importance="high" Text="Building new consolidated ref project (all TFMs)..." /> | ||
| <Exec Command="dotnet restore "$(NewRefProjectPath)"" /> |
There was a problem hiding this comment.
Why are we using <Exec> here instead of <MSBuild> ?
| dotnet msbuild build.proj /t:CompareRefAssemblies /p:BaselinePackageVersion=6.1.4 | ||
| ``` | ||
|
|
||
| - `BaselinePackageVersion` is **required** (no default). The user must specify which published package to compare against. | ||
| - Both the **legacy** ref projects (`netcore/ref/`, `netfx/ref/`) and the **new consolidated** ref project (`ref/`) are built and compared independently, so you can distinguish whether a difference comes from source consolidation vs. project restructuring. | ||
| - All comparisons run to completion (errors don't short-circuit), so you see all differences at once. | ||
|
|
||
| ## Implementation Steps | ||
|
|
||
| ### 1. Register `Microsoft.DotNet.ApiCompat.Tool` in `dotnet-tools.json` | ||
|
|
||
| Add an entry for version `9.0.200` alongside the existing `dotnet-coverage` entry. This enables `dotnet apicompat` after `dotnet tool restore`. | ||
|
|
||
| ### 2. Create `tools/targets/CompareRefAssemblies.targets` | ||
|
|
||
| A single new file containing all properties, items, and targets (steps 3–11 below). Follows the naming convention of existing files like `GenerateMdsPackage.targets`. | ||
|
|
||
| ### 3. Define properties | ||
|
|
||
| | Property | Value | Notes | | ||
| |----------|-------|-------| | ||
| | `BaselinePackageVersion` | *(none)* | Required; validated by guard target | | ||
| | `BaselinePackageDir` | `$(Artifacts)apicompat\` | Working directory for downloads | | ||
| | `BaselineNupkgPath` | `$(BaselinePackageDir)microsoft.data.sqlclient.$(BaselinePackageVersion).nupkg` | Downloaded nupkg path | | ||
| | `BaselinePackageUrl` | `https://api.nuget.org/v3-flatcontainer/microsoft.data.sqlclient/$(BaselinePackageVersion)/microsoft.data.sqlclient.$(BaselinePackageVersion).nupkg` | NuGet flat container URL | | ||
| | `BaselineExtractDir` | `$(BaselinePackageDir)extracted\$(BaselinePackageVersion)\` | Extraction destination | | ||
| | `NewRefProjectPath` | `$(ManagedSourceCode)ref\Microsoft.Data.SqlClient.csproj` | New consolidated ref project | | ||
| | `NewRefOutputDir` | `$(ManagedSourceCode)ref\bin\$(Configuration)\` | SDK default output for new ref project | | ||
| | `LegacyNetFxRefDir` | `$(Artifacts)Project\bin\Windows_NT\$(Configuration)\Microsoft.Data.SqlClient\ref\` | Legacy netfx ref output | | ||
| | `LegacyNetCoreRefDir` | `$(Artifacts)Project\bin\AnyOS\$(Configuration)\Microsoft.Data.SqlClient\ref\` | Legacy netcore ref output | | ||
|
|
||
| ### 4. `_ValidateBaselineVersion` target | ||
|
|
||
| Emits `<Error>` if `$(BaselinePackageVersion)` is empty, with a message instructing the user to pass `/p:BaselinePackageVersion=X.Y.Z`. | ||
|
|
||
| ### 5. `_DownloadBaselinePackage` target | ||
|
|
||
| - Depends on `_ValidateBaselineVersion` | ||
| - Uses `<DownloadFile>` to fetch the nupkg from nuget.org | ||
| - Uses `<Unzip>` to extract it into `$(BaselineExtractDir)` | ||
| - Conditioned to skip if `$(BaselineExtractDir)` already exists | ||
|
|
||
| ### 6. `_RestoreApiCompatTool` target | ||
|
|
||
| Runs `<Exec Command="dotnet tool restore" WorkingDirectory="$(RepoRoot)" />`. | ||
|
|
||
| ### 7. `_BuildLegacyRefNetFx` target | ||
|
|
||
| - Depends on `RestoreNetFx` | ||
| - Builds `$(NetFxSource)ref\Microsoft.Data.SqlClient.csproj` for `net462` | ||
| - Conditioned on Windows (`$(IsEnabledWindows)`) | ||
| - Output: `$(LegacyNetFxRefDir)net462\Microsoft.Data.SqlClient.dll` | ||
|
|
||
| ### 8. `_BuildLegacyRefNetCore` target | ||
|
|
||
| - Depends on `RestoreNetCore` | ||
| - Builds `$(NetCoreSource)ref\Microsoft.Data.SqlClient.csproj` with `/p:OSGroup=AnyOS` | ||
| - Output: `$(LegacyNetCoreRefDir){net8.0,net9.0,netstandard2.0}\Microsoft.Data.SqlClient.dll` | ||
|
|
||
| ### 9. `_BuildNewRefProject` target | ||
|
|
||
| - Runs `dotnet build` on `$(NewRefProjectPath)` | ||
| - Builds all 4 TFMs (`net462`, `net8.0`, `net9.0`, `netstandard2.0`) in one shot | ||
| - Output: `$(NewRefOutputDir){tfm}\Microsoft.Data.SqlClient.dll` | ||
|
|
||
| ### 10. `_RunRefApiCompat` target | ||
|
|
||
| - Depends on `_DownloadBaselinePackage;_RestoreApiCompatTool;_BuildLegacyRefNetFx;_BuildLegacyRefNetCore;_BuildNewRefProject` | ||
| - Iterates over 4 TFMs (`net462`, `net8.0`, `net9.0`, `netstandard2.0`) using item batching | ||
| - For each TFM, runs two `<Exec>` calls with `ContinueOnError="ErrorAndContinue"`: | ||
| - **Legacy vs baseline**: `dotnet apicompat -l {baseline-ref-dll} -r {legacy-ref-dll} --strict-mode` | ||
| - **New vs baseline**: `dotnet apicompat -l {baseline-ref-dll} -r {new-ref-dll} --strict-mode` | ||
| - Each `<Exec>` is conditioned on `Exists(...)` for both DLLs (gracefully skips missing TFMs) | ||
| - Preceded by `<Message Importance="high">` labelling each comparison | ||
| - Uses item metadata to map `net462` to `$(LegacyNetFxRefDir)` and others to `$(LegacyNetCoreRefDir)` | ||
|
|
||
| ### 11. `CompareRefAssemblies` target (public entry point) | ||
|
|
||
| - Declared with `DependsOnTargets="_RunRefApiCompat"` | ||
| - Emits a final `<Message>` summarizing completion | ||
|
|
||
| ### 12. Import into `build.proj` | ||
|
|
||
| Add one line after the existing `.targets` imports (after line 7): | ||
|
|
||
| ```xml | ||
| <Import Project="$(ToolsDir)targets\CompareRefAssemblies.targets" /> | ||
| ``` | ||
|
|
||
| ## Design Decisions | ||
|
|
||
| - **Single new file** at `tools/targets/CompareRefAssemblies.targets` — only one `<Import>` line added to `build.proj`. | ||
| - **Internal targets prefixed with `_`** to signal they're not intended to be called directly. | ||
| - **Strict mode** ensures API additions are also flagged — important for detecting accidental public surface changes during file reorganization. | ||
| - **`ContinueOnError="ErrorAndContinue"`** on each apicompat `Exec` so all 8 comparisons run and all differences are reported together. | ||
| - **`_BuildLegacyRefNetFx`** is conditioned on Windows (net462 can't build on Unix), matching the existing `BuildNetFx` pattern. | ||
| - **No default baseline version** — the user must always explicitly specify which published package to compare against. | ||
| - **Download caching** — re-runs skip the download if the extracted directory already exists. | ||
|
|
||
| ## Verification | ||
|
|
||
| ``` | ||
| dotnet msbuild build.proj /t:CompareRefAssemblies /p:BaselinePackageVersion=6.1.4 |
There was a problem hiding this comment.
The documentation plan states the target name should be CompareRefAssemblies (lines 12, 25, 88, 98, 114), but the actual implementation uses CompareMdsRefAssemblies (line 206 in the targets file and line 8 in build.proj). This creates confusion for users following the documentation. Update the plan document to use CompareMdsRefAssemblies consistently, matching the actual implementation.
| ### 2. Create `tools/targets/CompareRefAssemblies.targets` | ||
|
|
||
| A single new file containing all properties, items, and targets (steps 3–11 below). Follows the naming convention of existing files like `GenerateMdsPackage.targets`. | ||
|
|
||
| ### 3. Define properties | ||
|
|
||
| | Property | Value | Notes | | ||
| |----------|-------|-------| | ||
| | `BaselinePackageVersion` | *(none)* | Required; validated by guard target | | ||
| | `BaselinePackageDir` | `$(Artifacts)apicompat\` | Working directory for downloads | | ||
| | `BaselineNupkgPath` | `$(BaselinePackageDir)microsoft.data.sqlclient.$(BaselinePackageVersion).nupkg` | Downloaded nupkg path | | ||
| | `BaselinePackageUrl` | `https://api.nuget.org/v3-flatcontainer/microsoft.data.sqlclient/$(BaselinePackageVersion)/microsoft.data.sqlclient.$(BaselinePackageVersion).nupkg` | NuGet flat container URL | | ||
| | `BaselineExtractDir` | `$(BaselinePackageDir)extracted\$(BaselinePackageVersion)\` | Extraction destination | | ||
| | `NewRefProjectPath` | `$(ManagedSourceCode)ref\Microsoft.Data.SqlClient.csproj` | New consolidated ref project | | ||
| | `NewRefOutputDir` | `$(ManagedSourceCode)ref\bin\$(Configuration)\` | SDK default output for new ref project | | ||
| | `LegacyNetFxRefDir` | `$(Artifacts)Project\bin\Windows_NT\$(Configuration)\Microsoft.Data.SqlClient\ref\` | Legacy netfx ref output | | ||
| | `LegacyNetCoreRefDir` | `$(Artifacts)Project\bin\AnyOS\$(Configuration)\Microsoft.Data.SqlClient\ref\` | Legacy netcore ref output | | ||
|
|
||
| ### 4. `_ValidateBaselineVersion` target | ||
|
|
||
| Emits `<Error>` if `$(BaselinePackageVersion)` is empty, with a message instructing the user to pass `/p:BaselinePackageVersion=X.Y.Z`. | ||
|
|
||
| ### 5. `_DownloadBaselinePackage` target | ||
|
|
||
| - Depends on `_ValidateBaselineVersion` | ||
| - Uses `<DownloadFile>` to fetch the nupkg from nuget.org | ||
| - Uses `<Unzip>` to extract it into `$(BaselineExtractDir)` | ||
| - Conditioned to skip if `$(BaselineExtractDir)` already exists | ||
|
|
||
| ### 6. `_RestoreApiCompatTool` target | ||
|
|
||
| Runs `<Exec Command="dotnet tool restore" WorkingDirectory="$(RepoRoot)" />`. | ||
|
|
||
| ### 7. `_BuildLegacyRefNetFx` target | ||
|
|
||
| - Depends on `RestoreNetFx` | ||
| - Builds `$(NetFxSource)ref\Microsoft.Data.SqlClient.csproj` for `net462` | ||
| - Conditioned on Windows (`$(IsEnabledWindows)`) | ||
| - Output: `$(LegacyNetFxRefDir)net462\Microsoft.Data.SqlClient.dll` | ||
|
|
||
| ### 8. `_BuildLegacyRefNetCore` target | ||
|
|
||
| - Depends on `RestoreNetCore` | ||
| - Builds `$(NetCoreSource)ref\Microsoft.Data.SqlClient.csproj` with `/p:OSGroup=AnyOS` | ||
| - Output: `$(LegacyNetCoreRefDir){net8.0,net9.0,netstandard2.0}\Microsoft.Data.SqlClient.dll` | ||
|
|
||
| ### 9. `_BuildNewRefProject` target | ||
|
|
||
| - Runs `dotnet build` on `$(NewRefProjectPath)` | ||
| - Builds all 4 TFMs (`net462`, `net8.0`, `net9.0`, `netstandard2.0`) in one shot | ||
| - Output: `$(NewRefOutputDir){tfm}\Microsoft.Data.SqlClient.dll` | ||
|
|
||
| ### 10. `_RunRefApiCompat` target | ||
|
|
||
| - Depends on `_DownloadBaselinePackage;_RestoreApiCompatTool;_BuildLegacyRefNetFx;_BuildLegacyRefNetCore;_BuildNewRefProject` | ||
| - Iterates over 4 TFMs (`net462`, `net8.0`, `net9.0`, `netstandard2.0`) using item batching | ||
| - For each TFM, runs two `<Exec>` calls with `ContinueOnError="ErrorAndContinue"`: | ||
| - **Legacy vs baseline**: `dotnet apicompat -l {baseline-ref-dll} -r {legacy-ref-dll} --strict-mode` | ||
| - **New vs baseline**: `dotnet apicompat -l {baseline-ref-dll} -r {new-ref-dll} --strict-mode` | ||
| - Each `<Exec>` is conditioned on `Exists(...)` for both DLLs (gracefully skips missing TFMs) | ||
| - Preceded by `<Message Importance="high">` labelling each comparison | ||
| - Uses item metadata to map `net462` to `$(LegacyNetFxRefDir)` and others to `$(LegacyNetCoreRefDir)` | ||
|
|
||
| ### 11. `CompareRefAssemblies` target (public entry point) | ||
|
|
||
| - Declared with `DependsOnTargets="_RunRefApiCompat"` | ||
| - Emits a final `<Message>` summarizing completion | ||
|
|
||
| ### 12. Import into `build.proj` | ||
|
|
||
| Add one line after the existing `.targets` imports (after line 7): | ||
|
|
||
| ```xml | ||
| <Import Project="$(ToolsDir)targets\CompareRefAssemblies.targets" /> | ||
| ``` | ||
|
|
||
| ## Design Decisions | ||
|
|
||
| - **Single new file** at `tools/targets/CompareRefAssemblies.targets` — only one `<Import>` line added to `build.proj`. |
There was a problem hiding this comment.
The documentation plan states the filename should be CompareRefAssemblies.targets (line 25, 98, 103), but the actual filename is CompareMdsRefAssemblies.targets. Update the plan to match the actual implementation.
Description
As much as I want to remove the ref projects, they are tightly coupled to the netstandard binaries and the AnyOS "not supported" binaries. Detangling those will be a headache that we don't need to do right now. So instead, let's at least merge the ref projects into a common ref project just like we did for MDS projects.
The files were broken up differently from netcore, netfx, and netstandard, so I opted to break them down into one file per namespace. This made merging easier, and will likely make maintenance easier in the future.
No changes to the builds were made. Projects were updated to point to the merged files, and a common ref project was added. The solution file was modified slightly to make it clearer the difference between common ref project and common mds project.
This should unblock packing the common project.
Testing
Local builds of the ref projects still works, I presume they will continue to work in CI builds.