Copy per-test result files to results directory#7452
Conversation
When using MTP, files added via TestContext.AddResultFile were not copied to the results directory. The TRX file referenced them with absolute paths, making the results directory non-self-contained and causing warnings in Azure DevOps builds. Add FileArtifactCopyDataConsumer, a new IDataConsumer in the core platform that subscribes to TestNodeUpdateMessage. When it sees FileArtifactProperty, it copies the file to the results directory. This benefits all downstream consumers. Update TRX report engine to write just the filename in ResultFile path elements, since the file is now in the results directory. Also: replace unbounded while(true) with bounded for loop when resolving duplicate file names in TRX artifact copy. Fixes #6782 d
| return; | ||
| } | ||
|
|
||
| if (!Directory.Exists(resultsDirectory)) |
There was a problem hiding this comment.
If we go with this solution, I'll use file system wrapper instead to ease testing
| /// Copies per-test file artifacts to the test results directory so that the | ||
| /// results directory is self-contained. | ||
| /// </summary> | ||
| internal sealed class FileArtifactCopyDataConsumer : IDataConsumer |
There was a problem hiding this comment.
Maybe this should be enabled/disabled via CLI or option?
There was a problem hiding this comment.
Pull request overview
This PR moves the responsibility for copying per-test result files to the results directory from TRX-specific code to the core testing platform. This makes the results directory self-contained and benefits all downstream consumers, not just TRX reports.
Changes:
- Adds FileArtifactCopyDataConsumer in the core platform to automatically copy FileArtifactProperty files to the results directory
- Updates TRX report engine to use relative paths (just filename) instead of absolute paths in ResultFile elements
- Replaces unbounded while(true) loops with bounded for loops (max 10 attempts) when resolving duplicate filenames
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Platform/Microsoft.Testing.Platform/TestHost/FileArtifactCopyDataConsumer.cs | New IDataConsumer that subscribes to TestNodeUpdateMessage and copies FileArtifactProperty files to results directory |
| src/Platform/Microsoft.Testing.Platform/Hosts/TestHostBuilder.cs | Registers the new FileArtifactCopyDataConsumer in the test framework builder |
| src/Platform/Microsoft.Testing.Extensions.TrxReport/TrxReportEngine.cs | Updates to write relative paths in ResultFile elements and replaces unbounded loop with bounded loop for duplicate filename resolution |
| test/UnitTests/Microsoft.Testing.Platform.UnitTests/TestHost/FileArtifactCopyDataConsumerTests.cs | Unit tests for the new FileArtifactCopyDataConsumer functionality |
| test/UnitTests/Microsoft.Testing.Extensions.UnitTests/TrxTests.cs | Updates test expectations for relative paths in TRX ResultFile elements |
| src/Platform/Microsoft.Testing.Platform/Resources/*.xlf | Localization resource files for the new consumer's display name and description |
Comments suppressed due to low confidence (1)
test/UnitTests/Microsoft.Testing.Platform.UnitTests/TestHost/FileArtifactCopyDataConsumerTests.cs:122
- There is no test coverage for the scenario where more than 10 files with the same base name already exist in the results directory. When this happens, the bounded loop will exit after 10 attempts, and File.Copy will be called with overwrite: false on a destination that already exists, which will throw an IOException. Consider adding a test case to verify this behavior and ensure it's handled appropriately.
[TestMethod]
public async Task ConsumeAsync_WithDuplicateFileName_AppendsCounter()
{
// Pre-create a file in the results directory with the same name.
File.WriteAllText(Path.Combine(_resultsDirectory, "Duplicate.txt"), "original");
string sourceFile = CreateSourceFile("Duplicate.txt", "new content");
TestNodeUpdateMessage message = CreateMessage(new PropertyBag(
new PassedTestNodeStateProperty(),
new FileArtifactProperty(new FileInfo(sourceFile), "Duplicate.txt")));
await _consumer.ConsumeAsync(new DummyProducer(), message, CancellationToken.None);
string expectedDestination = Path.Combine(_resultsDirectory, "Duplicate_1.txt");
Assert.IsTrue(File.Exists(expectedDestination));
Assert.AreEqual("new content", File.ReadAllText(expectedDestination));
}
| { | ||
| destination = Path.Combine(resultsDirectory, $"{Path.GetFileNameWithoutExtension(file.Name)}_{nameCounter}{Path.GetExtension(file.Name)}"); | ||
| } | ||
|
|
There was a problem hiding this comment.
After the bounded loop completes (when nameCounter exceeds 10), the code calls File.Copy even if a file still exists at the destination path. This will throw an IOException when overwrite is false and the destination file exists.
The code should check if a unique filename was found before attempting the copy, or handle the potential exception. Consider either throwing a more informative exception when all 10 attempts are exhausted, or checking File.Exists(destination) after the loop and handling that case explicitly.
| if (File.Exists(destination)) | |
| { | |
| throw new IOException($"Failed to copy file '{file.FullName}' to results directory '{resultsDirectory}' because a unique destination filename could not be determined after 10 attempts."); | |
| } |
| break; | ||
| destination = Path.Combine(artifactDirectory, $"{Path.GetFileNameWithoutExtension(fileName)}_{nameCounter}{Path.GetExtension(fileName)}"); | ||
| } | ||
|
|
There was a problem hiding this comment.
After the bounded loop completes (when nameCounter exceeds 10), the code calls CopyFileAsync even if a file still exists at the destination path. While CopyFileAsync creates a new file with FileMode.Create (which will overwrite), it would be better to verify that a unique filename was found or handle the case explicitly for clarity and consistency with error handling.
| if (File.Exists(destination)) | |
| { | |
| throw new IOException($"Could not generate a unique file name for artifact '{fileName}' in directory '{artifactDirectory}'."); | |
| } |
| // Copyright (c) Microsoft Corporation. All rights reserved. | ||
| // Licensed under the MIT license. See LICENSE file in the project root for full license information. | ||
|
|
||
| using Microsoft.Testing.Platform.Configurations; | ||
| using Microsoft.Testing.Platform.Extensions; | ||
| using Microsoft.Testing.Platform.Extensions.Messages; | ||
| using Microsoft.Testing.Platform.TestHost; | ||
|
|
||
| using Moq; | ||
|
|
||
| namespace Microsoft.Testing.Platform.UnitTests; | ||
|
|
||
| [TestClass] | ||
| public sealed class FileArtifactCopyDataConsumerTests : IDisposable | ||
| { | ||
| private readonly string _resultsDirectory; | ||
| private readonly string _sourceDirectory; | ||
| private readonly FileArtifactCopyDataConsumer _consumer; | ||
|
|
||
| public FileArtifactCopyDataConsumerTests() | ||
| { | ||
| _resultsDirectory = Path.Combine(Path.GetTempPath(), $"TestResults_{Guid.NewGuid():N}"); | ||
| _sourceDirectory = Path.Combine(Path.GetTempPath(), $"TestSource_{Guid.NewGuid():N}"); | ||
|
|
||
| Directory.CreateDirectory(_resultsDirectory); | ||
| Directory.CreateDirectory(_sourceDirectory); | ||
|
|
||
| Mock<IConfiguration> configMock = new(); | ||
| configMock.Setup(c => c[PlatformConfigurationConstants.PlatformResultDirectory]).Returns(_resultsDirectory); | ||
|
|
||
| _consumer = new FileArtifactCopyDataConsumer(configMock.Object); | ||
| } | ||
|
|
||
| public void Dispose() | ||
| { | ||
| if (Directory.Exists(_resultsDirectory)) | ||
| { | ||
| Directory.Delete(_resultsDirectory, recursive: true); | ||
| } | ||
|
|
||
| if (Directory.Exists(_sourceDirectory)) | ||
| { | ||
| Directory.Delete(_sourceDirectory, recursive: true); | ||
| } | ||
| } | ||
|
|
||
| [TestMethod] | ||
| public async Task IsEnabledAsync_ReturnsTrue() | ||
| { | ||
| bool isEnabled = await _consumer.IsEnabledAsync(); | ||
|
|
||
| Assert.IsTrue(isEnabled); | ||
| } | ||
|
|
||
| [TestMethod] | ||
| public async Task ConsumeAsync_WithNoFileArtifacts_DoesNotCopyAnything() | ||
| { | ||
| TestNodeUpdateMessage message = CreateMessage(new PropertyBag(new PassedTestNodeStateProperty())); | ||
|
|
||
| await _consumer.ConsumeAsync(new DummyProducer(), message, CancellationToken.None); | ||
|
|
||
| Assert.AreEqual(0, Directory.GetFiles(_resultsDirectory).Length); | ||
| } | ||
|
|
||
| [TestMethod] | ||
| public async Task ConsumeAsync_WithFileArtifact_CopiesFileToResultsDirectory() | ||
| { | ||
| string sourceFile = CreateSourceFile("TestOutput.txt", "test content"); | ||
| TestNodeUpdateMessage message = CreateMessage(new PropertyBag( | ||
| new PassedTestNodeStateProperty(), | ||
| new FileArtifactProperty(new FileInfo(sourceFile), "TestOutput.txt"))); | ||
|
|
||
| await _consumer.ConsumeAsync(new DummyProducer(), message, CancellationToken.None); | ||
|
|
||
| string expectedDestination = Path.Combine(_resultsDirectory, "TestOutput.txt"); | ||
| Assert.IsTrue(File.Exists(expectedDestination)); | ||
| Assert.AreEqual("test content", File.ReadAllText(expectedDestination)); | ||
| } | ||
|
|
||
| [TestMethod] | ||
| public async Task ConsumeAsync_WithFileAlreadyInResultsDirectory_DoesNotCopyAgain() | ||
| { | ||
| string fileInResults = Path.Combine(_resultsDirectory, "AlreadyHere.txt"); | ||
| File.WriteAllText(fileInResults, "already here"); | ||
|
|
||
| TestNodeUpdateMessage message = CreateMessage(new PropertyBag( | ||
| new PassedTestNodeStateProperty(), | ||
| new FileArtifactProperty(new FileInfo(fileInResults), "AlreadyHere.txt"))); | ||
|
|
||
| await _consumer.ConsumeAsync(new DummyProducer(), message, CancellationToken.None); | ||
|
|
||
| Assert.AreEqual(1, Directory.GetFiles(_resultsDirectory).Length); | ||
| Assert.AreEqual("already here", File.ReadAllText(fileInResults)); | ||
| } | ||
|
|
||
| [TestMethod] | ||
| public async Task ConsumeAsync_WithNonTestNodeUpdateMessage_DoesNothing() | ||
| { | ||
| Mock<IData> dataMock = new(); | ||
|
|
||
| await _consumer.ConsumeAsync(new DummyProducer(), dataMock.Object, CancellationToken.None); | ||
|
|
||
| Assert.AreEqual(0, Directory.GetFiles(_resultsDirectory).Length); | ||
| } | ||
|
|
||
| [TestMethod] | ||
| public async Task ConsumeAsync_WithDuplicateFileName_AppendsCounter() | ||
| { | ||
| // Pre-create a file in the results directory with the same name. | ||
| File.WriteAllText(Path.Combine(_resultsDirectory, "Duplicate.txt"), "original"); | ||
|
|
||
| string sourceFile = CreateSourceFile("Duplicate.txt", "new content"); | ||
| TestNodeUpdateMessage message = CreateMessage(new PropertyBag( | ||
| new PassedTestNodeStateProperty(), | ||
| new FileArtifactProperty(new FileInfo(sourceFile), "Duplicate.txt"))); | ||
|
|
||
| await _consumer.ConsumeAsync(new DummyProducer(), message, CancellationToken.None); | ||
|
|
||
| string expectedDestination = Path.Combine(_resultsDirectory, "Duplicate_1.txt"); | ||
| Assert.IsTrue(File.Exists(expectedDestination)); | ||
| Assert.AreEqual("new content", File.ReadAllText(expectedDestination)); | ||
| } | ||
|
|
||
| private string CreateSourceFile(string name, string content) | ||
| { | ||
| string path = Path.Combine(_sourceDirectory, name); | ||
| File.WriteAllText(path, content); | ||
| return path; | ||
| } | ||
|
|
||
| private static TestNodeUpdateMessage CreateMessage(PropertyBag properties) | ||
| => new( | ||
| default, | ||
| new TestNode | ||
| { | ||
| Uid = new TestNodeUid("test-id"), | ||
| DisplayName = "TestMethod", | ||
| Properties = properties, | ||
| }); | ||
|
|
||
| private sealed class DummyProducer : IDataProducer | ||
| { | ||
| public Type[] DataTypesProduced => [typeof(TestNodeUpdateMessage)]; | ||
|
|
||
| public string Uid => nameof(DummyProducer); | ||
|
|
||
| public string Version => "1.0.0"; | ||
|
|
||
| public string DisplayName => string.Empty; | ||
|
|
||
| public string Description => string.Empty; | ||
|
|
||
| public Task<bool> IsEnabledAsync() => Task.FromResult(true); | ||
| } | ||
| } |
There was a problem hiding this comment.
There is no test coverage for the scenario where the source file does not exist or when File.Copy throws an exception (e.g., IOException due to file locks, UnauthorizedAccessException, etc.). Consider adding test cases to verify that appropriate exceptions are thrown or handled when file operations fail, or document the expected behavior when such failures occur.
This issue also appears on line 106 of the same file.
| { | ||
| // If the file is already under the results directory, skip. | ||
| string fullResultsDirectory = Path.GetFullPath(resultsDirectory); | ||
| if (file.FullName.StartsWith(fullResultsDirectory, StringComparison.OrdinalIgnoreCase)) |
There was a problem hiding this comment.
The path comparison using StartsWith with StringComparison.OrdinalIgnoreCase may not correctly handle directory separators on all platforms. For example, if resultsDirectory is "C:\results" and file.FullName is "C:\results_backup\file.txt", the StartsWith check will incorrectly consider the file to be under the results directory.
The codebase has a FileUtilities.PathComparison utility (src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/FileUtilities.cs) that properly handles case-sensitivity based on the file system. Additionally, similar code in AnsiTerminal.cs checks for directory separators after the path prefix to ensure it's a true subdirectory.
Consider using FileUtilities.PathComparison for the string comparison and adding a check for directory separators after the prefix, similar to the pattern in AnsiTerminal.cs lines 209-213.
Variant of #7449, where the copy is done at the core platform.
When using MTP, files added via TestContext.AddResultFile were not copied to the results directory. The TRX file referenced them with absolute paths, making the results directory non-self-contained and causing warnings in Azure DevOps builds.
Add FileArtifactCopyDataConsumer, a new IDataConsumer in the core platform that subscribes to TestNodeUpdateMessage. When it sees FileArtifactProperty, it copies the file to the results directory. This benefits all downstream consumers.
Update TRX report engine to write just the filename in ResultFile path elements, since the file is now in the results directory.
Also: replace unbounded while(true) with bounded for loop when resolving duplicate file names in TRX artifact copy.
Fixes #6782