chore: Update xUnit related package version to v3 #10474
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is Draft PR to migrate unit test projects to
xUnit v3
. (Prev draft PR is #10375)Currently test execution on VS is unstable
Known issues
(
https://github.com/xunit/visualstudio.xunit/issues/428
)TestTfmsInParallel: false
setting is not works as expected when running tests with VS Test Explorer(
https://github.com/xunit/visualstudio.xunit/issues/429
)(
https://github.com/xunit/xunit/issues/3091
)-> To suppress ANSI escape character output from
Spectre.Console
.It need to specify
NO_COLOR
environment variable temporary.--collect:"XPlat Code Coverage"
is not working when usingMicrosoft.Testing.Platform
.(
https://github.com/coverlet-coverage/coverlet/issues/1715
)What's changed to this PR
1. Switch xUnit.NET related NuGet packages to v3.
Verify.Xunit
->Verify.XunitV3
xunit
->xunit.v3
2. Add additional output log types (
*.log
/*.ctrf
)3. Change unittest's projects
OutputType from
Libraryto
Exe`.4. Add
Microsoft.Testing.Platform
related settingsMicrosoft.Testing.Platform is alternative unit test execution layers.
Currently It' used when running tests with following ways.
dotnet test
command.6. Add
xunit.runner.json
config file for test project.7. Pass
TestContext.Current.CancellationToken
that require Cancellation token.8. Modify
MetadataCommandTest.cs
And add steps to create dummyDirectory.Build.props
file.It's required to ignore project directories
Directory.Build.props
settings9. Add empty
Directory.Build.props
file to samples directory10. Modify CI workflow to use custom
seed
to reproduce issue on CIBy default. xUnit determine
seed
on build timing and seed value is different between assemblies.So to fix seed value. Use
github.run_number
as a seed value. (It's incremented from 1)If CI failed by parallel execution related issue.
It can re-run job with using same seed.
11. Add
SetBranchNameAttribute
that derived fromBeforeAfterTestAttribute
On current code base.
When running
SeedMarkdown
tests before other snapshot tests.It cause snapshot diffs because GitUtility.cs caching actual branch name.
To simplify environment variable set/restore.
Add custom
BeforeAfterTestAttribute
for this purpose.12. Add
[assembly:CaptureConsole]
attribute to test assemblies.13. Add
UseNullAnsiConsole
that derived fromBeforeAfterTestAttribute
This code might be removed. If TestExplorer console output issue is resolved.
14. Add ProcessHelper class to invoke external command (e.g. dotnet, docfx).
When using
Process.Start
to launch external command that stdout.Running tests is not completed when running tests on Test Explorer.
So I've added utility code to ignoring stdout/stdout.
15. Modify
dotnet test --filter Stage=Percy
command. to use-- --filter-trait "Stage=Percy"
When
TestingPlatformDotnetTestSupport
is enabled.dotnet test
command's--filter
argument is silently ignored. (Same behavior as that reported athttps://github.com/microsoft/testfx/issues/4401
)It need modify command to pass additional argument(
-- --filter-trait "Stage=Percy"
)16. Separate
Chromium Headless Shell
installation from test stepsChromium is installed on when running
docfx pdf
command first time.It take times and it affects test execution times.
So creating separate chromium install step that is executed before running
dotnet test
.https://github.com/dotnet/docfx/pull/10474/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR43-R48
17. Add
-nodereuse:false
and--no-dependencies
parameter todotnet build
command inside testsAfter migrated to xUnit v3.
SamplesTest::Extensions
dotnet build
command takes extra 15 minutes on Windows CI.It seems relating to
https://github.com/dotnet/sdk/issues/9452
so I've added-nodereuse:false
parameter to resolve issue.And adding
--no-dependencies
parameter to suppressDocfx.App
project build during test.https://github.com/dotnet/docfx/pull/10474/files#diff-7d53f25641c42f0268899c29446be9e322c11502bd26b0573e9a67d3309c6e0bR148-R155