Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Update xUnit related package version to v3 #10474

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

filzrev
Copy link
Contributor

@filzrev filzrev commented Dec 19, 2024

This is Draft PR to migrate unit test projects to xUnit v3. (Prev draft PR is #10375)
Currently test execution on VS is unstable

Known issues

  1. VS Test Explorer hangs when running tests that output stdout.
    (https://github.com/xunit/visualstudio.xunit/issues/428)
  2. TestTfmsInParallel: false setting is not works as expected when running tests with VS Test Explorer
    (https://github.com/xunit/visualstudio.xunit/issues/429)
  3. Console capture issues when reporter enabled and ANSI escape character is outputted.
    (https://github.com/xunit/xunit/issues/3091)
    -> To suppress ANSI escape character output from Spectre.Console .
    It need to specify NO_COLOR environment variable temporary.
  4. Code coverage with --collect:"XPlat Code Coverage" is not working when using Microsoft.Testing.Platform.
    (https://github.com/coverlet-coverage/coverlet/issues/1715)

What's changed to this PR

1. Switch xUnit.NET related NuGet packages to v3.

  • Verify.Xunit -> Verify.XunitV3
  • xunit -> xunit.v3

2. Add additional output log types (*.log/*.ctrf)

3. Change unittest's projects OutputType from LibrarytoExe`.

4. Add Microsoft.Testing.Platform related settings
Microsoft.Testing.Platform is alternative unit test execution layers.
Currently It' used when running tests with following ways.

6. Add xunit.runner.json config file for test project.

7. Pass TestContext.Current.CancellationToken that require Cancellation token.

8. Modify MetadataCommandTest.cs And add steps to create dummy Directory.Build.props file.
It's required to ignore project directories Directory.Build.props settings

9. Add empty Directory.Build.props file to samples directory

10. Modify CI workflow to use custom seed to reproduce issue on CI
By default. xUnit determine seed on build timing and seed value is different between assemblies.
So to fix seed value. Use github.run_number as a seed value. (It's incremented from 1)

If CI failed by parallel execution related issue.
It can re-run job with using same seed.

11. Add SetBranchNameAttribute that derived from BeforeAfterTestAttribute
On current code base.
When running SeedMarkdown tests before other snapshot tests.
It cause snapshot diffs because GitUtility.cs caching actual branch name.

To simplify environment variable set/restore.
Add custom BeforeAfterTestAttribute for this purpose.

12. Add [assembly:CaptureConsole] attribute to test assemblies.

13. Add UseNullAnsiConsole that derived from BeforeAfterTestAttribute
This code might be removed. If TestExplorer console output issue is resolved.

14. Add ProcessHelper class to invoke external command (e.g. dotnet, docfx).
When using Process.Start to launch external command that stdout.
Running tests is not completed when running tests on Test Explorer.
So I've added utility code to ignoring stdout/stdout.

15. Modify dotnet test --filter Stage=Percy command. to use -- --filter-trait "Stage=Percy"
When TestingPlatformDotnetTestSupport is enabled. dotnet test command's --filter argument is silently ignored. (Same behavior as that reported at https://github.com/microsoft/testfx/issues/4401)
It need modify command to pass additional argument(-- --filter-trait "Stage=Percy")

16. Separate Chromium Headless Shell installation from test steps
Chromium is installed on when running docfx pdf command first time.
It take times and it affects test execution times.
So creating separate chromium install step that is executed before running dotnet test.

https://github.com/dotnet/docfx/pull/10474/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR43-R48

17. Add -nodereuse:false and --no-dependencies parameter to dotnet build command inside tests
After migrated to xUnit v3.
SamplesTest::Extensions dotnet build command takes extra 15 minutes on Windows CI.
It seems relating to https://github.com/dotnet/sdk/issues/9452 so I've added -nodereuse:false parameter to resolve issue.
And adding --no-dependencies parameter to suppress Docfx.App project build during test.

https://github.com/dotnet/docfx/pull/10474/files#diff-7d53f25641c42f0268899c29446be9e322c11502bd26b0573e9a67d3309c6e0bR148-R155

@filzrev filzrev force-pushed the chore-update-xunit-to-v3 branch 9 times, most recently from 115da34 to 29128a9 Compare December 21, 2024 21:56
@filzrev filzrev force-pushed the chore-update-xunit-to-v3 branch from 29128a9 to 36241e0 Compare December 27, 2024 00:10
@filzrev filzrev force-pushed the chore-update-xunit-to-v3 branch from 36241e0 to ff545e2 Compare December 27, 2024 00:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant