Proposal for running dotnet test with Microsoft.Testing.Platform

[mariam-abdulla]

Current State

Currently, when we run dotnet test in CLI, we use vstest as a test runner/driver to run tests in test projects.

Motivation

With dotnet test, users should be able to use Microsoft Testing Platform to run their tests for the sake of improving their experience. They should have the option to opt-in/out this new experience.

The reason for opting-in/out this experience is

1. Autodetecting if test projects are using vstest or the testing platform is complex, and we would end up with many false positives, making it hard to deliver a working and consistent experience.
2. Mixed mode (i.e. having projects using vstest and testing platform in the same solution) will never work as the two platforms have different command line options and different features, thus the mapping will not work as expected.

Note

We want to design in a way that is future proof and easy to keep backwards compatible.

Proposal

Make this option configurable in global.json. We chose global.json because it's located on the current directory level or its parent directories and is picked up by the dotnet sdk.

Here is a global.json sample:

Examples of Usage

1. Example 1

{
  "test" : {
    "runner": {
      "name": "MicrosoftTestingPlatform"
    }
  }
}

It contains the properties below:

• test: Contains configuration related to the test settings.
• runner: Specifies the test runner details.
• name: The name of the test runner to be used, in this case, "MicrosoftTestingPlatform".

2. Example 2

{
  "test" : {
    "runner": {
      "name": "VSTest"
    }
  }
}

This design is extendable. If later on we decided to support dotnet test as an external dotnet tool.
We can simply add the type/source property and other options as well.

Default

• If no test runner was provided in global.json, the default is set to vstest.

[Evangelink]

cc @baronfel @OsirisTerje @rprouse @bradwilson @MarcoRossignoli @smithkaylee

[marcpopMSFT]

Triage:
I think we agree that there is a need for a repo root (or user root) level concept of .NET configuration. examples However, we already have global.json today and tooling around it so going with something else might be problematic.

Other options would include adding configuration to the sln file but that has some drawbacks as well as explore other file options like repo.json or user.json (but with global.json already being in use, those are probably not good).

I think we probably need to first determine if we're ok expanding global.json for this class, outline some requirements for adding to this, and making sure there is a central library for exposing the data to other consumers.

@richlander for his opinion
@elinor-fung @agocke on whether ya'll have concerns with expanding global.json beyond its current usage and where a shared library for this would live (I assume you wouldn't want to expand hostfxr)
@dsplaisted as this seems like a partner sync topic

[bradwilson]

This is to replace the current TestingPlatformDotnetTestSupport MSBuild property, right?

What happens for test projects which support VSTest but not MTP? For example, consider the scenario where a team is migrating from xUnit.net v2 to v3, and the former doesn't support MTP. What if I want to run v2 tests as VSTest and v3 tests as MTP? Since this lives in global.json, would that imply I'd put a new global.json into every test project folder to control it individually? Is that recommended behavior for global.json?

Will there be any other way to flip the switch besides global.json? In the case of a test framework like xUnit.net, I may want to run tests via dotnet test with VSTest as well as then re-running them a second time via dotnet test with MTP. Would the MSBuild property still be an option here, or would I be forced to edit the global.json to do this?

[richlander]

I have believed for some time that we need an SDK config system, similar (at least in spirit) to what git has. I probably have some issues filed on it.

I'd like to see us keep global.json limited to SDK version scenarios. This will help us avoid problems that Marc is hinting at.

Brad also raises some good points/challenges.