dotnet test on .NET Framework test suite requires runtime identifier?

[thomhurst]

Is this anything to do with testing platform?

I've just added .NET Standard support to TUnit, and if I do a normal dotnet test it doesn't run unless I pass in a runtime identifier too.

Original issue is here: thomhurst/TUnit#1465

dotnet test:

PS C:\Users\Tom\Downloads\repro1465\repro1465> dotnet test
Restore complete (1.7s)
  repro1465 failed with 1 error(s) (0.4s)
    C:\Program Files\dotnet\sdk\9.0.101\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(266,5): error NETSDK1047: Assets file 'C:\Users\Tom\Downloads\repro1465\repro1465\obj\project.assets.json' doesn't have a target for 'net472/win-x86'. Ensure that restore has run and that you have included 'net472' in the TargetFrameworks for your project. You may also need to include 'win-x86' in your project's RuntimeIdentifiers.

dotnet test -r win-x86:

PS C:\Users\Tom\Downloads\repro1465\repro1465> dotnet test -r win-x86
Restore complete (0.8s)
  repro1465 succeeded (7.2s) → bin\Debug\net472\win-x86\repro1465.exe
  repro1465 test succeeded (1.4s)

Test summary: total: 1, failed: 0, succeeded: 1, skipped: 0, duration: 1.3s

[Evangelink]

Looking at that issue now

[Evangelink]

@thomhurst could you please confirm the following behaviors:

1. From @AArnott sample, doing dotnet build fails with the same issue
2. Building from VS and then running dotnet test --no-build results in green run of tests

It would seems to indicate this is a msbuild/SDK issue on building (which we call when doing dotnet test).

@baronfel @JanKrivanek @YuliiaKovalova is it something you are aware of?

[baronfel]

Got a binlog of what happens when you call MSBuild @Evangelink?

[Evangelink]

Here you go: msbuild.binlog.txt

[thomhurst]

You're right. It happens on the build. dotnet build causes it too:

PS C:\Users\Tom\Downloads\repro1465\repro1465> dotnet build
Restore complete (0.9s)
  repro1465 failed with 1 error(s) (0.4s)
    C:\Program Files\dotnet\sdk\9.0.101\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(266,5): error NETSDK1047: Assets file 'C:\Users\Tom\Downloads\repro1465\repro1465\obj\project.assets.json' doesn't have a target for 'net472/win-x86'. Ensure that restore has run and that you have included 'net472' in the TargetFrameworks for your project. You may also need to include 'win-x86' in your project's RuntimeIdentifiers.

Build failed with 1 error(s) in 6.4s

I managed to build in VS and then run on the CLI with --no-build like you suggested and that works fine:

PS C:\Users\Tom\Downloads\repro1465\repro1465> dotnet test --no-build
  repro1465 test succeeded (2.4s)

Test summary: total: 1, failed: 0, succeeded: 1, skipped: 0, duration: 1.7s
Build succeeded in 3.0s

[thomhurst]

I'm not well versed in the SDK and how msbuild does stuff, but is the problem that the ResolvePackageAssets task is using a win-x86 runtime identifier parameter?

[JanKrivanek]

Unfortunately it's not easily trackable from the binlog where that inital value comes from

@YuliiaKovalova - this is another example where dotnet/msbuild#11106 will help a ton!

[JanKrivanek]

But it seems to go from here:

https://github.com/dotnet/sdk/blob/094238ce2d4f69fbfdf0fdf862ef714b3dd027ea/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.RuntimeIdentifierInference.targets#L63

While not being used during Restore.

@marcpopMSFT - could you suggest who should look at this?

[marcpopMSFT]

This is specific to TUNIT correct? From the binlog, it appears that HasRuntimeOutput is true for the build but not for the restore which is why a RID is getting set for the build but not project assets json gets created during the restore. This is happening because OutputType is ending up as an exe in the build but not during the restore.

The restore downloads the tunit package which ends up setting the OutputType to exe after the restore phase in TUNIT.Engine.props.

@rainersigwald @dsplaisted any suggestions how tunit can work around this? Do we need a property they can disable the implicit RID inference for netfx apps? They could set HasRuntimeOutput to false but I don't know what other side effects that'll have.

[Evangelink]

@thomhurst I confirm that setting <OutputType>Exe</OutputType> directly in the project fixes the issue. So it's neither a dotnet test nor a MTP issue which is good.

[thomhurst]

@thomhurst I confirm that setting <OutputType>Exe</OutputType> directly in the project fixes the issue. So it's neither a dotnet test nor a MTP issue which is good.

Ah that's good!
I've got a buildTransitive props file in my NuGet package that should be setting that though: https://nuget.info/packages/TUnit.Engine/0.6.52

But it looks like it's in an empty nested folder...

Not sure why as it should be pulling through the TargetFramework

<PackagePath>buildTransitive/$(TargetFramework)/</PackagePath>

[thomhurst]

Anyway - This is a me problem. Sorry for the red herring. Thanks for the help everyone!

[baronfel]

@thomhurst just confirming - that structure would 100% interfere with the auto-loading behaviors of NuGet. Check for double-slashes in the PackagePath for the items you're bundling?

[rainersigwald]

The restore downloads the tunit package which ends up setting the OutputType to exe after the restore phase in TUNIT.Engine.props.

This is the core of it--I think the other test packages require that the project itself set OutputType. If it's set in the import, it will be set for "real" post-restore builds, but not the restore itself.

[dsplaisted]

Yes, props and targets files in packages should not be setting any properties which would modify the input to Restore, which includes the OutputType.

[Evangelink]

The restore downloads the tunit package which ends up setting the OutputType to exe after the restore phase in TUNIT.Engine.props.

This is the core of it--I think the other test packages require that the project itself set OutputType. If it's set in the import, it will be set for "real" post-restore builds, but not the restore itself.

One reason why custom MSBuild SDKs are good. When using it, we can actually set the output type in the props and everything works fine while for a normal package we have this error.

[thomhurst]

Yes, props and targets files in packages should not be setting any properties which would modify the input to Restore, which includes the OutputType.

Weird. This has been in TUnit for ages and doesn't seem to affect .NET (Core) builds?

Only for .NET Framework has it not worked.

I'd really like to be able to abstract away all the properties from users to simplify things for them.

[AArnott]

I'd really like to be able to abstract away all the properties from users to simplify things for them.

@thomhurst for what it's worth IMO some properties are better left in the project. I wouldn't typically expect a PackageReference to change something so fundamental for the project as its OutputType property.

[Evangelink]

@AArnott I think @bradwilson is doing the same for xUnit. I do tend to think it's more expected for test because with the new platform each test app has to be an executable so it feels boilerplate. But again we saw similar problem in MSTest and that's why we went with output type set by user for normal nuget and automatically set when using MSTest.Sdk custom MSBuild SDK.

Fyi I'll be having some discussions next week for the possibility to have some generic test SDK linked to the new platform.

[rainersigwald]

doing the same for xUnit. I do tend to think it's more expected for test because with the new platform each test app has to be an executable so it feels boilerplate.

I'd recommend that the NuGet package validate the exe requirement but not try to impose it (for this reason that it does other stuff).

[thomhurst]

doing the same for xUnit. I do tend to think it's more expected for test because with the new platform each test app has to be an executable so it feels boilerplate.

I'd recommend that the NuGet package validate the exe requirement but not try to impose it (for this reason that it does other stuff).

How would I do this?

[rainersigwald]

Something like (untested)

<Target Name="ValidateExeProject"
        BeforeTargets="BeforeBuild"
        Condition="'$(BuildingProject)' == 'true'">
    <Error Condition="'$(OutputType)' != 'Exe'"
           Text="TUnit test projects must have OutputType Exe as of version XXX" />
</Target>

[bradwilson]

I'm happy to do this for xUnit.net as well, but I have no idea what '$(BuildingProject)' == 'true' is. I can't find it documented anywhere, and when set as a condition my target never triggers.

My temptation is to use '$(TargetFramework)' != '' instead.

[thomhurst]

Echoing brad. Also does this go in the props/targets file too?

[bradwilson]

Also, Microsoft.NET.Test.Sdk unilaterally sets <OutputType>Exe</OutputType> for netcoreapp3.1, which actually doesn't work properly (presumably for the reasons mentioned here), but also still ends up bypassing the notification message.

This puts things in a weird position: you'll get a notification for .NET Framework projects, but not .NET projects, and also the .NET projects aren't actually building executables despite trying to hard-set <OutputType>Exe</OutputType>. Kind of the worst case situation.

[bradwilson]

https://github.com/microsoft/vstest/blob/bcef12d909709f13027afe1557c724f11bc8df05/src/package/Microsoft.NET.Test.Sdk/netcoreapp/Microsoft.NET.Test.Sdk.targets#L20

[thomhurst]

Im struggling with props files on net framework ATM.

Net core I put files in build transitive and it works great I'm trying just build for net framework and not getting the results I expect.

[bradwilson]

Edit: I'm able to fix the lack of AppHost by injecting <UseAppHost>true</UseAppHost>. I'll also print out a message if this is set wrong (for example, overridden in the project file).

[bradwilson]

v3 1.0.2-pre.3 has the updates above.