Create our own .NET SDKs (OSOE-1208)

[Piedone]

With custom .NET SDKs, we can further DRY projects by adding common code that would normally go into csproj files, to cut down repetition between projects. This would also reduce the amount of Renovate updates in submodules. Check out this blog post for details and inspiration, but, for example, we could have the following SDKs:

• Orchard Core module, extending Microsoft.NET.Sdk.Razor, adding the .NET framework version, common OC package references, <FrameworkReference Include="Microsoft.AspNetCore.App" />.
• Orchard Core theme, similar to the module one.
• Test project with xUnit and Microsoft.NET.Test.Sdk package references. Perhaps separately for unit and UI test projects.
• Auto-detect LibMan and import our MSBuild Targets project in the Module SDK.
• Add Renovate custom manager

Questions:

1. ✅ How would this work with both submodules and NuGet packages?
   • No local SDKs, but we can <Import> the props and targets files.
   • <Sdk Name="name" Version="version" /> can't use Condition, but we can use <Import Project="Sdk.props" Sdk="name" Version="version" /> which is still much better than a Targets NuGet package.
2. ✅ How can Renovate handle updates, within the SDKs (like, is it simply updating package references there too)?
   • The SDK just contains props and targets files, which should already work with Renovate.
3. ✅ How can Renovate handle updates of the references to SDKs in consumer projects?
   • We have to figure out if <Import Project="..." Sdk="..." Version="..." /> is detected by Renovate or not. I'd guess probably not and we will have to add a custom resolver.
4. ✅ We'd version the SDK references, but then how can we avoid e.g. having to update that in all test projects when the xUnit version is updated in the test SDK?
   • This is why we should only include high impact packages such as HL and UITT in the SDK. When we have to update those, it cascades down anyway.
5. ✅ Can we just use the latest version when in submodules? (Kind of how submodules have version-less references to other projects coming from submodules, but use concrete versions in NuGet package references.)
   • There is no versioning when using submodules (we don't want to, but it's also not even possible).

Jira issue

[wAsnk]

@Piedone
I'm trying to understand the task here, so I started with what I believe is one of the easier SDKs: the UI test SDK.

For the UI test SDK, do you imagine something like this?

Nothing in Sdk.targets.

Sdk.props:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup Condition="'$(TargetFramework)' == ''">
    <TargetFramework>net8.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="18.0.1" />
    <PackageReference Include="xunit.v3" Version="3.2.2" />
    <PackageReference Include="xunit.runner.visualstudio" Version="3.1.5">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>

  <ItemGroup Condition="'$(NuGetBuild)' != 'true' and '$(LombiqTestRoot)' != ''">
    <ProjectReference Include="$(LombiqTestRoot)\test\Lombiq.UITestingToolbox\Lombiq.Tests.UI\Lombiq.Tests.UI.csproj"
                      Condition="Exists('$(LombiqTestRoot)\test\Lombiq.UITestingToolbox\Lombiq.Tests.UI\Lombiq.Tests.UI.csproj')" />
  </ItemGroup>

  <ItemGroup Condition="'$(NuGetBuild)' == 'true'">
    <PackageReference Include="Lombiq.Tests.UI" Version="14.2.0" />
  </ItemGroup>
</Project>

And you could use it, e.g. in Lombiq.Tests.UI.Samples.csproj as:

<Project Sdk="Lombiq.OrchardCore.UiTest.Sdk/1.0.5">

  <PropertyGroup Condition="'$(NuGetBuild)' != 'true'">
    <LombiqTestRoot>$(MSBuildProjectDirectory)\..\..\..</LombiqTestRoot>
  </PropertyGroup>
....

And you can remove the above added PackageReferences and ProjectReferences from Lombiq.Tests.UI.Samples.csproj.

Is this what you had in mind?

[Piedone]

Something like that, yes. But a usual test project shouldn't need to include LombiqTestRoot (Lombiq.Tests.UI.Samples is an exception due to being bundled with UITT), and note the questions at the bottom of the issue for the SDK version reference.

[sarahelsaig]

How would this work with both submodules and NuGet packages?

According to this official MS documentation, MSBuild SDKs are picked up by resolver plugins in this order:

1. from NuGet packages
2. from .NET SDK installations
3. from the MSBuild installation itself

So there is no default way to resolve SDKs from a local path. According to this open issue in the dotnet/msbuild repo, there is currently no easy way to add custom resolves either. You can place them in the $(MSBuildToolsDirectory32)/SdkResolvers path, but on Windows that requires admin privileges so it's not a realistic pathway.

An alternative is generating a local NuGet feed, but that's its own can of worms that would explode the build complexity. In CI, I think it's only possible by doing two separate builds. First to build the local SDK and register it as a local NuGet, then to do the actual build. I really don't like this.

The alternative according to the above mentioned Microsoft docs is to just directly import the props and targets files. I think this negates the point of the exercise. Perhaps it's better to bite the bullet and accept that this will be a NuGet-only thing? A slight hitch in development efficiency, but if we are conservative about what <PackageReference>es are used in the SDK then it should be fine. (I'd only reference OC, HL, BT, UITT & LibMan where applicable.)

What do you think?

[Piedone]

One of the points of using SDKs is centralizing package references and versions for project types. This also includes updates to those package references. Meaning that e.g. if Microsoft.NET.Test.Sdk or xUnit.v3 is updated, Renovate will only update it in one place as opposed to every single test project across repositories, as it happens today.

If we need to release SDKs to NuGet, then that would make updates more complex than today: instead of updating e.g. the xUnit reference in 19 Lombiq projects + client projects, as it is today, it would update one project, what will need to be published to NuGet, and then its reference updated in 19 Lombiq projects + client projects. We'd need to same amount of updates plus a NuGet release, what also means it can't be all at once, and harder to automate (due to having to wait for the NuGet package).

How about paths in global.json? https://learn.microsoft.com/en-us/dotnet/core/tools/global-json#paths

Also, you previously looked into using common props/targets files. Where did you arrive with that, in the end?

[sarahelsaig]

How about paths in global.json?

That's for ".NET SDKs", not "MSBuild SDKs". Those are completely different, where the ".NET SDK" is what you download and install from here while the "MSBuild SDK" is the collection of prop and target files. This won't be super confusing. 🙄

Also, you previously looked into using common props/targets files. Where did you arrive with that, in the end?

Oh right, I forgot about that! Actually I found that <PackageReferences> didn't work at all when you release the prop file in a NuGet package. It's because by the time the prop and target files are loaded from a package, the dependency graph is already build so it's too late to add more packages. It's part of why this SDK thing was exciting since this is resolved as the project file is loaded, before anything else.

[Piedone]

Eh.

<PackageReferences> should work even then, since this is what we do in Lombiq.Analyzers too.

[sarahelsaig]

In Lombiq.Analyzers we are doing a bit of hax to generate a bespoke nuspec file in the publish-nuget action, which prevents dotnet pack from automatically generating one with missing dependencies. In a way this is hard coding package references and bypass whatever is actually in the props file. We could do the same here as well, if we must.

[Piedone]

We are leet haxors, after all!

This isn't nice, but it would work. I'd prefer SDKs but not for the price of making everything else more complex. Or do you have a better idea for the use case under #1190 (comment)?

[sarahelsaig]

I vaguely recall <import>s actually worked reliably for submodules and all problems were on the NuGet side. The best case scenario would be if we could use an SDK for the NuGet, import for local, and then no need for ugly hack. We'd have a project file like this:

<Project>
  <Import Condition="'$(NuGetBuild)' != 'true'" Project=".../LombiqSdk.props" />
  <Sdk Condition="'$(NuGetBuild)' == 'true'" Name="LombiqSdk" Version="1.0.0" />
  ...
</Project>

I'm going to have limited availability tomorrow, but I will ponder this in the background and do some experiments on Friday.

[Piedone]

Something that would be good, yes. Thanks!

[sarahelsaig]

Updates:

You can't actually do <Sdk Condition="'$(NuGetBuild)' == 'true'" ... />, because <Sdk> is one of the few elements that don't support Condition. If you try, the condition is ignored and you get a "Multiple Frameworks are detected" exception. Instead, it's possible to do this

<Import Condition="'$(NuGetBuild)' == 'true'"
        Project="Sdk.props"
        Sdk="Lombiq.MSBuild.OrchardCore.Module.Sdk"
        Version="1.0.1-alpha.0.osoe-1208" />

Note the Sdk attribute, which makes the path relative to that NuGet package. This is still handled by the MSBuild SDK resolver before all other NuGet packages; so far it doesn't look like we have to do any nuspec black magic.

LibMan-using project to include LibMan targets from our MSBuild Targets project.

That's not worth it in my opinion. Since you can't have multiple SDKs in a single project, we'd have to create a LibMan version of each SDK and that's very cumbersome. This kind of addon feature should be kept as a Targets project, but I will check a bit later if we can automatically import it from the module SDK when a libman.json file exists to streamline the most common use-case.