[.NET 6] [Source Compat Failure] NSK app doesn't publish with .net6.0 SDK

[jiangzeng01]

Application Name: NSK
OS: Windows 10 RS5
CPU: X64
.NET Build Number: 6.0.100-preview.1.21071.10
Project Location: https://github.com/andysal/NSK

Verify Scenarios:
1). Windows 10 RS5 X64 + .NET Core SDK build 6.0.100-preview.1.21071.10: Fail
2). Windows 10 RS5 X64 + .NET Core SDK build 5.0.102: Pass

Repro steps

1. Source Code check at https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1268776
2. dotnet publish Nsk.Web.Site.sln

Expected Result:
Build succeeded

Actual Result:

C:\Program Files\dotnet\sdk\6.0.100-preview.1.21071.10\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ConflictResolution.targets(108,5): error NETSDK1148: Found multiple publish output files with the same relative path: C:\Users\valtunc\Desktop\AppSources\NSK\netcore\src\Nsk.Web.Site.Areas.Admin\libman.json, C:\Users\valtunc\Desktop\AppSources\NSK\netcore\src\Nsk.Web.Site\libman.json. [C:\Users\valtunc\Desktop\AppSources\NSK\netcore\src\Nsk.Web.Site\Nsk.Web.Site.csproj]

@dotnet-actwx-bot @dotnet/compat

[javiercn]

@vijayrkn I believe this is your team?

Otherwise, can you route it appropriately?

[vijayrkn]

@javiercn - I am not aware of any specific globs that pertains to libman.json. Adding @jodavis who owns the library manager feature to see if he has come across anything in this space.

[jodavis]

Library Manager doesn't use globbing patterns. It has a build task that could run during a publish, but that only looks at libman.json at the root of a given project.

Adding @jimmylewis in case I'm missing anything.

[jimmylewis]

If I understand correctly, there are two projects containing libman.json files, and they both publish to the root, which is causing a conflict? I'm not sure how LibMan fits in here in any special way that an arbitrary foo.json wouldn't. What's the difference in the outputted files between 5.0 and 6.0? Was there only one file output before but no conflict detection? Or were there two separate files before and now there's only one?

(LibMan's publish task would add other files to the output groups when publishing the project, but we don't do anything to include the libman.json file in publishing.)

[jiangzeng01]

@jimmylewis @jodavis Thanks for investigation! Do you have next step for this issue?

[jimmylewis]

@jiangzeng01 LibMan isn't in play here. We don't publish the libman.json file, so any conflicts over it must come from somewhere higher up in the publish process (e.g. if the project is publishing it, but that's out of our control).

[jiangzeng01]

@jimmylewis Is there any change in .NET 6 causing this failure? As we can publish this app with .NET 5 SDK, but failed with .NET 6 SDK. Something should be changed in .NET 6?

[jimmylewis]

@jiangzeng01 that would be my guess but I don't know what's changed in 6.0.

@vijayrkn do you know who we can loop in here?

[vijayrkn]

@jiangzeng01 - Can you share a sample that fails on "dotnet publish" with the above issue? That will help speed up the investigation.

[jiangzeng01]

@jiangzeng01 - Can you share a sample that fails on "dotnet publish" with the above issue? That will help speed up the investigation.

Repro steps
Source Code check at https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1268776
dotnet publish Nsk.Web.Site.sln

You can check this app, it fails on publish if you use .NET 6 SDK.

[vijayrkn]

@jiangzeng01 Thanks for the repro steps.

Adding @sfoslund & @dsplaisted - Based on the binlog, this seems to be caused by this target _HandleFileConflictsForPublish added as a part of this PR- dotnet/sdk#14020

I haven't gone through the details but these seems to be flagged as dupes just based on the relative path though they are from 2 different projects.

@sfoslund - Can you please take a look?

[dsplaisted]

It is trying to copy both copies of libman.json to the publish folder for Nsk.Web.Site.csproj. We added an error for this for .NET 6. Previously, it would silently just choose one of the files to copy arbitrarily.

You can set the ErrorOnDuplicatePublishOutputFiles property to false to unblock testing. But we should figure out what the right thing to do here is. Is libman.json supposed to be published with the app at all? It may be that we need to exclude it from the default Content glob.

[jimmylewis]

libman.json is a tooling asset, not a runtime asset. I can't think of a reason why it would need to be deployed.

[jiangzeng01]

@vijayrkn @dsplaisted @jimmylewis
We just found another app failed with similar issue, but not about libman.json.
Could you also please take a look at this to see?

App name: SCF
App source:
SCF-master.zip

Repro steps:

1. Install VS 16.9 Preview 4 + .NET 6 Preview 1 daily SDK (dotnet-sdk-6.0.100-preview.1.21102.8)
2. Open the solution in VS and publish project Senparc.Web
   (This app does not work with CLI dotnet publish, need to use VS to build/publish)

Error:

Error        
Found multiple publish output files with the same relative path:
C:\Users\appcompat\Desktop\AppSources\SCF-master\src\Senparc.Xscf.ExtensionAreaTemplate\App_Data\DataBase\SenparcConfig.config, 
C:\Users\appcompat\Desktop\AppSources\SCF-master\src\Senparc.Web\App_Data\DataBase\SenparcConfig.config,
C:\Users\appcompat\Desktop\AppSources\SCF-master\src\Senparc.Xscf.ExtensionAreaTemplate\appsettings.json,
C:\Users\appcompat\Desktop\AppSources\SCF-master\src\Senparc.Web\appsettings.json.  

[vijayrkn]

Adding @pranavkm as well. It looks the customers mostly will hit this scenario when an ASP.NET Core app has a reference to a RCL and the contents from RCL's are copied over to the Web App output with the same name.

Agreed that we may have to remove libman from the content glob but that could be handed differently. we may have to solve the RCL reference problem first.

[pranavkm]

Isn't this pointing to an actual problem in the RCL project rather than the SDK? There are files in conflict that would end up in the same location and that's non-deterministic. Wouldn't the fix be to change the project to exclude these files from being published?

[vijayrkn]

@pranavkm - Is there anything in the RCL template that would cause a dupe?

[pranavkm]

I don't think so. The template is very empty. I think it's specific to this particular project.

[vijayrkn]

@pranavkm sounds good. Let's keep an eye out for feedback to see if we hear more issues in this front.

@jiangzeng01 - For the tests, can you fix the projects to not have duplicate outputs with the same name? Until then, you could use Daniel's suggestion to pass /p:ErrorOnDuplicatePublishOutputFiles=false to the dotnet publish command

[dsplaisted]

I think we may have to look at the default globs which pick up .config and .json files. They are in the Razor SDK here:

https://github.com/dotnet/sdk/blob/e8149a370fca24f151362297387ede475c241625/src/RazorSdk/Sdk/Sdk.Razor.StaticAssets.ProjectSystem.props#L32-L37

I think there are similar globs for the Web SDK.

[vijayrkn]

WebSDK also uses the same glob but the tricky part here is we need the glob to behave differently when it is used from the main project vs when it is used from the dependent project.

[pranavkm]

Plus there's the problem that users expect these files to be published based on the behavior today. We have trouble where we'd to back changes to these between releases.

[vijayrkn]

I still think this is a good change. Should RazorSDK opt out of this behavior (ErrorOnDuplicatePublishOutputFiles to false) until we have figured out a good way to handle this for all TFMs?

or Selectively enable this change only for projects targeting net6.0 & above so that customers won't have to go back and fix up their 2.1/3.1/5.0 apps.

[dsplaisted]

In the repro it was a project using the Web SDK referencing a project using the Razor SDK. So the Razor SDK opting out of the check wouldn't suppress the error, it would have to be the Web SDK.

If the impact of this is large then it would be OK for the Web and/or Razor SDKs to turn this off for some of the previews until we figure out how to handle it.

I don't know what the desired behavior is. Is it that these files should be published with a project, but should not flow to be published with projects that reference them?

[vijayrkn]

WebSDK is actually a wrapper sdk. Globs for WebSDK projects & Razor SDK projects comes from RazorSDK. So adding the check in RazorSDK should fix it for all WebProjects including Razor Class libraries.

I agree that turning it off just hides the actual problem but I am more worried about projects that targets old TFMs will suddenly start failing during publish. I will leave it to @pranavkm to make a call on whether we need to condition it based on the tfms or turn off the behavior for the first few previews.

We could also do nothing at this point and wait for customer feedback to see if this is really a big problem or not.

[ghost]

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[pranavkm]

Parking this in our sprint planning to follow up.

[javiercn]

I think the analysis here is correct. The issue is a combination between the move of the globs to the Razor SDK and the new check added in 6.0 for published assets. Based on that, there are three things we can do:

• Disable the check for duplicates in the SDK (same state as before it was added, maybe instead convert it to a warning)
• Update the Razor SDK to have a more "restricted" set of globs when used via a library
  • This should be doable by creating two sets of globs and having the Web SDK turn on the "least restrictive" (current) one before importing the Razor SDK props.
• Update the .NET Core SDK to offer override behavior instead, where if the file comes from the main project or a direct reference takes over a transitive reference.
  • This is somewhat similar to how the Razor pages override behavior works in RCLs.

As a side note, Static Web Assets has had a check for this type of issue to prevent exactly the issue where two files end up in the same publish output path. This affects content not handled directly with static web assets.

[mkArtakMSFT]

Thanks @javiercn.
After some further conversation around this I'm closing this issue as by-design.
We don't want to change this behavior as it is actually hiding a problem customers already have. And if they'll be willing to leave with it in their existing projects, they have the option to set the ErrorOnDuplicatePublishOutputFiles flag.
The other suggestions from @javiercn above will be breaking changes on their own, which eliminates the value of pursuing them.