Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running a .NET 5 project on macOS M1 with .NET 6 leads to confusing error message #59168

Closed
vcsjones opened this issue Sep 15, 2021 · 62 comments
Closed
Labels
arch-arm64 area-Host os-mac-os-x macOS aka OSX untriaged New issue has not been triaged by the area owner

Comments

@vcsjones
Copy link
Member

Steps to reproduce:

  1. Install .NET 6 RC1 on an M1 Mac with the ARM64 architecture. Make sure there are no previous installs of .NET.

    .NET SDK (reflecting any global.json):
    Version:   6.0.100-rc.1.21458.32
    Commit:    d7c22323c4
    
    Runtime Environment:
    OS Name:     Mac OS X
    OS Version:  12.0
    OS Platform: Darwin
    RID:         osx-arm64
    Base Path:   /usr/local/share/dotnet/sdk/6.0.100-rc.1.21458.32/
    
    Host (useful for support):
      Version: 6.0.0-rc.1.21451.13
      Commit:  d7619cd4b1
    
    .NET SDKs installed:
      6.0.100-rc.1.21458.32 [/usr/local/share/dotnet/sdk]
    
    .NET runtimes installed:
      Microsoft.AspNetCore.App 6.0.0-rc.1.21452.15 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
      Microsoft.NETCore.App 6.0.0-rc.1.21451.13 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
    
  2. Try to run a .NET application that targets .NET 5 with dotnet run You'll get output that looks like:

    Failed to load /usr/local/share/dotnet/host/fxr/6.0.0-rc.1.21451.13/libhostfxr.dylib, error: dlopen(/usr/local/share/dotnet/host/fxr/6.0.0-rc.1.21451.13/libhostfxr.dylib, 0x0001): tried: '/usr/local/share/dotnet/host/fxr/6.0.0-rc.1.21451.13/libhostfxr.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/local/lib/libhostfxr.dylib' (no such file), '/usr/lib/libhostfxr.dylib' (no such file)
    The library libhostfxr.dylib was found, but loading it from /usr/local/share/dotnet/host/fxr/6.0.0-rc.1.21451.13/libhostfxr.dylib failed
      - Installing .NET prerequisites might help resolve this problem.
        https://go.microsoft.com/fwlink/?linkid=2063366
    

I would expect to see a better error message, either indicating that .NET 5 is not installed, or that an available SDK/runtime is not available.

Interestingly enough, even with a rollForward policy of major, it still fails with the same error message.

global.json:

{
    "sdk": {
      "allowPrerelease": true,
      "rollForward": "major"
    }
  }
@sfoslund sfoslund transferred this issue from dotnet/core Sep 15, 2021
@dotnet-issue-labeler dotnet-issue-labeler bot added area-Host untriaged New issue has not been triaged by the area owner labels Sep 15, 2021
@ghost
Copy link

ghost commented Sep 15, 2021

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

Steps to reproduce:

  1. Install .NET 6 RC1 on an M1 Mac with the ARM64 architecture. Make sure there are no previous installs of .NET.

    .NET SDK (reflecting any global.json):
    Version:   6.0.100-rc.1.21458.32
    Commit:    d7c22323c4
    
    Runtime Environment:
    OS Name:     Mac OS X
    OS Version:  12.0
    OS Platform: Darwin
    RID:         osx-arm64
    Base Path:   /usr/local/share/dotnet/sdk/6.0.100-rc.1.21458.32/
    
    Host (useful for support):
      Version: 6.0.0-rc.1.21451.13
      Commit:  d7619cd4b1
    
    .NET SDKs installed:
      6.0.100-rc.1.21458.32 [/usr/local/share/dotnet/sdk]
    
    .NET runtimes installed:
      Microsoft.AspNetCore.App 6.0.0-rc.1.21452.15 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
      Microsoft.NETCore.App 6.0.0-rc.1.21451.13 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
    
  2. Try to run a .NET application that targets .NET 5 with dotnet run You'll get output that looks like:

    Failed to load /usr/local/share/dotnet/host/fxr/6.0.0-rc.1.21451.13/libhostfxr.dylib, error: dlopen(/usr/local/share/dotnet/host/fxr/6.0.0-rc.1.21451.13/libhostfxr.dylib, 0x0001): tried: '/usr/local/share/dotnet/host/fxr/6.0.0-rc.1.21451.13/libhostfxr.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/local/lib/libhostfxr.dylib' (no such file), '/usr/lib/libhostfxr.dylib' (no such file)
    The library libhostfxr.dylib was found, but loading it from /usr/local/share/dotnet/host/fxr/6.0.0-rc.1.21451.13/libhostfxr.dylib failed
      - Installing .NET prerequisites might help resolve this problem.
        https://go.microsoft.com/fwlink/?linkid=2063366
    

I would expect to see a better error message, either indicating that .NET 5 is not installed, or that an available SDK/runtime is not available.

Interestingly enough, even with a rollForward policy of major, it still fails with the same error message.

global.json:

{
    "sdk": {
      "allowPrerelease": true,
      "rollForward": "major"
    }
  }
Author: vcsjones
Assignees: -
Labels:

area-Host, untriaged

Milestone: -

@eerhardt
Copy link
Member

cc @ericstj

@richlander
Copy link
Member

Here are the "instructions" on how to configure your machine today.

dotnet/sdk#20584

@ericstj
Copy link
Member

ericstj commented Sep 15, 2021

@richlander it's not clear from the report that folks are trying to do x64 on ARM64 here. My impression is that this is plain ARM64 but maybe @vcsjones can confirm. cc @sfoslund who's doing work on MAC installers for ARM64.

@ericstj
Copy link
Member

ericstj commented Sep 15, 2021

I see, twitter has more info 😆 https://twitter.com/vcsjones/status/1438155493990707207?s=20

Still looks like this might be a host or host deployment issue. I would expect plain-old ARM64 to "just work" and tell folks to either install a 5.0 shared framework or let them roll-forward (if that's what their runtimeconfig says).

@vcsjones
Copy link
Member Author

vcsjones commented Sep 15, 2021

My impression is that this is plain ARM64 but maybe @vcsjones can confirm.

Right. This is just trying to use ARM64 .NET 6 to run a .NET 5 TFM app. I would expect the roll forward to work, but it does not. For the non-rollforward case the error is very unclear.

@agocke
Copy link
Member

agocke commented Sep 15, 2021

I'm surprised this wouldn't fail earlier. How is dotnet build succeeding if the app is .NET 5 and there's no .NET 5 SDK installed?

@vcsjones Does just plain "dotnet build" succeed without an problems?

@vcsjones
Copy link
Member Author

vcsjones commented Sep 15, 2021

Does just plain "dotnet build" succeed without problems?

@agocke It does. The exact project I am using is here: https://gist.github.com/vcsjones/1fef26b5a9ad4bfe934425b272d56797

dotnet build succeeds. dotnet run fails.

The dotnet --info output is the same as in the initial issue comment.

@agocke
Copy link
Member

agocke commented Sep 15, 2021

Thanks, I'll try to figure out what's going on using my M1. I assume that the problem is that the build is publishing the .NET 5 apphost, which is only built for x64. Then, when it tries to run on .NET 6, the runtime is ARM64, so the load fails.

I suspect the only option here is to avoid using the apphost for .NET 5 apps, but that would break non-roll-forward scenarios.

@agocke
Copy link
Member

agocke commented Sep 15, 2021

@vcsjones I just tried this with RC2 and can't replicate the problem -- it's possible an SDK change to fix dotnet run missed RC1.

I'm using

.NET SDK (reflecting any global.json):
 Version:   6.0.100-rc.2.21465.13
 Commit:    0d1cdfa6a0

Could you try an RC2 build from https://github.com/dotnet/installer and see if that fixes the problem? Sorry, getting everything sorted for emulation has been a long and complicated process.

@vcsjones
Copy link
Member Author

Still fails for me. Something I want to call out though because I didn't stress it enough in the original post: this is on macOS 12 (Monterey) beta 21A5506j.

Failed to load /usr/local/share/dotnet/host/fxr/6.0.0-rc.2.21460.8/libhostfxr.dylib, error: dlopen(/usr/local/share/dotnet/host/fxr/6.0.0-rc.2.21460.8/libhostfxr.dylib, 0x0001): tried: '/usr/local/share/dotnet/host/fxr/6.0.0-rc.2.21460.8/libhostfxr.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/local/lib/libhostfxr.dylib' (no such file), '/usr/lib/libhostfxr.dylib' (no such file)
The library libhostfxr.dylib was found, but loading it from /usr/local/share/dotnet/host/fxr/6.0.0-rc.2.21460.8/libhostfxr.dylib failed
  - Installing .NET prerequisites might help resolve this problem.
     https://go.microsoft.com/fwlink/?linkid=2063366

Just for kicks I tried a nightly:

Failed to load /usr/local/share/dotnet/host/fxr/7.0.0-alpha.1.21425.4/libhostfxr.dylib, error: dlopen(/usr/local/share/dotnet/host/fxr/7.0.0-alpha.1.21425.4/libhostfxr.dylib, 0x0001): tried: '/usr/local/share/dotnet/host/fxr/7.0.0-alpha.1.21425.4/libhostfxr.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/local/lib/libhostfxr.dylib' (no such file), '/usr/lib/libhostfxr.dylib' (no such file)
The library libhostfxr.dylib was found, but loading it from /usr/local/share/dotnet/host/fxr/7.0.0-alpha.1.21425.4/libhostfxr.dylib failed
  - Installing .NET prerequisites might help resolve this problem.

Just to make sure I am in a known good state: is sudo rm -rf /usr/local/share/dotnet enough to remove all installations of .NET on macOS previously? I want to make sure I don't have any lingering copies of anything anywhere.

@agocke
Copy link
Member

agocke commented Sep 16, 2021

Hmm, I'm on Big Sur, but it's possible that I'm the one insufficiently clearing my machine install.

@agocke
Copy link
Member

agocke commented Sep 16, 2021

Yup, that was it. I hadn't fully deleted my machine install -- this now fails. However, I can get it to run by using dotnet ../blah.dll instead of using dotnet run.

I think this is the 5.0 apphost. Unfortunately it's x64 only, so if we start the runtime with it we'll be stuck in x64 mode, even if roll forward would work.

  • @mateoatr @vitek-karas I think I missed this case in the interaction table. I suppose in roll forward cases we can't use the apphost if we want this to work, but I'm not sure that's even possible (to properly determine whether or not to publish an apphost based on roll forward).

@agocke
Copy link
Member

agocke commented Sep 16, 2021

@vcsjones Is there a reason you decided to enable roll-forward rather than switching to net6.0 TFM? Or installing the net5.0 runtime?

@vcsjones
Copy link
Member Author

vcsjones commented Sep 16, 2021

@vcsjones Is there a reason you decided to enable roll-forward rather than switching to net6.0 TFM? Or installing the net5.0 runtime?

I just bumped the TFM but I will admit I wasted a bit of time getting detoured by this. I don't really expect it to work now that I understand what is happening, but it took a while to get there. I only called out rollForward because I would expect that to work. Why would it load the .NET 5 AppHost if I should be rolling forward to 6? (Keep in mind I have a very naive understanding with how this works).

Here are the events that led to this:

  • "Kevin, please run this .NET repo"
  • Okay, let me clone this repository, and dotnet run it
  • Sees:

    but is an incompatible architecture (have 'arm64', need 'x86_64'))

  • Oh wow, something must be broken with my installation. Let's go down a technical rabbit hole of inspecting libraries with otool -vh instead of looking at more obvious things.
  • I don't know. Let's re-install?
  • There is nothing at the bottom of this rabbit hole. Let's try something else.
  • Let's try on a brand new .NET app with dotnet new
  • Weird. That worked. What's the difference?
  • Oh. dotnet new made the project with a .NET 6 TFM. Is that it?
  • It was the TFM.
  • ... I wonder if roll forward will fix it?
  • It does not.
  • Let's write an issue.

My hope is that at least this error path can have better output for folks.

@agocke
Copy link
Member

agocke commented Sep 16, 2021

Makes sense. I think our intended path was for you to install the 5.0 runtime instead of enabling roll forward. This is the message I get if I run the app without roll forward:

It was not possible to find any compatible framework version
The framework 'Microsoft.NETCore.App', version '5.0.0' (x64) was not found.
  - The following frameworks were found:
      6.0.0-rc.2.21460.8 at [/Users/angocke/Downloads/dntest/shared/Microsoft.NETCore.App]

You can resolve the problem by installing the specified framework and/or SDK.

The specified framework can be found at:
  - https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=5.0.0&arch=x64&rid=osx.11.0-x64

I grant you -- doing a dual install of ARM64 .NET 6 and x64 .NET 5 is not easy right now (because of dotnet/installer#11164), but I think that if that is fixed, that seems like a better path.

@vcsjones
Copy link
Member Author

vcsjones commented Sep 16, 2021

This is the message I get if I run the app without roll forward:

That's not the error that I am getting. Without roll forward I still get the obtuse "libhostfxr.dylib" experience.

With no global.json:

Failed to load /usr/local/share/dotnet/host/fxr/7.0.0-alpha.1.21425.4/libhostfxr.dylib, error: dlopen(/usr/local/share/dotnet/host/fxr/7.0.0-alpha.1.21425.4/libhostfxr.dylib, 0x0001): tried: '/usr/local/share/dotnet/host/fxr/7.0.0-alpha.1.21425.4/libhostfxr.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/local/lib/libhostfxr.dylib' (no such file), '/usr/lib/libhostfxr.dylib' (no such file)
The library libhostfxr.dylib was found, but loading it from /usr/local/share/dotnet/host/fxr/7.0.0-alpha.1.21425.4/libhostfxr.dylib failed
  - Installing .NET prerequisites might help resolve this problem.
     https://go.microsoft.com/fwlink/?linkid=2063366

This is from the nightly.

Screen Shot 2021-09-15 at 9 27 50 PM

@vcsjones
Copy link
Member Author

@agocke if I got your experience then I don't think I would have ended up filing an issue. I don't want to get too fixated on roll forward because the problem occurs for me roll forward or not.

@agocke
Copy link
Member

agocke commented Sep 16, 2021

Ugh, got my x64 and arm64 copy screwed up again. I get a slightly different, but still not great error message, because I have /etc/dotnet/install_location set up for how it should be after the installer is fixed (pointing to an empty x64 subdirectory).

My error message is:

$ dotnet run
A fatal error occurred. The required library libhostfxr.dylib could not be found.
If this is a self-contained application, that library should exist in [/Users/angocke/tmp/net5/bin/Debug/net5.0/].
If this is a framework-dependent application, install the runtime in the global location [/usr/local/share/dotnet/x64] or use the DOTNET_ROOT environment variable to specify the runtime location or register the runtime location in [/etc/dotnet/install_location].

The .NET runtime can be found at:
  - https://aka.ms/dotnet-core-applaunch?missing_runtime=true&arch=x64&rid=osx.11.0-x64&apphost_version=5.0.10

Unfortunately this is still very misleading.

@ericstj
Copy link
Member

ericstj commented Sep 16, 2021

Should we change the SDK when run from ARM64 host? Seems to me like it shouldn’t be leading users down this path without error. The SDK knows what Host RIDs are supported on different target frameworks.

@vitek-karas
Copy link
Member

We actually did change SDK to adapt to these cases. Originally running ARM64 SDK always produced ARM64 host - even for downlevel apps. With the changes SDK knows to target x64 for downlevel apps. But that assumes that x64 framework is installed on the machine (actually it assumes that the target framework is installed, .NET 5 only ships as x64 for macOS).

.NET 6 SDK doesn't know if .NET 5 is installed on the machine, and I don't think it should try to figure that out.

If the framework is missing the host will fail. The error messages are not good in this case, unfortunately we didn't manage to improve those experiences in .NET 6. The tracking issue is #56146, this specific case is very close to #51102.

@vcsjones
Copy link
Member Author

unfortunately we didn't manage to improve those experiences in .NET 6. The tracking issue is #56146, this specific case is very close to #51102.

I see. Thanks! Please feel free to close this one if y'all think this is covered by existing issues.

@agocke
Copy link
Member

agocke commented Sep 17, 2021

Yeah I'm going to close this as a dup of #51102

In this case it's not just error message -- I think we also try to load a hostfxr with the wrong bitness, which presumably the apphost will need to guard against somehow.

@agocke agocke closed this as completed Sep 17, 2021
@vitek-karas
Copy link
Member

In this case it's not just error message -- I think we also try to load a hostfxr with the wrong bitness, which presumably the apphost will need to guard against somehow.

I would very much like to avoid looking at random binaries on disk and trying to guess their architecture (especially on mac with fat binaries and so on) - especially from the apphost itself (should be as small and as simple as possible).

I looked into this a little bit - on Windows this is "easy" because the LoadLibrary fails with a specific error code. Unfortunately on Linux/macOS I could not find any docs about what error code dlopen returns. You can get the textual message for the error (which we do), but parsing that is not something I'd like to do (it also differs between OSes wildly).

@ericstj
Copy link
Member

ericstj commented Dec 2, 2021

If you want you can install the 6.0 x64 version of dotnet on mac and it will be installed to /usr/local/share/dotnet/x64.
The 6.0 arm64 version of dotnet is installed to /usr/local/share/dotnet

I believe OmniSharp is making updates to make it aware of both arm64 and x64 dotnet SDKs. @JoeRobich is that correct?

@david-d-an
Copy link

david-d-an commented Dec 2, 2021

@ericstj, that certainly is a stop gap solution for the time being. However, I would like to find if MS is considering any mitigation to this ARM/x64 mismatch / mismatch to allow full benefits of M1 power to us DotNet core lovers.

@JoeRobich
Copy link
Member

JoeRobich commented Dec 3, 2021

@ericstj The C# extension is waiting to see if this becomes a common problem for our users or if they simply move to working from the arm64 SDK. FYI @nohwnd

@dong82 I would recommend letting the arm64 CLI use the dotnet name and choose a different alias for the x64 version, since I would expect it to become less and less useful.

@vitek-karas
Copy link
Member

@dong82 I understand the need for downlevel runtimes, but there should be much less need for downlevel SDKs. The .NET 6 SDK should be mostly backward compatible. Meaning that you can use .NET 6 SDK to build .NET 5/.NET Core 3.1 apps. Both the SDK and runtime should now be updated such that trying to run the downlevel apps will pick the correct runtime (x64). The only known limitation currently is certain specifics around dotnet test which doesn't yet fully support this setup, it's in the works and should hopefully ship in a future update to .NET 6 SDK.

@david-d-an
Copy link

@vitek-karas, the important key is "mostly backward compatible". I might bite a bullet and try net6.0 as a single SDK on my personal M1 Mac. Until we could guarantee 100% compatibility, the team wouldn't be able to push Mac developers to eliminate the older SDKs.

Although it is not my favorite, the best alternative seems to have Mac developers use x64 net6.0 for the time being.
Thanks for the updates.

@dustinmoris
Copy link

Heya, I just got here because I have the exact same error as @vcsjones.

I kind of skim read this whole thread and I'm left with the conclusion that it was closed without any guidance or a fix or any advice on how to overcome this problem. What am I missing or is this broken state the expected behaviour?

@vitek-karas
Copy link
Member

@dong82 Just to make sure I understand and to set the right expectations. Your main issue is that VS Code doesn't seamlessly support multiple SDKs across architectures, right? As for the rest:

  • The arm64 dotnet can't run downlevel (5.0, 3.1) SDKs - this was a conscious decision mainly based on the assumption that 6.0 SDK can build/publish downlevel apps just fine.
  • The arm64 SDK does support running the app using the x64 runtimes - that is dotnet run will do the right thing based on the TFM of the app. The same will happen for dotnet test in an upcoming update.

@dustinmoris can you please be a bit more specific what is the "broken state"? Please note that for the various runtimes to coexist you need to update them to the latest versions as described here: dotnet/sdk#22380.

@dustinmoris
Copy link

I am on an M1. I only have the latest .NET 6 SDK installed. I have opened a .NET 5 project and added <RollForward>Major</RollForward> expecting that I can run this project locally without having to change the target framework for my co-workers. Then I got the same exception as described by this issue. Looks like I'm not the only one who has this problem. I don't understand what you even mean by having to have the various runtimes on the latest version. There is only ONE runtime which supports Apple Silicon, right?

@vitek-karas
Copy link
Member

@dustinmoris thanks for the description. There are 4 different runtimes which are supported on Apple Silicon (at least I think):

  • .NET 6 arm64 (natively running on the arm64)
  • .NET 6, 5 and 3.1 x64 - running in emulation

Currently the SDK is designed to work like this:

  • If the app is targeting 6+ (based on the TargetFramework in the project), by default run this on the native architecture if available (so arm64 6 SDK will try to run it on arm64, x64 6 SDK would run it on x64).
  • If the app is targeting <=5 (again based on TargetFramework) the SDK knows that there's no arm64 runtime available for .NET 5 and below, so it will always generate the app as x64. Then it's going to try to run it - which is there's no x64 runtime available will fail.

What we didn't do and maybe we should is to take into account RollForward - that is if you allow the app to run on 6, somehow figure out that there's no x64 runtime on the machine and thus compile the app for arm64, even though it's a 5.0 app. Note that this would be tricky to achieve, since for example the apphost (the small native executable which is used to launch the app) is used based on the TFM, so 5.0 app will get a 5.0 apphost, but that doesn't exist for arm64.
It should be possible to run such app without the apphost, so you can specify <UseAppHost>false</UseAppHost> and that might make it work (not 100% sure, didn't try this) - in essence, running the app via:

dotnet --roll-forward Major app.dll

That should work - assuming the app has no native dependencies (as those are most likely x64 and would fail to load in a arm64 process).

@dustinmoris
Copy link

mhh interesting. Yes I think what you described as "what we didn't do and maybe we should..." is kind of what I thought RollForward was doing ^^

@david-d-an
Copy link

@vitek-karas, thanks for the detailed updates. --roll-forward seems like an interesting approach. In the meantime, I will write down my approach to figure out the problem.

  1. I had older version 3.1 and 5.0 SDK, which were installed in /usr/local/share/dotnet
  2. I installed 6.0.0 SDK for ARM64, which was also installed in /usr/local/share/dotnet
  3. dotnet --info shows all three versions, all looked fine. But I got The required library libhostfxr.dylib could not be found.
  4. . I followed MS suggestions to remove older 3.1 and 5.0 by the instructions here and installed SDK versions 3.1.408 and 5.0.402.
  5. I noticed 3.1.408 and 5.0.402 were installed in /usr/local/share/dotnet/x64 folder instead of /usr/local/share/dotnet/ like the older versions.
  6. I also noticed dotnet binary in /usr/local/share/dotnet folder got deleted during Step 4. So, I ran the 6.0.0. SDK for ARM64 again. dotnet binary appears again.
  7. dotnet --info showed only 6.0.0. To see 3.1. and 5.0 info, I had to /usr/local/share/dotnet/x64/dotnet --info
  8. Runtimes are finally working for all TFMs (net3.1, net5.0 and net6.0)

I think the missing one dotnet binary pointed out at Step 6 was the problem that confused me the most the whole time.

--roll-forward seems like the ultimate approach but if you could update the installer to ensure the removal of 3.1 and 5.0 and also ensure the placement of dotnet binary for version 6.0 would be a quicker and cheaper solution, in my opinion.

@zaneclaes
Copy link

zaneclaes commented Dec 28, 2021

I'm unable to get past this issue even with 6.0.1 (LTS) and quite confused how to proceed. None of the 3 projects I work on will run on my laptop with Dotnet 6. @dong82 perfectly described my current setup (except I have 6.0.1 and 5.0.403 installed, without 3.1) on my M1 laptop. Attempting to start any of the 3 different projects I work on results in the original error message:

watch : Started
Failed to load /usr/local/share/dotnet/x64/host/fxr/5.0.12/libhostfxr.dylib, error: dlopen(/usr/local/share/dotnet/x64/host/fxr/5.0.12/libhostfxr.dylib, 0x0001): tried: '/usr/local/share/dotnet/x64/host/fxr/5.0.12/libhostfxr.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/libhostfxr.dylib' (no such file), '/usr/lib/libhostfxr.dylib' (no such file)
The library libhostfxr.dylib was found, but loading it from /usr/local/share/dotnet/x64/host/fxr/5.0.12/libhostfxr.dylib failed
  - Installing .NET prerequisites might help resolve this problem.
     https://go.microsoft.com/fwlink/?linkid=2063366

I've tried all the solutions here, including the <RollForward>Major</RollForward> and I only have the two specified SDKs installed...

@vitek-karas
Copy link
Member

@zaneclaes Can you please describe in more detail what does this mean "Attempting to start any of the 3 different projects"?

The error you get would typically mean that you're trying to use ARM64 executable (either dotnet or the app's app executable) but for some reason it tries to load the hostfxr from the X64 install location.
Can you check what you have in /etc/dotnet/install_location please?

I only have the two specified SDKs installed...

In the case of 6.0.1 which architecture of the SDK do you have?

@zaneclaes
Copy link

zaneclaes commented Dec 28, 2021

Thanks for looking into it, @vitek-karas ...

Can you please describe in more detail what does this mean "Attempting to start any of the 3 different projects"?

I merely meant to say that I actively contribute to 3 different dotnet projects (with different dependencies), and none of them work — so to your point about "either dotnet or the app's app executable," I was assuming the former.

Can you check what you have in /etc/dotnet/install_location please?

Even after re-installing 6.0.1 multiple times it still points to the 5.0.4 install:

cat /etc/dotnet/install_location
/usr/local/share/dotnet/x64

Though the binary is correct:

which dotnet
/usr/local/share/dotnet/dotnet

I tried manually editing the install location file to /usr/local/share/dotnet, but that did not fix the issue.

In the case of 6.0.1 which architecture of the SDK do you have?

Here's the full dotnet —info:

.NET SDK (reflecting any global.json):
 Version:   6.0.101
 Commit:    ef49f6213a

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  12.1
 OS Platform: Darwin
 RID:         osx.12-arm64
 Base Path:   /usr/local/share/dotnet/sdk/6.0.101/

Host (useful for support):
  Version: 6.0.1
  Commit:  3a25a7f1cc

.NET SDKs installed:
  6.0.101 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.1 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.1 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

Finally, note this is a brand new computer (just got the M1 laptop). I started by installing dotnet 5, then tried to also install dotnet 6.

Screen Shot 2021-12-28 at 12 44 33 PM

@vitek-karas
Copy link
Member

I merely meant to say that I actively contribute to 3 different dotnet projects (with different dependencies), and none of them work — so to your point about "either dotnet or the app's app executable," I was assuming the former.

What I was curious about is what exact command is failing to run (as that can make a difference in figuring out the root cause).

Sorry - I asked the wrong question - in the /etc/dotnet directory, do you have more files and what is their content?

Note that the one command which is currently known to not work (yet) is dotnet test, I'm not sure if it would fail with the exact error you're seeing, but we simply didn't finish all the work on it to make it correctly work in the multi-arch environment. That work is nearly done though and should be available in an upcoming patch release (I don't know exactly which one).

@zaneclaes
Copy link

zaneclaes commented Dec 28, 2021

What I was curious about is what exact command is failing to run (as that can make a difference in figuring out the root cause).

Ah, sorry. dotnet watch run is usually what I use. But now that you mention it... dotnet run seems to work!

Is watch not supported yet, as well? Will it be supported in the same patch release?

Sorry - I asked the wrong question - in the /etc/dotnet directory, do you have more files and what is their content?

There's also install_location_arm64, which does indeed point to /usr/local/share/dotnet.

@david-d-an
Copy link

david-d-an commented Dec 28, 2021

@zaneclaes, There is no such folder as /etc/dotnet in my M1 MacBook and I wonder what created the folder.
However, I noticed that you have two instances dotnet executables under /user/local/share/dotnet folder. I used to have both and that caused conflicts as only one can be invoked.

Unlike other versions, dotnet 3.1.408 and 5.0.402 don't create detnet under /usr/local/share/dotnet/x64 folder during the installation to prevent conflicts with 6.0's dotnet created in /usr/local/share/dotnet. i.e., you should have only one dotnet.
What worked for me was achieved this way:

  1. Remove all dotnet's from M1 Mac system. (rm -rf /usr/local/share/dotnet)
  2. Remove symlinks to dotnet if you have any.
  3. Install SDK's 3.1.408 and 5.0.402
  4. Install SDK 6.0 (any version).

dotnet --info shows Runtime 3.1.14 and Runtime 5.0.11.
However net5.0 and netcoreapp3.1 run off on 5.0.12 and Rtm 3.1.21 respectively.

If done properlyFollowing the instructions, dotnet --info should show all three versions in one output. My current working structure is as follows:

> pwd
/usr/local/share/dotnet
> tree -L 2
.
├── LICENSE.txt
├── ThirdPartyNotices.txt
├── **dotnet**
├── host
│   └── fxr
├── packs
│   ├── Microsoft.AspNetCore.App.Ref
│   ├── Microsoft.NETCore.App.Host.osx-arm64
│   ├── Microsoft.NETCore.App.Host.osx-x64
│   ├── Microsoft.NETCore.App.Ref
│   └── NETStandard.Library.Ref
├── sdk
│   ├── 3.1.408
│   ├── 5.0.402
│   └── 6.0.100
├── sdk-manifests
│   └── 6.0.100
├── shared
│   ├── Microsoft.AspNetCore.App
│   └── Microsoft.NETCore.App
├── templates
│   ├── 3.1.15
│   ├── 5.0.11
│   └── 6.0.0
└── x64
    ├── **dotnet**
    ├── host
    ├── packs
    ├── sdk
    ├── shared
    └── templates

@zaneclaes
Copy link

@dong82

There is no such folder as /etc/dotnet in my M1 MacBook and I wonder what created the folder.

Hm, @vitek-karas told me to look for that folder — so it sounds like it is expected to exist. Maybe that was the problem you had?

Note that, as I mentioned in my last post, dotnet run actually works fine... it's specifically dotnet watch run which causes this error.

@vitek-karas
Copy link
Member

I'm actually not sure about dotnet watch. On the first look I would expect it to work, but that means little. I assume you're trying to run a .NET 5 project that way, right?

What is the exact command line, just dotnet watch?
Can you try running dotnet watch run -a x64 and see if that helps?
Does the project specify UseAppHost=false? (another way to check, if you build it, does it produce executable in the output directory?)

Per dotnet/sdk#21404 it should more or less work as expected.

The new installers (so anything .NET 6 and the latest of .NET 5 and .NET 3.1) should create /etc/dotnet/install_location and possibly other files there. If they don't, that should be reported as a bug. (Those files are necessary for the applications to find the right runtime, so it's not surprising it doesn't work without those files).

@david-d-an
Copy link

david-d-an commented Dec 28, 2021

@zaneclaes,
According to the error statement, dotnet watch run is looking for net5.0 runtime, which is not visible through dotnet command. Your dotnet --info only shows 6.0.1 runtime. dotnet --info on my M1 Mac shows all three versions:

> dotnet --info
.NET SDK (reflecting any global.json):
 Version:   6.0.100
 Commit:    9e8b04bbff

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  12.0
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /usr/local/share/dotnet/sdk/6.0.100/

Host (useful for support):
  Version: 6.0.0
  Commit:  4822e3c3aa

.NET SDKs installed:
  3.1.408 [/usr/local/share/dotnet/sdk]
  5.0.402 [/usr/local/share/dotnet/sdk]
  6.0.100 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 3.1.14 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.11 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.14 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.11 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

@vitek-karas
Copy link
Member

@dong82 Based on the output of your dotnet --info you got your machine into a state which will not work reliably. The dotnet is an arm64 executable, so I assume you installed .NET 6 ARM64. But at the same time you have .NET 5 and .NET 3.1, both x64, installed into the same location /usr/local/share/dotnet. So even if you can see all the frameworks, I would expect that trying to run anything on .NET 5 or .NET 3.1 will fail (since an arm64 process will try to load an x64 runtime).

The fact that dotnet --info when executed from an arm64 install doesn't show frameworks installed into the x64 location is a known issue #51101. But running apps with those should work.

@zaneclaes
Copy link

zaneclaes commented Dec 28, 2021

I'm actually not sure about dotnet watch. On the first look I would expect it to work, but that means little. I assume you're trying to run a .NET 5 project that way, right?

No: <TargetFramework>net6.0</TargetFramework>.

What is the exact command line, just dotnet watch?

dotnet watch run
edit Simply using dotnet watch produces the same error.

Can you try running dotnet watch run -a x64 and see if that helps?

Same error.

Does the project specify UseAppHost=false? (another way to check, if you build it, does it produce executable in the output directory?)

I'm not sure about that flag, but running dotnet build does produce a unix executable file (i.e., MyProject as well as MyProject.dll) in the debug folder output.

@david-d-an
Copy link

@vitek-karas,
The aforementioned runtime is able to run my netcoreapp3.1 app suite which includes dotnet/angular, API, and Identity Server 4 without trouble. This also runs my experimental net6.0 apps properly. After multiple installation/uninstallation trials, the installation steps I describe above got me the current settings and it's been running reliably that way at least for my non-production apps. The difference between SDK's 3.1.408 / 5.0.402 and all other previous versions is the omission of dotnet runtime in x64 folder, and I thought that was Microsoft's intention to have one-runtime-fits-all, but you are saying that that's not. Now, I am scratching my head. 😕

@vitek-karas
Copy link
Member

@zaneclaes that is weird - would it be possible to grab a host trace? export COREHOST_TRACE=1, export COREHOST_TRACEFILE=host.txt and then run the repro. And then share the host.txt. Please note that the file may contain details about your machine like file paths, env. variables and so on. If you can either share it here or send it to me vitkaras at microsoft com.

Can you also try dotnet watch run -a arm64 then?

@dong82 I don't know why your dotnet/x64 doesn't have a dotnet in it - I think it should. But regardless, the reason why things work for you is that you actually have both 3.1 and 5 installed in dotnet/x64. I don't think the 3.1 and 5 installed in dotnet will be picked up (I would expect it to fail if it did). You can probably verify this by running a simple 3.1/5 app and let it print out the location of the runtime. I would recommend you uninstall/remove the 3.1 and 5 from the dotnet directory as that will just cause confusion (and possible failures).

@zaneclaes
Copy link

Can you also try dotnet watch run -a arm64 then?

facepalm I blindly copied your last post, which had said to use dotnet watch run -a x64 ... but it worked this time with -a arm64!

@david-d-an
Copy link

@vitek-karas
You are right, I must have deleted /usr/local/share/dotnet/x64/dotnet by mistake. I have redone all installation steps and saw the runtime in the folder. I updated my previous comments accordingly to prevent confusion.

@david-d-an
Copy link

david-d-an commented Dec 31, 2021

@vitek-karas
After I have redone the whole installation steps again and found what works and what doesn't. Below is what I did to make it mostly work.

  1. Installed SDK 3.1.408
    Works well with netcoreapp3.1
  2. Installed SDK 5.0.402
    Works well with netcoreapp3.1 and net5.0
  3. System works well with 3.1 and 5.0 and this is what I had before attempting 6.0 installation two months ago.
  4. Installed SDK 6.0.10
    dotnet --info still shows all three versions of SDKs and Runtimes
    3.1 and 5.0 stopped working. SDK 3.1.408 and SDK 5.0.402 were wiped up from /usr/local/share/dotent
    /usr/local/share/dotent/x64 folder doesn’t exist
  5. Installed SDK 3.1.415 and SDK 5.0.403 (per Microsoft)
  6. /usr/local/share/dotent/x64 folder showed up.
    SDK 3.1.415 and SDK 5.0.403 appeared in x64 folder
    dotnet created by SDK 6.0.10 got wiped up and disappeared from /usr/local/share/dotnet
  7. Reinstalled SDK 6.0.10
  8. dotnet appeared in /usr/local/share/dotnet
  9. dotnet --info shows
    SDK 3.1.408 Rtm 3.1.14
    SDK 5.0.402 Rtm 5.0.11
    These are older versions I installed (Steps 1 & 2) but got wiped up during 6.0 installation
  10. System.Runtime.InteropServices.RuntimeInformation.FrameworkDescription captures different runtimes:
    Rtm 3.1.21 and Rtm 5.0.12
    I installed these after 6.0 installation wiped up my previous SDK/Runtime
  11. dotnet run works well with netcoreapp3.1, net5.0, net6.0
  12. dotnet publish works well with netcoreapp3.1, net5.0, net6.0
  13. dotnet {appname}.dll works well with net6.0
  14. dotnet {appname}.dll does NOT work with netcoreapp3.1 or net5.0
  15. /usr/local/share/dotnet/x64/dotnet {appname}.dll works with netcoreapp3.1 or net5.0

The tricky parts are misleading dotnet --info and Step 15. Otherwise, this setting works for me.
Happy new year.

@vitek-karas
Copy link
Member

@dong82

Anything below 3.1.415 and 5.0.403 is not supported on M1 hardware (because it installs into the wrong location). As per the issues referenced above, the recommended action is to:

  • Completely clear the entire /usr/local/share/dotnet
  • Install the latest versions only (which should install into the correct locations)

With that the dotnet -info showing old versions in step 4 is expected... and we won't fix this (unsupported scenario).

Step 6. where you say it removed the dotnet from /usr/local/share/dotnet - this is weird, I would not expect this to happen. If you can reliably repro this, ideally without installing the unsupported versions, please create a new issue for that.

Again, step 9. is expected and won't be fixed. On top of that dotnet --info is relatively "stupid" in that it will show these even if they're just empty directories. Again, this should not happen with the installers.

Step 10. The fact that the runtime version differs from the SDK version is expected - I don't know if these are the versions to be expected, but there's no rule which says that the SDK and runtime versions have to match exactly (typically they don't).

Step 14. is currently expected and by design. Without the unsupported versions, you should get an error about missing framework. Ideally when we fix all of the details, this error should contain the fact that you have that framework installed in the x64 directory as well. We expect most users will use the generated appname executable which should work in all cases. The dotnet appname.dll invocation is considered a bit more advanced scenario. In this case it should work if you use x64/dotnet to run it (as you verified in step 15).

Happy new year to you as well.

@david-d-an
Copy link

@vitek-karas,
I finally caught up with your level of understanding. All works fine and I appreciate it. Hopefully, my trial-error helps other people trying similar settings.

As you mentioned, bin/Release/net5.0/publish/{appname} executable is a convenient way to run the app with hardware architectures abstracted away. I just have to be careful not to rely on this too much as my deployment scenarios do include x64 architectures most of the time.

The problems on Step 6 are persistent and I will submit a separate issue for this.

Thanks

@ghost ghost locked as resolved and limited conversation to collaborators Feb 1, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-arm64 area-Host os-mac-os-x macOS aka OSX untriaged New issue has not been triaged by the area owner
Projects
None yet
Development

No branches or pull requests