Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aspire 9 fails to stop projects #6704

Closed
1 task done
yoDon opened this issue Nov 18, 2024 · 45 comments
Closed
1 task done

Aspire 9 fails to stop projects #6704

yoDon opened this issue Nov 18, 2024 · 45 comments
Assignees
Milestone

Comments

@yoDon
Copy link

yoDon commented Nov 18, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Describe the bug

Aspire 9 seems to have lots of problems stopping projects.

In Aspire 8, I routinely started and stopped solutions without issue.

In Aspire 9 on OSX with Podman Desktop and the Rider Aspire plugin, about half the time there are stuck ports preventing Aspire projects from running correctly. I don't know about other operating systems.

All of this is somewhat anecdotal, as I don't have a repro that happens every time, I just know I commonly have the problem. That said, I think closing open browser tabs from the previous session may reduce the incidence of this issue (that was not required with Aspire 8).

Expected Behavior

Starting, stopping, and restarting Aspire projects should "just work", including regardless of whether there are browser tabs left open from a previous run.

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version info

IMPORTANT NOTE: I'm running via JetBrains Rider 2024.3 Build #RD-243.21565.191 on OSX, which I believe contains its own dotnet that doesn't show up using dotnet --info.

That said, here is my dotnet --info:

.NET SDK:
Version: 8.0.201
Commit: 4c2d78f037
Workload version: 8.0.200-manifests.2772ffde

Runtime Environment:
OS Name: Mac OS X
OS Version: 14.6
OS Platform: Darwin
RID: osx-arm64
Base Path: /usr/local/share/dotnet/sdk/8.0.201/

.NET workloads installed:
[aspire]
Installation Source: SDK 8.0.200
Manifest Version: 8.2.1/8.0.100
Manifest Path: /usr/local/share/dotnet/sdk-manifests/8.0.100/microsoft.net.sdk.aspire/8.2.1/WorkloadManifest.json
Install Type: FileBased

Host:
Version: 8.0.4
Architecture: arm64
Commit: 2d7eea2529

.NET SDKs installed:
6.0.401 [/usr/local/share/dotnet/sdk]
6.0.403 [/usr/local/share/dotnet/sdk]
6.0.404 [/usr/local/share/dotnet/sdk]
6.0.405 [/usr/local/share/dotnet/sdk]
6.0.407 [/usr/local/share/dotnet/sdk]
6.0.408 [/usr/local/share/dotnet/sdk]
6.0.410 [/usr/local/share/dotnet/sdk]
6.0.413 [/usr/local/share/dotnet/sdk]
6.0.415 [/usr/local/share/dotnet/sdk]
7.0.100 [/usr/local/share/dotnet/sdk]
7.0.101 [/usr/local/share/dotnet/sdk]
7.0.102 [/usr/local/share/dotnet/sdk]
7.0.202 [/usr/local/share/dotnet/sdk]
7.0.203 [/usr/local/share/dotnet/sdk]
7.0.304 [/usr/local/share/dotnet/sdk]
7.0.307 [/usr/local/share/dotnet/sdk]
7.0.410 [/usr/local/share/dotnet/sdk]
8.0.100 [/usr/local/share/dotnet/sdk]
8.0.201 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 6.0.9 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.11 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.12 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.13 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.15 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.16 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.18 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.21 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.23 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.1 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.2 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.4 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.5 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.7 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.10 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.20 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.2 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 6.0.9 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.11 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.12 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.13 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.15 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.16 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.18 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.21 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.23 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.1 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.2 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.7 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.10 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.20 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.2 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
x64 [/usr/local/share/dotnet/x64]

Environment variables:
Not set

global.json file:
/Users/........./global.json

Learn more:
https://aka.ms/dotnet/info

Download .NET:
https://aka.ms/dotnet/download

Anything else?

No response

@yoDon
Copy link
Author

yoDon commented Nov 18, 2024

Adding sample exceptions:

fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
      could not start the proxy {"ServiceName": {"name":"aspnetcore_project01"}, "Reconciliation": 4, "error": "could not start the proxy for the service: listen tcp [::1]:5263: bind: address already in use"}
fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
      could not start the proxy {"ServiceName": {"name":"aspnetcore_project03-http"}, "Reconciliation": 5, "error": "could not start the proxy for the service: listen tcp [::1]:5236: bind: address already in use"}
fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
      could not start the proxy {"ServiceName": {"name":"aspnetcore_project02-http"}, "Reconciliation": 12, "error": "could not start the proxy for the service: listen tcp [::1]:5019: bind: address already in use"}
fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
      could not start the proxy {"ServiceName": {"name":"yarp"}, "Reconciliation": 29, "error": "could not start the proxy for the service: listen tcp [::1]:6001: bind: address already in use"}
fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
      could not start the proxy {"ServiceName": {"name":"aspnetcore_project01"}, "Reconciliation": 30, "error": "could not start the proxy for the service: listen tcp [::1]:5263: bind: address already in use"}
fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
      could not start the proxy {"ServiceName": {"name":"aspnetcore_project02-http"}, "Reconciliation": 31, "error": "could not start the proxy for the service: listen tcp [::1]:5019: bind: address already in use"}
...

and

fail: Aspire.Hosting.Dashboard.Microsoft.Extensions.Hosting.Internal.Host[11]
      Hosting failed to start
      System.IO.IOException: Failed to bind to address http://127.0.0.1:15071: address already in use.
       ---> Microsoft.AspNetCore.Connections.AddressInUseException: Address already in use
       ---> System.Net.Sockets.SocketException (48): Address already in use
         at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
         at System.Net.Sockets.Socket.Bind(EndPoint localEP)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
         --- End of inner exception stack trace ---
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass28_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
      --- End of stack trace from previous location ---

@jack775544
Copy link

A coworker of mine has been getting similar issues. In that case they were running an ARM MacOS (not sure of the OS version right now) device and starting the app host using dotnet run from the command line.

Upon stopping and starting the app host, all the C# projects were failing to start since the ports were already in use.

tjementum added a commit to platformplatform/PlatformPlatform that referenced this issue Nov 19, 2024
### Summary & Motivation

Aspire is now configured to keep SQL Server alive between debug sessions
using the new Aspire 9 `.WithLifetime(ContainerLifetime.Persistent)`,
significantly improving secondary startup.

Unfortunately, Aspire does not seem to release ports immediately upon
shutdown, requiring 10 seconds to a minute before the AppHost can
restart. To mitigate this issue temporarily, logic has been added to
kill any Developer Control Pane processes (`dcpctrl`) from previous
sessions when starting the AppHost, preventing conflicts and strange
errors. For more details, see this issue:
dotnet/aspire#6704.

The SPA generation and `index.html` caching strategy have been updated
to align with the new Aspire startup behavior during debugging.

The AppGateway no longer launches automatically when starting Aspire
AppHost, as it needs a moment to be ready.

Additionally, a bug has been resolved where the frontend incorrectly
used the old `platformplatform.pfx` certificate instead of
`localhost.pfx`.

Lastly, a fix for a duplicated `dotnet dotnet` issue in the README has
been applied.

### Checklist

- [x] I have added a Label to the pull-request
- [x] I have added tests, and done manual regression tests
- [x] I have updated the documentation, if necessary
@tjementum
Copy link

I have the same issue. It seems like the Developer Control Pane takes up to 1 minute to shut down. I made an ugly hack to find and kill any hanging control pane as the first step when starting the AppHost.

See this commit: platformplatform/PlatformPlatform@1772f94

@joperezr joperezr added this to the Backlog milestone Nov 20, 2024
@joperezr joperezr removed the untriaged New issue has not been triaged label Nov 20, 2024
@yoDon
Copy link
Author

yoDon commented Nov 20, 2024

If you are impacted by this, it MAY help to edit your AppHost/Properties/launchSettings.json file between debugger runs to change the Dashboard port number between launches (I bump the port number up or down by 1 each time).

This gives the Dashboard extra time to reset (others have reported the Dashboard is the worst/longest duration stuck port offender here).

If you have multiple other projects also getting stuck, this unfortunately won't unblock the others.

@tjementum
Copy link

others have reported the Dashboard is the worst/longest duration stuck port offender here

That’s what I see too. If I restart before the DCP process is ready, it sometimes takes more than a 1 minute to release the ports... but if you wait 10–15 seconds before starting, it almost always releases. In Aspire 8.2, this used to be 3-5 seconds, and in Aspire 8.0 this was not a problem at all.

The code below works for me (only tested on macOS). The only problem is that the browser starts immediately, and the dashboard is not properly shown until I manually hit refresh.

using System.Diagnostics;
using AppHost;
using Projects;

// Detect if Aspire ports from the previous run are released. See https://github.com/dotnet/aspire/issues/6704
EnsureDeveloperControlPaneIsNotRunning();

var builder = DistributedApplication.CreateBuilder(args);

// Aspire configuraiton  here...

await builder.Build().RunAsync();
 

void EnsureDeveloperControlPaneIsNotRunning()
{
    const string processName = "dcpctrl"; // The Aspire Developer Control Pane process name

    var process = Process.GetProcesses()
        .SingleOrDefault(p => p.ProcessName.Contains(processName, StringComparison.OrdinalIgnoreCase));

    if (process == null) return;

    Console.WriteLine($"Shutting down developer control pane from previous run. Process: {process.ProcessName} (ID: {process.Id})");

    Thread.Sleep(TimeSpan.FromSeconds(5)); // Allow Docker containers to shut down to avoid orphaned containers

    try
    {
        process.Kill();
        Console.WriteLine($"Process {process.Id} killed successfully.");
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Failed to kill process {process.Id}: {ex.Message}");
    }
}

@yoDon
Copy link
Author

yoDon commented Nov 28, 2024

@tjementum's work around has been working great for me

@Kralizek
Copy link

Kralizek commented Dec 3, 2024

Also impacted by this. Quickly stopping and restarting the application might get you in the reconciliation hell. stop, wait and start seems to solve it.

Happy to provide logs if needed.

@thomasvdb
Copy link

Using the workaround of @tjementum as well by opening the task manager and filtering the processes on dcpctrl and ending the task before starting the application again.

@davidfowl davidfowl modified the milestones: Backlog, 9.1 Dec 3, 2024
@davidfowl
Copy link
Member

@dbreshears putting this in 9.1

@karolz-ms
Copy link
Member

@yoDon I would appreciate a report if the latest bits in the Aspire main branch work any better for you. This PR #6802 should help a lot.

@Kralizek yes, please, if you could share the logs, that would be very helpful

@tjementum @thomasvdb would you be wiling to share some detailed logs with us?
Killing dcpctrl is not a great long-term solution; the post-run cleanup does important things, e.g. deleting the dedicated Aspire container network, if you do not let it happen, you will run into issues like #6639. We have seen cases when shutdown is slow because containers one uses are slow to shut down gracefully; using a persistent container in that case might be a better workaround.

@tjementum
Copy link

Would you be wiling to share some detailed logs with us?

Please find my log below. It's very easy to reproduce:

  1. Clone: https://github.com/platformplatform/PlatformPlatform (requires .NET 9, Docker, and NodeJS)
  2. Remove the EnsureDeveloperControlPaneIsNotRunning() work around from AppHost/Program.cs
  3. Start the AppHost, and wait until all services are running
  4. Stop the project
  5. Start the AppHost

Please note that all ports are hardcoded. It is not a good developer experience when Aspire changes all ports when AppHost is restarted. If the ports were changed each time, this issue would be resolved.

Here is a recording.

Image

info: Aspire.Hosting.DistributedApplication[0]
      Aspire version: 9.0.0+01ed51919f8df692ececce51048a140615dc759d
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application starting.
info: Aspire.Hosting.DistributedApplication[0]
      Application host directory is: /Users/thomasjespersen/Developer/PlatformPlatform/application/AppHost
fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
      could not start the proxy {"ServiceName": {"name":"sql-server"}, "Reconciliation": 2, "error": "could not start the proxy for the service: listen tcp [::1]:9002: bind: address already in use"}
fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
      could not start the proxy {"ServiceName": {"name":"azure-storage-blob"}, "Reconciliation": 5, "error": "could not start the proxy for the service: listen tcp [::1]:10000: bind: address already in use"}
fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
      could not start the proxy {"ServiceName": {"name":"mail-server-http"}, "Reconciliation": 13, "error": "could not start the proxy for the service: listen tcp [::1]:9003: bind: address already in use"}
fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
      could not start the proxy {"ServiceName": {"name":"mail-server-tcp"}, "Reconciliation": 14, "error": "could not start the proxy for the service: listen tcp [::1]:9004: bind: address already in use"}
fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
      could not start the proxy {"ServiceName": {"name":"app-gateway"}, "Reconciliation": 16, "error": "could not start the proxy for the service: listen tcp [::1]:9000: bind: address already in use"}
info: Aspire.Hosting.DistributedApplication[0]
      Now listening on: https://localhost:9001
info: Aspire.Hosting.DistributedApplication[0]
      Login to the dashboard at https://localhost:9001/login?t=747bbd50-7624-46ac-be4d-e823cd594683
fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
      could not start the proxy {"ServiceName": {"name":"sql-server"}, "Reconciliation": 17, "error": "could not start the proxy for the service: listen tcp [::1]:9002: bind: address already in use"}
fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
      could not start the proxy {"ServiceName": {"name":"mail-server-http"}, "Reconciliation": 18, "error": "could not start the proxy for the service: listen tcp [::1]:9003: bind: address already in use"}
fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
      could not start the proxy {"ServiceName": {"name":"mail-server-tcp"}, "Reconciliation": 19, "error": "could not start the proxy for the service: listen tcp [::1]:9004: bind: address already in use"}
fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
      could not start the proxy {"ServiceName": {"name":"azure-storage-blob"}, "Reconciliation": 20, "error": "could not start the proxy for the service: listen tcp [::1]:10000: bind: address already in use"}

@Kralizek
Copy link

Kralizek commented Dec 6, 2024

My system behaves exactly the same as shown by @tjementum

@davidfowl
Copy link
Member

Are you using Rider as well @Kralizek ? Can you reproduce this using dotnet run as well?

@Kralizek
Copy link

Kralizek commented Dec 6, 2024

Also using Rider.

Never occurred in console, mostly because i use dotnet watch in console so i don't need stop/start sp quickly

@davidfowl
Copy link
Member

We're trying to narrow down if this is a Rider issue or an aspire orchestration issue. So far it looks like the common denominator so far is rider (or some combination). Can you see if you can reproduce the issue on the command line with dotnet run? Watch has other problems.

@Kralizek
Copy link

Kralizek commented Dec 6, 2024

I confirm I wasn't able to reproduce the issue with dotnet run.

$ dotnet run --project .\tools\AppHost\ --no-build
Using launch settings from .\tools\AppHost\Properties\launchSettings.json...
info: Aspire.Hosting.DistributedApplication[0]
      Aspire version: 9.0.0+01ed51919f8df692ececce51048a140615dc759d
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application starting.
info: Aspire.Hosting.DistributedApplication[0]
      Application host directory is: C:\Users\rg1844\Development\XXX\tools\AppHost
info: Aspire.Hosting.DistributedApplication[0]
      Now listening on: https://localhost:17197
info: Aspire.Hosting.DistributedApplication[0]
      Login to the dashboard at https://localhost:17197/login?t=74b0335988b84d6689b5e91c02e9f0cb
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application started. Press Ctrl+C to shut down.

$ dotnet run --project .\tools\AppHost\ --no-build
Using launch settings from .\tools\AppHost\Properties\launchSettings.json...
info: Aspire.Hosting.DistributedApplication[0]
      Aspire version: 9.0.0+01ed51919f8df692ececce51048a140615dc759d
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application starting.
info: Aspire.Hosting.DistributedApplication[0]
      Application host directory is: C:\Users\rg1844\Development\XXX\tools\AppHost
info: Aspire.Hosting.DistributedApplication[0]
      Now listening on: https://localhost:17197
info: Aspire.Hosting.DistributedApplication[0]
      Login to the dashboard at https://localhost:17197/login?t=aa42706c7ba4b03ff0518a69e1c1a4bc
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application started. Press Ctrl+C to shut down.

@thomasvdb
Copy link

Here using Rider as well with this plugin: https://github.com/JetBrains/aspire-plugin

@karolz-ms
Copy link
Member

I was able to reproduce the issue with Visual Studio after a few tries. Investigating...

@NiklasEi
Copy link

NiklasEi commented Dec 6, 2024

But I would not consider this fixed if the only change is randomizing the ports. In some cases I want hardcoded ports to, for example, make it easier to run tests against the locally running system.
In my opinion, the shutdown should wait until all resources are gone.

@karolz-ms
Copy link
Member

#6931 should have fixed this, but I would appreciate a report if people still see issues with projects not stopping as they should

@andrem0
Copy link

andrem0 commented Dec 13, 2024

@karolz-ms when will this be released? we are very unpatently waiting for the fix :'D

@karolz-ms
Copy link
Member

@andrem0 I hear you. You can try the fix today with the Aspire daily builds https://github.com/dotnet/aspire/blob/main/docs/using-latest-daily.md

As for the "official" release, I think the plan of record is to have one sometime in the second half of January (plans might change).

@feritzcan2
Copy link

@karolz-ms I tried daily builds but its still not fixed for me... on m4 mac - rider.. this issue is blocking my company from switching to aspire... suggested solution above also doesnt work all time

@karolz-ms
Copy link
Member

@feritzcan2 does it reproduce when the app is run from the command line (dotnet run?). If yes, could share a repro project?

@feritzcan2
Copy link

@karolz-ms I cant share from our project unfortunatelly but we have a lot of services and we created a layer in top of aspire.
I had limited time to test on visual studio- windows, couldnt produce it. It might be only happening on rider plugin - mac

@karolz-ms
Copy link
Member

@feritzcan2 understood, no worries. It would be really interesting to understand if the issue happens when the app host is started from command line (on the Mac), so we can differentiate between this being Aspire/DCP issue vs Aspire Rider plugin issue.

What are the errors you see in the app host output?

@feritzcan2
Copy link

Those are part of errors

info: Aspire.Hosting.Dcp.dcpctrl.ExecutableReconciler[0]
process started {"Executable": {"name":"FunctionPaymentsWorker-dnnkquvx"}, "Reconciliation": 30, "executable": "/opt/homebrew/bin/func", "PID": 47831}
info: Aspire.Hosting.Dcp.dcpctrl.ExecutableReconciler[0]
starting process... {"Executable": {"name":"FunctionSessionsFunction-fwtkrgqd"}, "Reconciliation": 31, "executable": "/opt/homebrew/bin/func"}
info: Aspire.Hosting.Dcp.dcpctrl.ExecutableReconciler[0]
process started {"Executable": {"name":"FunctionSe-fwtkrgqd"}, "Reconciliation": 31, "executable": "/opt/homebrew/bin/func", "PID": 47834}
info: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
could not use the same port for all addresses associated with requested service address, service will be reachable only using specific IP address {"ServiceName": {"name":"bff-https"}, "Reconciliation": 29, "AttemptedCommonPort": 7201, "RequestedServiceAddress": "localhost"}
fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]

@feritzcan2
Copy link

yes I got the same from commandd line dotnet run @karolz-ms
Image

@karolz-ms
Copy link
Member

@feritzcan2 thank you

When you start the application, please take a note the path to the dcp and dcpctrl processes that the Aspire application host is launching. Then from the command line do <path-to-dcp-executable>/dcp version. It should be version 0.9.2. If it is anything else, you are not running the latest.

If that checks out, would you be willing to send me detailed execution logs? You can reach me at (the part of my GH alias before -ms)@microsoft.com and I will send you instructions.

@feritzcan2
Copy link

@karolz-ms yes it seems like its not using latest version. But we have nuget config and I set dotnet9 source there. What else should I do?

Image

@karolz-ms
Copy link
Member

Thanks @feritzcan2

To verify, I followed the instructions in https://github.com/dotnet/aspire/blob/main/docs/using-latest-daily.md. I do recommend to create a local nuget.config file to avoid changing your global Nuget configuration.

After the application is built and started search for dcp process and note the path

> ps aux | grep dcp

...
karolz           37808   0.0  0.2 35441568  52788 s005  S     8:59AM   0:01.62 
/Users/karolz/.nuget/packages/aspire.hosting.orchestration.osx-x64/9.1.0-preview.1.24619.1/tools/dcp 
start-apiserver --monitor 37740 (other parameters)

Verify the version

> /Users/karolz/.nuget/packages/aspire.hosting.orchestration.osx-x64/9.1.0-preview.1.24619.1/tools/dcp version
{"version":"0.9.2","commitHash":"e1586b9dd7b6b4bb15cf0b81000ba8ec55854887","buildTimestamp":"2024-12-11T17:11:30Z"}

0.9.2 is what you want.

Here is my local nuget.config, note the dotnet9 source that enables getting latest daily Aspire builds

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <!--To inherit the global NuGet package sources remove the <clear/> line below -->
    <clear />
    <add key="nuget" value="https://api.nuget.org/v3/index.json" />
    <add key="dotnet9" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet9/nuget/v3/index.json" />
  </packageSources>
</configuration>

... and the AppHost project, note SDK and Aspire.Hosting.AppHost references

<Project Sdk="Microsoft.NET.Sdk">

  <Sdk Name="Aspire.AppHost.Sdk" Version="9.1.0-preview.1.24619.1" />

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net9.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <IsAspireHost>true</IsAspireHost>
    <UserSecretsId>93445db8-0185-405d-8d0e-84e3c7e02bd1</UserSecretsId>
  </PropertyGroup>

  <ItemGroup>
    <ProjectReference Include="..\starterapp.ApiService\starterapp.ApiService.csproj" />
    <ProjectReference Include="..\starterapp.Web\starterapp.Web.csproj" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Aspire.Hosting.AppHost" Version="9.1.0-preview.1.24619.1" />
  </ItemGroup>

</Project>

Hope this helps! LMK if after switching ot DCP 0.9.2 your app is shut down properly or not--TIA

@feritzcan2
Copy link

@karolz-ms unfortunatelly still not working :(

info: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
could not use the same port for all addresses associated with requested service address, service will be reachable only using specific IP address {"ServiceName": {"name":"pay-https"}, "Reconciliation": 65, "AttemptedCommonPort": 52258, "RequestedServiceAddress": "localhost"}

Image Image

@karolz-ms
Copy link
Member

@feritzcan2 is this the only unusual message that you see? Note that this is an "info" message, not an error--the app should still work, although connecting to pay-https service may be slow. Is the app working?

Regarding port that the message mentions (52258):

  1. Is it part of the application configuration? In particular, does it show up in any launchsettings.json file?
  2. When the application is stopped (and all the application processes are cleaned up, manually if necessary): is there another process on the machine that is listening on that port? Do lsof -i @localhost to check please

@feritzcan2
Copy link

@karolz-ms Hey karol, unfortunatelly there is also error that i didnt share... It happens almost everytime I stop and start immediately, and cannot send requests to the ports that I m using.

I'm sharing only part of log but that fail logs exist for all services

fail: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
could not start the proxy {"ServiceName": {"name":"payment-https"}, "Reconciliation": 160, "error": "could not start the proxy for the service: listen tcp [::1]:52258: bind: address already in use"}
info: Aspire.Hosting.Dcp.dcpctrl.ServiceReconciler[0]
could not use the same port for all addresses associated with requested service address, service will be reachable only using specific IP address {"ServiceName": {"name":"payment-http"}, "Reconciliation": 161, "AttemptedCommonPort": 52257, "RequestedServiceAddress": "localhost"}

@feritzcan2
Copy link

Running that command for DCP process running also validates I'm running 9.1.0

ps xuwww -p 35241

USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
fx 35241 0.2 0.1 411919040 74528 s000 S 12:07PM 0:02.43 /Users/fx/.nuget/packages/aspire.hosting.orchestration.osx-arm64/9.1.0-preview.1.24619.1/tools/dcp start-apiserver --monitor 35204 --kubeconfig /var/folders/gz/10gsc5ts1xb1kq6ptm52k3k80000gn/T/aspire.PuOBWE/kubeconfig --container-runtime

@karolz-ms
Copy link
Member

@feritzcan2 thank you. Would you be willing to send me detailed logs from the failing run? Email me please for instructions if yes--thanks in advance!

@feritzcan2
Copy link

@karolz-ms yeah I also reallywant to see this is fixed , sending you an email

@karolz-ms
Copy link
Member

Status update: I got the logs from @feritzcan2 (many thanks!) that revealed another case where the Executable shutdown is not done as quickly as it can and should, which causes subsequent run to time out trying to acquire the TCP port. Working on a fix...

@AndiRudi
Copy link

AndiRudi commented Jan 5, 2025

I somehow feel this issue is really important. We are using asp.net mvc with aspire and the developer satisfaction right now is not good. On the one side dotnet watch does not work well which already makes it hard to do frontend changes because you need to start and stop the solution all the time. But now with this issue I even have to stop the solution, kill processes and then restart just to see minor html changes.

Thanks for taking a look.

@karolz-ms
Copy link
Member

@AndiRudi agreed. @feritzcan2 helped me test the fix (many thanks!) which should resolve the issue for good; that fix should hit Aspire main branch sometime later this week, so at least you will be able to use Aspire daily builds with the fix until a new Aspire release is made available. I will everybody know via this issue when that happens.

dotnet watch is another story; it is being fixed as well to make it work well with Aspire, but it is a pretty big change for the SDK. I do not have an ETA at this point.

@Kralizek
Copy link

Kralizek commented Jan 6, 2025

What are the symptoms of the dotnet watch issue?

I see some strange errors at least once per session asking me to reload the application but I see the application is still running and being updated as usual.

@karolz-ms
Copy link
Member

What are the symptoms of the dotnet watch issue?

See dotnet/sdk#36971

@karolz-ms
Copy link
Member

Should be fixed (Aspire main branch) by #7085.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests