Consume Aspire Resource Metadata to Influence IaC generation

[ellismg]

Aspire is adding the concept of a metadata bag on each resource, which can be used to smuggle information from the app host into consumers of the manifest.

dotnet/aspire#1619

We'd like to use this as a way to enable a way to make targeted configuration changes to the generated IaC (to save folks the trouble of running infra synth, modifying the bicep or yaml and then keeping it up to date.

We should support this in azd and also understand a set of metadata we define that allows customization of our bicep and yaml generation.

A good end to end here would be the ability in the app host to add metadata to a project resource with keys like: azd.minInstances and azd.maxInstances which would control the min and max instance count in the generated container app.

We may also use this technology to control things like "redis.v0" means "use a production version instead of the ACA dev service".

Let's use this issue to track building the infrastructure needed to do this and one small end to end and then sit and think about the knobs we want to add across the components that continue to exist after we do the bicep.v0 resource work and deprecate support for all the azure. resources.

[davidfowl]

We should support this in azd and also understand a set of metadata we define that allows customization of our bicep and yaml generation.

I was thinking we should look for a generic solution here. For the bicep generation we're moving to bicep resources so that basically out of azd's hand (at least for those resources). That leaves a couple of components where infra synth is required to tweak:

1. The resources that azd creates for the compute environment (resources.bicep):

   • Container registry
   • Managed identity
   • LA workspace
   • Container apps environment

2. The container apps themselves (containerApp.yaml.tmpl)

Invisible resources are a problem because there's no representation for them in the app model at all. I'm not sure we'll have time to model them or not, but we should enable modifying arbitrary container apps properties. I think we want this to be as flexible as possible (following the pattern of the bicep resource).

Since we're on the path with the bicep resource, it seems like it would make sense to model the container app manifest as more bicep. If we went this route, aspire would attach/add the container app bicep resource to the manifest and azd would produce the right scaffolding to make this work.

Still working through a proposal.

[ellismg]

Still working through a proposal.

Yeah - I was looking at this again and in light of where I think we are moving, I'm not sure if we'll need this in the fullness of time.

I think there are three "categories" of resource types floating around today:

1. Cloud Agnostics Resources (e.g. redis.v0, postgres.v0, etc). Today azd does the infrastructure generation for these resource types and infra syth is the customization story.
2. Cloud specific resources (e.g. stuff from Aspire.Hosting.Azure). Today these are special azure.XYZ resources that azd understands, but we are moving to a world where they no longer exist and instead Aspire.Hosting.Azure produces some bicep which we include in our graph.
3. Aspire specific resources like project.v0 which implicitly have cloud resources (like the container app environment and container app) which back them.

For (2) the plan is to move to a world where you have full control over customization (via the C# type system). For (1), I think we are thinking we'd be on the same plan, where you'd be able to write some code in your app host that sets what implementation you want to use at deployment time.

For (3) (which may only be project.v0) resources I'm still not sure what the customization plan is. I think in the fullness of time we want to support a model like (1) where you can control the IaC at the C# layer (e.g. some sort of .WithContainerAppForDeployment(app => { /* party on a type from Azure.ResourceManager.AppContainers or something */}) and this just generates bicep (or the yaml rendering of the resource) and we deploy that.

In that world, I can see how we don't need this at all.

But I'm also not sure if we get the project stuff figured out for GA, so I'm not sure if we want to use this as a way to control metadata on the app itself in the short term.

Invisible resources are a problem because there's no representation for them in the app model at all. I'm not sure we'll have time to model them or not, but we should enable modifying arbitrary container apps properties. I think we want this to be as flexible as possible (following the pattern of the bicep resource).

I think the right way for us to do this is to have metadata on parameters that say "what type" you need and azd tries to wire up the right thing (creating a default one if needed). Then, over time, as more and more of these invisible resources are generated on the C# side of the house, you can just drop the parameters from the .bicep file you generate, and azd can use that as a signal that it need not generate these resources anymore (if nothing requests them).

[rajeshkamal5050]

• Not in scope for Aspire Preview 4. Assigning to the Germanium bucket.
• We can pull this back into milestones when we firm up on the proposal/scope.

@ellismg @davidfowl

[davidfowl]

For (2) the plan is to move to a world where you have full control over customization (via the C# type system). For (1), I think we are thinking we'd be on the same plan, where you'd be able to write some code in your app host that sets what implementation you want to use at deployment time.

Right, we got here in preview4 with the bicep resource. So the abstract resource types will go away and azd will stich things together.

For (3) (which may only be project.v0) resources I'm still not sure what the customization plan is. I think in the fullness of time we want to support a model like (1) where you can control the IaC at the C# layer (e.g. some sort of .WithContainerAppForDeployment(app => { /* party on a type from Azure.ResourceManager.AppContainers or something */}) and this just generates bicep (or the yaml rendering of the resource) and we deploy that.

In that world, I can see how we don't need this at all.

There's a world in the future where we can use the cdk to spit out main.bicep, but we're not there yet.

But I'm also not sure if we get the project stuff figured out for GA, so I'm not sure if we want to use this as a way to control metadata on the app itself in the short term.

Right, I think we want to spend a little bit of time thinking about modeling not just the container apps settings but also the compute environment in the generated resources.bicep (keyvaults + LA workspaces). Even if we don't make it for GA and we have a backup plan (using metadata).

It'll allow us to remove some of the magic inference that azd is doing today. Some of that inference is stuff that we can easily represent via some contract, the rest of it we would keep in azd because it's a little harder to describe.

I think the right way for us to do this is to have metadata on parameters that say "what type" you need and azd tries to wire up the right thing (creating a default one if needed). Then, over time, as more and more of these invisible resources are generated on the C# side of the house, you can just drop the parameters from the .bicep file you generate, and azd can use that as a signal that it need not generate these resources anymore (if nothing requests them).

I think more fine-grained metadata is needed or people will end up doing infra synth way more than expected. Originally I was thinking something like JSON patch via metadata but the more time I spent playing with various approaches, the one I liked best was something like this:

{
  "resources": {
    "storage": {
      "type": "azure.bicep.v0",
      "path": "aspire.hosting.azure.bicep.storage.bicep",
      "params": {
        "principalId": "{mid.outputs.principalId}",
        "principalType": "ServicePrincipal",
        "storageName": "storage"
      }
    },
    "containerAppEnviorment": {
      "type": "azure.bicep.v0",
      "path": "aspire.hosting.azure.bicep.containerAppEnviorment.bicep"
    },
    "blobs": {
      "type": "azure.bicep.v0",
      "connectionString": "{storage.outputs.blobEndpoint}",
      "parent": "storage"
    },
    "api-containerApp": {
      "type": "azure.bicep.v0",
      "path": "aspire.hosting.azure.bicep.containerApp.bicep",
      "params": {
        "containerAppName": "{api.name}",
        "containerAppImage": "{api.containerImage}",
        "containerAppPort": "8080",
        "containerRegistryName": "{containerAppEnviorment.outputs.containerRegistry}",
        "containerRegisteryIdentity" : "{containerAppEnviorment.outputs.containerRegistryIdentity}",
        "minReplicas": "1",
        "maxReplicas": "10",
        "userAssignedIdentities": ["{mid.outputs.principalId}", "{containerAppEnviorment.outputs.containerRegistryIdentity}"]
      }
    },
    "api": {
      "type": "project.v0",
      "path": "../WebApplication1/WebApplication1.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "ConnectionStrings__blobs": "{blobs.connectionString}",
        "AZURE_CLIENT_ID": "{containerAppEnviorment.outputs.clientId}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    }
  }
}

In the above containerAppEnviorment replaces resources.bicep, at least the core of it. To make this work we would need a contract about what this module looked like inputs and outputs wise. This feels achievable but I wouldn't want to make this standard until we better understood AKS (or some other compute environment). This would allow aspire to define the bicep for the compute environment and would allow users to customize it without dropping to infra synth.

The second part is less fleshed out, but there would be a container apps bicep file that references properties from the project itself. If you look at the "api-containerApp" resource, this is just a normal bicep resource that references some magic properties from the project (api.containerImage). In the above pattern, azd would understand the dependency graph here such that these bicep resources get pushed into the deploy phase (they depend on the project outputs).

I haven't figured out how to model secrets or urls in this world as yet. That part is still a bit magical. I feel if we iterated on this we could come up with something reasonable. The container app bicep becomes the way that you break glass to set whatever properties you want to and azd doesn't have to keep exposing properties piecemeal and users don't need to do infra synth to change settings.

My end goal would be that infra synth isn't used as an escape hatch, it's used to store things in source control. You never tweak the outputs, you always tweak the apphost and then see the what if via infra synth.

[DamianEdwards]

What are next steps to make progress on this in preview.5? For your last strawman proposal @davidfowl are you thinking this model requires the user to reference Aspire.Hosting.Azure? What happens to the default story? Does azd instruct the user to add Aspire.Hosting.Azure if the required resources are not in the manifest, or does azd keep some of its magic inference for the vanilla-app scenario?

[mitchdenny]

I think that this is probably a two step process. And we need to see what we can achieve by GA. I think the next click stop is to take over generating these resources but still generate the manifest and get AZD to effectively do the main.bicep piece.

I'd prefer to define these resources via CDK constructs and there are a few missing:

• AppInsights
• Log Workspaces
• Container Apps / Container Environments
• Container Registry

We need to finish the CDK constructs for the existing resource we have for P5. If we want more (like the ones above) we need tdo let the AzSDK team know ASAP as they might need to get more people working on it.

I think we could probably adapt the Azure Provisioner lifecycle hook that we have today to generate the extra resources in the App Model (or just create a new one that injects them on publish mode) - once we have the CDK constructs available).

[mitchdenny]

Oh and we'll need a mechanism to signal to AZD we are working in this mode. I would prefer that we don't break the existing approach until this approach is fully in and working. So maybe some kind of feature flag in AZD?

[davidfowl]

@vhvb1989 and I have been discussing how to take over resources.bicep.

[ellismg]

Oh and we'll need a mechanism to signal to AZD we are working in this mode. I would prefer that we don't break the existing approach until this approach is fully in and working. So maybe some kind of feature flag in AZD?

We can do feature flags - or we can figure out some way to reflect this in the manifest. Part of me is wondering if we should add an azure.aspireRuntime.containerApps.v0 resource or something to the manifest for cases where azd needs to generate the container app environment, acr and supporting infrastructure. If we had that, then it might be easier to signal when we are on the new plan (since it no longer shows up in the graph)? I know long term we wouldn't have this resource (and I suspect in the CDK world it's like an L2 or L3 construct that you add to the builder via a gesture like builder.UseAzureContainerAppsRuntime() (and then later there is a builder.UseAzureKurbernetesServiceRuntime() or something) which ends up just injecting some bicep.v0 resources - but perhaps having it be a first class thing in the manifest until then would be helpful? It would let us start to define the shape of the output contract perhaps?

The second part is less fleshed out, but there would be a container apps bicep file that references properties from the project itself. If you look at the "api-containerApp" resource, this is just a normal bicep resource that references some magic properties from the project (api.containerImage). In the above pattern, azd would understand the dependency graph here such that these bicep resources get pushed into the deploy phase (they depend on the project outputs).

We should consider tying together the project.v0 resource and the bicep that represents the "runtime configuration" somehow. Perhaps this bicep.v0 can have a parent that ties it to the project resource? azd could take that as a signal that it should not do generation?

[bradygaster]

A good end to end here would be the ability in the app host to add metadata to a project resource with keys like: azd.minInstances and azd.maxInstances which would control the min and max instance count in the generated container app.

Definitely a solid use-case.

[davidfowl]

@DamianEdwards

What are next steps to make progress on this in preview.5? For your last strawman proposal @davidfowl are you thinking this model requires the user to reference Aspire.Hosting.Azure? What happens to the default story? Does azd instruct the user to add Aspire.Hosting.Azure if the required resources are not in the manifest, or does azd keep some of its magic inference for the vanilla-app scenario?

I think it does require an explicit call to AddAzureProvisioning. I don't see a way around it. The user has to add something to the application to make it deploy or to customize it. We can keep azd's defaults (the 5 mins to wow experience) but any customization would require adding this package.

We can do feature flags - or we can figure out some way to reflect this in the manifest. Part of me is wondering if we should add an azure.aspireRuntime.containerApps.v0 resource or something to the manifest for cases where azd needs to generate the container app environment, acr and supporting infrastructure.

I like the idea but that resource would need to be added by an azure aware component. Which means it would signal that we are on the new plan not the existing plan. I think less resources in the manifest = azd does more magic.

I know long term we wouldn't have this resource (and I suspect in the CDK world it's like an L2 or L3 construct that you add to the builder via a gesture like builder.UseAzureContainerAppsRuntime() (and then later there is a builder.UseAzureKurbernetesServiceRuntime() or something) which ends up just injecting some bicep.v0 resources - but perhaps having it be a first class thing in the manifest until then would be helpful? It would let us start to define the shape of the output contract perhaps?

I think this is problematic unless we make up a manifest type that represents some abstract notion of a containerized compute environment (which I'm not sure how useful it is).

We should consider tying together the project.v0 resource and the bicep that represents the "runtime configuration" somehow. Perhaps this bicep.v0 can have a parent that ties it to the project resource? azd could take that as a signal that it should not do generation?

I like the dependency model. The project resource isn't the same as the container app resource, instead, "provisioning" the project gets you access to the published container image. That image property can then be referenced by other items in the manifest (like the container app bicep resource).

[DamianEdwards]

To add a strawman of what I'm currently thinking this starts to look like on the "new plan". I imagine we want something where the defaults can be built-on progressively, rather than a full escape-hatch type of experience where once you start customizing you have to customize everything.

E.g. using defaults for ACA environment but customizing the apps (making up lots of API):

var builder = DistributedApplication.CreateBuilder(args);

var cache = builder.AddRedis("cache")
    .PublishAsAzureContainerApp(app => app.MaxReplicas = 1);

var apiService = builder.AddProject<Projects.AspireAppWithHttps_ApiService>("apiservice")
    .PublishAsAzureContainerApp(app => app.MinReplicas = 0);

var webfrontend = builder.AddProject<Projects.AspireAppWithHttps_Web>("webfrontend")
    .WithReference(cache)
    .WithReference(apiService)
    .AsExternal();

webfrontend.PublishAsAzureContainerApp(app =>
    {
        app.MinReplicas = 0;
        app.ScaleRule = new("HttpScale100", httpConcurrency: 100)};
        // Can we make referring to endpoint details of the resource easier/automatic for things like this?
        app.HealthProbes.Add(new LivenessProbe("/alive", port: webfrontend.GetPrimaryEndpoint().Port));
        app.HealthProbes.Add(new ReadinessProbe("/health", port: webfrontend.GetPrimaryEndpoint().Port)));
    });;

builder.Build().Run();

E.g. when I want to start customizing the ACA environment itself:

var builder = DistributedApplication.CreateBuilder(args);

var acaEnv = builder.AddAzureContainerAppsEnvironment("myacaenv", env =>
    {
        // I still get the defaults for RG, LAW, etc. if they're not customized
        env.EnableWorkloadProfiles = true;
        env.WorkloadProfiles.Add(new ("HungryApps", WorkloadProfileType.DedicatedD16, minNodes: 0, maxNodes: 1));
    });

var cache = builder.AddRedis("cache")
    .PublishAsAzureContainerApp(acaEnv, app =>
    {
        app.MaxReplicas = 1;
        app.WorkloadProfile = "HungryApps";
    });

var apiService = builder.AddProject<Projects.AspireAppWithHttps_ApiService>("apiservice")
    .PublishAsAzureContainerApp(acaEnv, app => app.MinReplicas = 0);

var webfrontend = builder.AddProject<Projects.AspireAppWithHttps_Web>("webfrontend")
    .WithReference(cache)
    .WithReference(apiService)
    .AsExternal();

webfrontend.PublishAsAzureContainerApp(acaEnv, app =>
    {
        app.MinReplicas = 0;
        app.ScaleRule = new("HttpScale100", httpConcurrency: 100)};
        // Can we make referring to endpoint details of the resource easier/automatic for things like this?
        app.HealthProbes.Add(new LivenessProbe("/alive", port: webfrontend.GetPrimaryEndpoint().Port));
        app.HealthProbes.Add(new ReadinessProbe("/health", port: webfrontend.GetPrimaryEndpoint().Port)));
    });;

builder.Build().Run();

[ellismg]

I think this is problematic unless we make up a manifest type that represents some abstract notion of a containerized compute environment (which I'm not sure how useful it is).

I feel like there is an implicit input in the manifest today, for every project.v0 and dockerfile.v0 and container.v0 resource, which is "what compute environment do you want to run this in". azd today is both providing that environment (implicitly) and threading it to every project like resource so that it can do things like: "deploy this project to this environment". From that point of view an "environment" is basically a control plane type (e.g. aca and later k8s) + a container registry. The implementation of that environment is an implementation detail of whatever component creates it.

Perhaps we want something like this in the manifest:

"resources": {
   "runtime": {
      "type": "azure.aspire.containerApps.v0" /* this is new - it represents the container app environment + the acr to store the images, it is what azd creates implicitly today */
   },
   "api": {
      "type": "project.v0",
      "target": "runtime", /* this is new, and ties the project to the runtime it will be deployed to.
      /// other properties as before...
  }
}

Then, if you want to change how the environment is customized, you can express that with the parent trick:

"resources": {
   "runtime": {
      "type": "azure.aspire.containerApps.v0" /* this is new - it represents the container app environment + the acr to store the images, it is what azd creates implicitly today */
   },
   "api": {
      "type": "project.v0",
      "target": "runtime", /* this is new, and ties the project to the runtime it will be deployed to.
      /// other properties as before...
  },
  "runtimeCustomized": {
     "type": "bicep.v0",
     "path": "<path to the bicep file created by the CDK that creates an CAE and CR customized as the user pleases>"
  }
}

Now, azd does not create the infrastructure for the azure.aspire.containerApps.v0 type - instead the stuff in bicep.v0 replaces it. This module must have a few well-known output properties.

[ellismg]

Looking at @DamianEdwards examples - I sort of lose track about what APIs are types that are coming from the CDK and what are net new Aspire concepts (if any). I feel like this WorkloadsProfile stuff is net new - and something that would not be coming from CDK - but much of the app passed into PublishAsAzureContainerApp callback is just a normal L1 CDK type for the ContainerApp resource (and so the shape matches the shape of the control plane type). I think that's what we want to have happen and where the puck is headed, but I wanted to be explicit.

I feel like in CDK terms, the thing added by AddAzureContainerAppsEnvironment is really a L2 or L3 "component". I think the story of "I can customize some parts but others get the defaults" is a general concept that CDK would need an answer for (which is probably some combination of API design guidelines and set of primitives that allow you to do things like "register a default value for a resource of this type", but I don't have a clear enough understanding of how that works or what the CDK pushes folks towards for these sort of patterns.

[DamianEdwards]

Everything net new was intended as coming from the CDK. Workload profiles are an ACA concept so I would expect them to be types in the CDK. I was just making up API based on ACA-isms but the intention was that the app parameter was a type from the CDK.

[davidfowl]

If we bit the bullet and require a call in the app to add azure specific things (which I think we should just lean into at this point), then that would be the trigger for us to do more by default. Let's say this is the trigger to enable users to customize anything.

Now assuming we have this, there are 2 things we need to model:

• The compute environment (azd needs the image registry as an output)
• Each container app resource (which needs data from the project resource)

Given that azd understands the bicep resource in the manifest, I still think we should continue to lean into it as much as we can here to avoid creating new manifest types.

This is why I keep coming back to the idea of modeling this as more bicep inputs and a new outputs from the project resource to the bicep.

@DamianEdwards What you have is fine from an app model top level API perspective, but when we translate it to the manifest, I still think we should model container apps as separate resources from the projects. Unlike what you have with redis, we don't want to change the underlying resource type in the manifest for projects.

What if we emitted this:

{
  "resources": {
    "storage": {
      "type": "azure.bicep.v0",
      "path": "aspire.hosting.azure.bicep.storage.bicep",
      "params": {
        "principalId": "{mid.outputs.principalId}",
        "principalType": "ServicePrincipal",
        "storageName": "storage"
      }
    },
    "compute": {
      "type": "azure.bicep.v0",
      "path": "resources.bicep"
    },
    "blobs": {
      "type": "value.v0",
      "connectionString": "{storage.outputs.blobEndpoint}"
    },
    "api-containerApp": {
      "type": "azure.bicep.v0",
      "path": "aspire.hosting.azure.bicep.containerApp.bicep",
      "params": {
        "containerAppImage": "{api.containerImage}",
        "containerRegistryName": "{compute.outputs.containerRegistry}",
        "containerRegisteryIdentity" : "{compute.outputs.containerRegistryIdentity}",
        "userAssignedIdentities": ["{mid.outputs.principalId}", "{compute.outputs.containerRegistryIdentity}"],
         "env": {
            "ConnectionStrings__blobs": "{blobs.connectionString}",
            "AZURE_CLIENT_ID": "{compute.outputs.clientId}"
          }
      }
    },
    "api": {
      "type": "project.v0",
      "path": "../WebApplication1/WebApplication1.csproj"
    }
  }
}

If you see a bicep resource with the name "resources.bicep", that becomes the trigger to back off and use this as the replacement for what azd does today. If this is too hacky, we can introduce another way for azd to know (using a new resource type as a last resort). AZD needs to have a set of known outputs from this resource (the container apps domain, and the container apps registry).

I got stuck earlier because I was trying to figure out how much of the azd magic around secrets we would need to replicate in this mode, but now I'm thinking the bicep file for the container app would assume the same defaults as azd has today. We would just shift that logic into the generated bicep for each container app.

What I think that means is azd no longer:

• Maps environment variables to secrets
• Maps bindings to ACA ingress

In the above, I remove the env variables and bindings from the project since they are going to be used by the container app. Instead of the parent resource trick, I was thinking that instead we can build the dependency graph by using {projectresource.containerImage} to signal to azd that we want you to put the published container image into this value in the bicep for the container app. This also implicitly brings this bicep file into the deploy phase of the pipeline since it is dependent on the output of the project resource.

[davidfowl]

I made some big strides with this change dotnet/aspire#2965. The focus so far is modeling all of the resources in the manifest and stopping azd from generating resources.bicep (#3548).

A couple of big work items for azd (I spoke to @vhvb1989) about some of this:

1. We need to tell azd about the container registry (we may even be able to Feed two birds with one seed and support existing registries this way). Though it might make sense to completely externalize this config in azd itself.
2. azd should analyze the parameters of bicep resources and understand the {project.containerImage} property. This means that the bicep resource in this case needs to be moved to the deployment phase instead of the provision phase (this is probably the biggest work item)
3. Change how secret outputs work. Bicep resources have secret outputs and we magically associate a keyvault name with the resource. We will make this more explicit and express the keyvault resource in the manifest as well as describe the secret outputs keyvault association so that azd can properly pass secret output values as parameters.

[davidfowl]

Just FYI, we decided to punt this work to the first post GA release. There's enough going on that we will want to get our existing fundamentals right. With my spike I think we can prove out what is needed and missing in parallel.

[mip1983]

Looking forward to being able to do this stuff in AppHost and hopefully not needing any bicep customization at some point. I think the most significant things for me are the ability to set the price plan on the container apps environment and still support consumption only (it's currently forcing a workload profile unless you change the bicep output).

The other big one that can hopefully be tackled as part of this is the custom domain binding story (here). It would be great if you could go:

webfrontend.PublishAsAzureContainerApp(acaEnv, app =>
{
    app.CustomDomainBindingFromEnv("EnvironmentVariableContainingDomain", createManagedCertificate: true)
}

[scottroot-msft]

Same here, we spent a fair amount of time today looking for a means to specify Replicas in our AppHost. We saw the .WithReplicas(int) piece, and were hoping there was possibly also a .WithReplicas(min,max) option so we could scale down to 0 in ACA just from our AppHost code.

[davidfowl]

Linked issue is done in aspire dotnet/aspire#5470. The PR for azd is here #4286

[davidfowl]

Closing this out as #4286 was merged

[davidfowl]

We should file follow up issues about taking over container app environment customization.