Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[META 388] Proposal for collecting Azure App Service cloud metadata #365

Merged
merged 9 commits into from
Mar 11, 2021
Merged

[META 388] Proposal for collecting Azure App Service cloud metadata #365

Changes from 4 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
fc81ca6
Proposal for collecting Azure App Service cloud metadata
russcam Nov 11, 2020
e262bba
Update specs/agents/metadata.md
russcam Nov 23, 2020
2259cd4
Add optional note for Azure App Service cloud metadata
russcam Nov 25, 2020
e737a81
Merge branch 'azure-app-service-metadata' of github.com:russcam/apm i…
russcam Nov 25, 2020
d459a6e
Update specs/agents/metadata.md
russcam Feb 19, 2021
af4b6ea
Merge branch 'master' into azure-app-service-metadata
russcam Feb 19, 2021
9768039
Add gherkin specs for Azure App Metadata
russcam Feb 19, 2021
c86dce2
Merge branch 'master' into azure-app-service-metadata
russcam Mar 9, 2021
dd3b003
Update spec and link to gherkin specs
russcam Mar 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 105 additions & 0 deletions specs/agents/metadata.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,111 @@ metadata is available.
A sample implementation of this metadata collection is available in
[the Python agent](https://github.com/elastic/apm-agent-python/blob/master/elasticapm/utils/cloud.py).

#### AWS metadata

[Instance metadata about an EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html) can be retrieved from the internal metadata endpoint, `http://169.254.169.254`.

As an example with curl, first, an API token must be created

```sh
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 300"`
```

Then, metadata can be retrieved, passing the API token

```sh
curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data
```

From the returned metadata, the following fields are useful

| Cloud metadata field | AWS Metadata field |
| -------------------- | ------------------- |
| `account.id` | `accountId` |
| `instance.id` | `instanceId` |
| `availability_zone` | `availabilityZone` |
| `machine.type` | `instanceType` |
| `provider` | aws |
| `region` | `region` |

#### GCP metadata

Metadata about a GCP machine instance can be retrieved from the
metadata service, `http://metadata.google.internal`.

An example with curl

```sh
curl -X GET "http://metadata.google.internal/computeMetadata/v1/?recursive=true" -H "Metadata-Flavor: Google"
```

From the returned metadata, the following fields are useful

| Cloud metadata field | AWS Metadata field |
| -------------------- | ------------------- |
| `instance.id` | `instance.id` |
| `instance.name` | `instance.name` |
| `project.id` | `project.numericProjectId` as a string |
| `project.name` | `project.projectId` |
| `availability_zone` | last part of `instance.zone`, split by `/` |
| `machine.type` | last part of `instance.machineType`, split by `/` |
| `provider` | gcp |
| `region` | last part of `instance.zone`, split by `-` |

#### Azure metadata

##### Azure VMs

Metadata about an Azure VM can be retrieved from the internal metadata
endpoint, `http://169.254.169.254`.

An example with curl

```sh
curl -X GET "http://169.254.169.254/metadata/instance/compute?api-version=2019-08-15" -H "Metadata: true"
```

From the returned metadata, the following fields are useful

| Cloud metadata field | AWS Metadata field |
| -------------------- | ------------------- |
| `account.id` | `subscriptionId` |
| `instance.id` | `vmId` |
| `instance.name` | `name` |
| `project.name` | `resourceGroupName` |
| `availability_zone` | `zone` |
| `machine.type` | `vmSize` |
| `provider` | azure |
| `region` | `location` |

##### Azure App Services _(Optional)_

Azure App Services are a PaaS offering within Azure which does not
have access to the internal metadata endpoint. Metadata about
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What @axw said in an earlier comment:

What method should be used to identify that you're running within Azure App Service? Presence of valid WEBSITE_* environment variables?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The presence of WEBSITE_* environment variables and the format of WEBSITE_OWNER_NAME should be sufficient.

an App Service can however be retrieved from environment variables


| Cloud metadata field | Environment variable |
| -------------------- | ------------------- |
| `account.id` | first part of `WEBSITE_OWNER_NAME`, split by `+` |
| `instance.id` | `WEBSITE_INSTANCE_ID` |
| `instance.name` | `WEBSITE_SITE_NAME` |
| `project.name` | `WEBSITE_RESOURCE_GROUP` |
| `provider` | azure |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there value in capturing the cloud product, perhaps cloud.product: azure-app-service as well? It appears we'd need a new cloud ecs field.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ I think there could be. cloud fields look to be very much intended for VMs e.g. availability_zone, machine_type.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With elastic/ecs#1204 we will have an ECS field for this as of next release - cloud.service.name. elastic/apm-server#4625 is tracking addition to the intake for this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for somehow capturing the fact that this set of metadata is being captured in an azure-app-services environment in order to make it distinct from data collected via the instance endpoints. I was thinking similar thoughts when I investigated this for node.js recently specifically around a theoretical problem of a user looking at two sets of azure data and wondering why different fields are collected from each.

| `region` | last part of `WEBSITE_OWNER_NAME`, split by `-`, trim end `"webspace"` |

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@russcam datapoint: When I deployed a node application via azure app services my region looked like

    "WEBSITE_RESOURCE_GROUP": "appsvc_linux_centralus",

i.e. the WEBSITE_OWNER_NAME was delimited with underscores characters (_) and not the hyphen/dash. You can see this here (will be slow in coming up if the app's gone idle)

What do you think about spec-ing this as splitting by - OR _ -- or perhaps even "splitting by non-alpha-numeric-characters"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the additional data point. Looks like there may be some cross-language and cross-platform differences in environment variable values. For example, the node app value is

"WEBSITE_OWNER_NAME": "7657426d-c4c3-44ac-88a2-3b2cd59e6dba+appsvc_linux_centralus-CentralUSwebspace-Linux"

The extracted region should be CentralUS, though the value includes a -Linux suffix which wasn't present on the .NET app service.

I think what might be good for me to do before updating the PR is to deploy some other language app services to see if there is more variation in values. It might be a simple case of updating the WEBSITE_OWNER_NAME rule to substring from 0 up to start of webspace, then split by - and take the last value.

The environment variable `WEBSITE_OWNER_NAME` has the form

```
{subscription id}+{app service plan resource group}-{region}webspace
```

an example of which is `f5940f10-2e30-3e4d-a259-63451ba6dae4+elastic-apm-AustraliaEastwebspace`

Cloud metadata for Azure App Services is optional; it is up
to each agent to determine whether it is useful to implement
for their language ecosystem.

### Global labels

Events sent by the agents can have labels associated, which may be useful for custom aggregations, or document-level access control. It is possible to add "global labels" to the metadata, which are labels that will be applied to all events sent by an agent. These are only understood by APM Server 7.2 or greater.
Expand Down