GCP logs exporter - resource fields mapping

[wgebis]

Component(s)

exporter/googlecloud

What happened?

Hi there!

I'm using GCP logs exported for pushing long to GCP from on-premiss VM isntances. Log entries in GCP has an empty section:

resource: {
labels: {
  location: "global"
  namespace: ""
  node_id: ""
  project_id: "example"
}
type: "generic_node"
}```

Is it possible to override fields like `namespace` or `node_id` with OPEL Collector configuration?

Best,
W


### Collector version

v0.82

### Environment information

## Environment
OS: MacOS
Compiler(if manually compiled): binary from dockerhub


### OpenTelemetry Collector configuration

```yaml
receivers:
  filelog:
    include:
      - /logs/logs/**/*.log
    start_at: beginning
    attributes:
      source: "example"
    operators:
      - type: regex_parser
        regex: '(?s)^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) \[(?P<millis>[^\]]+)\] \[(?P<sev>[^\]]+)\] \[(?P<logger>[^\]]+)\]\s?(?P<msg>.*)$'
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d %H:%M:%S'
        severity:
          parse_from: attributes.sev
    multiline:
      line_start_pattern: '^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})'

exporters:
  googlecloud:
    log:
      default_log_name: "sample_log_name"

service:
  telemetry:
    logs:
      level: 'debug'
  pipelines:
    logs:
      receivers:
        - filelog
      exporters:
        - googlecloud

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/googlecloud: @aabmass @dashpole @jsuereth @punya @damemi @psx95

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Hello @wgebis, I just have some questions help me better direct you here.

Your goal is to fill in the resource attributes namespace and node_id (and maybe more), is that correct?

Which environment (resource) are you trying to get the resource information (like namespace and node_id) from? Is it from the logs themselves, or are you trying to get these values from the environment that the collector is running in? Or would this be something else I missed?

[wgebis]

Hi there!

Thanks for reply! I configure logging outside of GCP from external 3rd party k8s cluster. Finally I've figured it out by using resources attributes:

receivers:
  filelog/kit:
    resource:
      cloud.region: ${region}
      k8s.cluster.name: "name-k8s"
      k8s.namespace.name: ${namespace}
      k8s.container.name: ${deployment_name}
      k8s.pod.name: $${env:HOSTNAME}

Unfortunately I had to studied source code a bit: https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/blob/82615d06c71484903ae6bfe917f87d66e76c00fe/internal/resourcemapping/resourcemapping.go#L191

Here is implementation of Google Cloud Exporter monitored resource mappings. It could be great to get it in README somewhere.

Thanks!

[dashpole]

Thanks for reporting @wgebis. I'll make sure this gets documentd

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/googlecloud: @aabmass @dashpole @jsuereth @punya @damemi @psx95

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/googlecloud: @aabmass @dashpole @jsuereth @punya @damemi @psx95

See Adding Labels via Comments if you do not have permissions to add labels yourself.