Investigate jet-aws #219 security bug

[muvaf]

What happened?

crossplane-contrib/provider-jet-aws#219 is opened in upstream and we need to know whether that bug exists in Upjet as well.

How can we reproduce it?

Try to reproduce the bug for that specific resource and if it exists, fix it ASAP.

[donovanmuller]

This also appears to be an issue in provider-gcp where the private key is leaked in the debug logs (failed plan's main.tf.json output). Example below:

1.659351665068625e+09	DEBUG	provider-gcp	apply async ended	{"workspace": "/var/folders/ht/n9hkq1z555vdwmbb10nfqjjh0000gn/T/93acf211-398c-4d0c-b100-b4197774204b", "out": "{\"@level\":\"info\",\"@message\":\"Terraform 1.2.5\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-08-01T13:01:02.423937+02:00\",\"terraform\":\"1.2.5\",\"type\":\"version\",\"ui\":\"1.0\"}\n{\"@level\":\"info\",\"@message\":\"google_container_node_pool.platform-example: Plan to create\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-08-01T13:01:03.819407+02:00\",\"change\":{\"resource\":{\"addr\":\"google_container_node_pool.platform-example\",\"module\":\"\",\"resource\":\"google_container_node_pool.platform-example\",\"implied_provider\":\"google\",\"resource_type\":\"google_container_node_pool\",\"resource_name\":\"platform-example\",\"resource_key\":null},\"action\":\"create\"},\"type\":\"planned_change\"}\n{\"@level\":\"info\",\"@message\":\"Plan: 1 to add, 0 to change, 0 to destroy.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-08-01T13:01:03.819454+02:00\",\"changes\":{\"add\":1,\"change\":0,\"remove\":0,\"operation\":\"plan\"},\"type\":\"change_summary\"}\n{\"@level\":\"info\",\"@message\":\"google_container_node_pool.platform-example: Creating...\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-08-01T13:01:04.962099+02:00\",\"hook\":{\"resource\":{\"addr\":\"google_container_node_pool.platform-example\",\"module\":\"\",\"resource\":\"google_container_node_pool.platform-example\",\"implied_provider\":\"google\",\"resource_type\":\"google_container_node_pool\",\"resource_name\":\"platform-example\",\"resource_key\":null},\"action\":\"create\"},\"type\":\"apply_start\"}\n{\"@level\":\"info\",\"@message\":\"google_container_node_pool.platform-example: Creation errored after 0s\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-08-01T13:01:04.967149+02:00\",\"hook\":{\"resource\":{\"addr\":\"google_container_node_pool.platform-example\",\"module\":\"\",\"resource\":\"google_container_node_pool.platform-example\",\"implied_provider\":\"google\",\"resource_type\":\"google_container_node_pool\",\"resource_name\":\"platform-example\",\"resource_key\":null},\"action\":\"create\",\"elapsed_seconds\":0},\"type\":\"apply_errored\"}\n{\"@level\":\"error\",\"@message\":\"Error: Cannot set both initial_node_count and node_count on node pool platform-example\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-08-01T13:01:05.033883+02:00\",\"diagnostic\":{\"severity\":\"error\",\"summary\":\"Cannot set both initial_node_count and node_count on node pool platform-example\",\"detail\":\"\",\"address\":\"google_container_node_pool.platform-example\",\"range\":{\"filename\":\"main.tf.json\",\"start\":{\"line\":1,\"column\":3561,\"byte\":3560},\"end\":{\"line\":1,\"column\":3562,\"byte\":3561}},\"snippet\":{\"context\":\"resource.google_container_node_pool.platform-example\",\"code\":\"{\\\"provider\\\":{\\\"google\\\":{\\\"credentials\\\":\\\"{\\\\n  \\\\\\\"type\\\\\\\": \\\\\\\"service_account\\\\\\\",\\\\n  \\\\\\\"project_id\\\\\\\": \\\\\\\"crossplane-playground\\\\\\\",\\\\n  \\\\\\\"private_key_id\\\\\\\": \\\\\\\"01e20c6c3a5b97f8355e9be0bab4a72eede2abf7\\\\\\\",\\\\n  \\\\\\\"private_key\\\\\\\": \\\\\\\"-----BEGIN PRIVATE KEY-----\\\\\\\\nMIIEuwIBADANBgkqhkiG9w0BAQEFAASCBKUwggShAgEAAoIBAQC6zecV799yKio/\\\\\\\\nT0tjXNy6JsgxCLPmSvgBTOa...\n-----END PRIVATE KEY-----\\\\\\\\n\\\\\\\",\\\\n  \\\\\\\"client_email\\\\\\\": \\\\\\\"[email protected]\\\\\\\",\\\\n  \\\\\\\"client_id\\\\\\\": \\\\\\\"118411762237969717491\\\\\\\",\\\\n  \\\\\\\"auth_uri\\\\\\\": \\\\\\\"https://accounts.google.com/o/oauth2/auth\\\\\\\",\\\\n  \\\\\\\"token_uri\\\\\\\": \\\\\\\"https://oauth2.googleapis.com/token\\\\\\\",\\\\n  \\\\\\\"auth_provider_x509_cert_url\\\\\\\": \\\\\\\"https://www.googleapis.com/oauth2/v1/certs\\\\\\\",\\\\n  \\\\\\\"client_x509_cert_url\\\\\\\": \\\\\\\"https://www.googleapis.com/robot/v1/metadata/x509/donovan-dev%40crossplane-playground.iam.gserviceaccount.com\\\\\\\"\\\\n}\\\\n\\\",\\\"project\\\":\\\"crossplane-playground\\\"}},\\\"resource\\\":{\\\"google_container_node_pool\\\":{\\\"platform-example\\\":{\\\"autoscaling\\\":[{\\\"max_node_count\\\":1,\\\"min_node_count\\\":1}],\\\"cluster\\\":\\\"projects/crossplane-playground/locations/us-west1-c/clusters/platform-example\\\",\\\"initial_node_count\\\":1,\\\"lifecycle\\\":{\\\"prevent_destroy\\\":true},\\\"management\\\":[{\\\"auto_repair\\\":true,\\\"auto_upgrade\\\":false}],\\\"max_pods_per_node\\\":110,\\\"name\\\":\\\"platform-example\\\",\\\"node_config\\\":[{\\\"disk_size_gb\\\":100,\\\"disk_type\\\":\\\"pd-standard\\\",\\\"image_type\\\":\\\"COS_CONTAINERD\\\",\\\"machine_type\\\":\\\"e2-standard-4\\\",\\\"metadata\\\":{\\\"disable-legacy-endpoints\\\":\\\"true\\\"},\\\"oauth_scopes\\\":[\\\"https://www.googleapis.com/auth/logging.write\\\",\\\"https://www.googleapis.com/auth/monitoring\\\",\\\"https://www.googleapis.com/auth/cloud-platform\\\",\\\"https://www.googleapis.com/auth/cloud_debugger\\\",\\\"https://www.googleapis.com/auth/trace.append\\\",\\\"https://www.googleapis.com/auth/devstorage.read_only\\\"],\\\"preemptible\\\":false,\\\"shielded_instance_config\\\":[{\\\"enable_integrity_monitoring\\\":true}],\\\"workload_metadata_config\\\":[{\\\"mode\\\":\\\"GKE_METADATA\\\"}]}],\\\"node_count\\\":1,\\\"node_locations\\\":[\\\"us-west1-c\\\"],\\\"version\\\":\\\"1.22.8-gke.201\\\"}}},\\\"terraform\\\":{\\\"required_providers\\\":{\\\"google\\\":{\\\"source\\\":\\\"hashicorp/google\\\",\\\"version\\\":\\\"4.22.0\\\"}}}}\",\"start_line\":1,\"highlight_start_offset\":3560,\"highlight_end_offset\":3561,\"values\":[]}},\"type\":\"diagnostic\"}\n"}

[luebken]

The solution should fix it for all for all OPs. So to be fixed in Upjet & Terrajet.

[sergenyalcin]

I am investigating this issue. Thank you @donovanmuller for your example in provider-gcp.

Now, I realized that we have not any filtering mechanism while dumping the debug logs:

https://github.com/upbound/upjet/blob/d108c3986da8a1e0a7a081f027df35e9c43d2f1b/pkg/terraform/workspace.go#L225

So, we see all credentials of providers in logs. I think, we need a filtering process before printing these debug logs. I want to suggest two solutions and I want to hear your ideas. Firstly I want to share an example main.tf.json (as references) that is dumped in logs.

{
  "provider": {
    "aws": {
      "access_key": "***", (I hide this field here because of privacy but in debug logs we see the value)
      "region": "us-east-1",
      "secret_key": "***", (I hide this field here because of privacy but in debug logs we see the value)
      "skip_region_validation": true,
      "token": ""
    }
  },
  "resource": {
    "aws_iam_user": {
      "sample-user": {
        "lifecycle": {
          "prevent_destroy": true
        },
        "name": "sample-user",
        "tags": {
          "crossplane-kind": "user.iam.aws.upbound.io",
          "crossplane-name": "sample-user",
          "crossplane-providerconfig": "default"
        }
      }
    }
  },
  "terraform": {
    "required_providers": {
      "aws": {
        "source": "hashicorp/aws",
        "version": "4.15.1"
      }
    }
  }
}

• The provider credentials are located in provider block in json. An option for filtering this credentials can be not printing this provider block in logs.
• The another solution can be that removing the specified keys in provider block (such as access_key and secret_key for provider-aws) instead of removing all block. But my concern about this solution is that these specific field change for each provider. So we need to determine them and maybe try to implement provider specific filtering mechanisms. And also, in future, as new providers are added, it will be necessary to make new maintenance for these providers.

Today I want to open a PR that implements the first approach. Let's discuss here or PR!