Added PSC Private Service Connect for GCP CloudSQL #27889

shinji62 · 2024-07-29T08:32:47Z

Added PrivateIP support for GCP MySQL

Description

This PR add Private Service Connect for CloudSQL (MySQL and PostgresSQL), this PR also add PrivateIP support for MySQL, previous PR( #26827) just added to Postgres.

Close #27867

TODO only if you're a HashiCorp employee

[N/A] Labels: If this PR is the CE portion of an ENT change, and that ENT change is
getting backported to N-2, use the new style backport/ent/x.x.x+ent labels
instead of the old style backport/x.x.x labels.
-[N/A] Labels: If this PR is a CE only change, it can only be backported to N, so use
the normal backport/x.x.x label (there should be only 1).
[N/A] ENT Breakage: If this PR either 1) removes a public function OR 2) changes the signature
of a public function, even if that change is in a CE file, double check that
applying the patch for this PR to the ENT repo and running tests doesn't
break any tests. Sometimes ENT only tests rely on public functions in CE
files.
[N/A] Jira: If this change has an associated Jira, it's referenced either
in the PR description, commit message, or branch name.
[N/A] RFC: If this change has an associated RFC, please link it in the description.
[N/A] ENT PR: If this change has an associated ENT PR, please link it in the
description. Also, make sure the changelog is in this PR, not in your ENT PR.

Added PrivateIP support for GCP MySQL

hashicorp-cla-app · 2024-07-29T08:33:00Z

All committers have signed the CLA.

shinji62 · 2024-08-01T04:15:02Z

@fairclothjm Any chance something from Hashicorp can take a look ? Thanks

schavis

Content portion lgtm

plugins/database/mysql/connection_producer.go

sdk/database/helper/connutil/sql.go

changelog/27889.txt

fairclothjm · 2024-08-14T14:29:25Z

plugins/database/mysql/connection_producer.go

-	usePrivateIP             bool        `json:"use_private_ip" mapstructure:"use_private_ip" structs:"use_private_ip"`
-	usePSC                   bool        `json:"use_psc" mapstructure:"use_psc" structs:"use_psc"`


Do we still want these fields for mysql?

Yes, I havee tested them as well.

shinji62 · 2024-08-15T03:39:45Z

So I did a few test across 6 databases, MySQL and Postgresql with PSC or PrivateIP or default settings.
All of them pass the ping test and failed when need to failed

For example for Postgresql with PSC

    vault write database/config/my-postgresql-database-psc \
      plugin_name="postgresql-database-plugin" \
      allowed_roles="my-role" \
     connection_url="host=sej-tools-hashicorp:asia-northeast1:psc-enabled-main-instance [email protected] dbname=postgres sslmode=disable" \
    auth_type="gcp_iam" \
    use_psc=true
    Success! Data written to: database/config/my-postgresql-database-psc

when trying Prrivate Ip for the PSC enable instance

    vault write database/config/my-postgresql-database-psc     \
        plugin_name="postgresql-database-plugin"     \
       allowed_roles="my-role"     \
       connection_url="host=test-hashicorp:asia-northeast1:psc-enabled-main-instance [email protected] dbname=postgres sslmode=disable"     \
       auth_type="gcp_iam"     \
       use_private_ip=true

      Error writing data to database/config/my-postgresql-database-psc: Error making API request.
    
    URL: PUT http://127.0.0.1:8200/v1/database/config/my-postgresql-database-psc
    Code: 400. Errors:
    
    * error creating database object: error verifying connection: ping failed: failed to connect to `host=localhost [email protected] database=postgres`: dial error (Config error: instance does not have IP of type "PRIVATE" (connection name = "test-hashicorp:asia-northeast1:psc-enabled-main-instance"))

you can find the repos to setup DB and related infra in GCP : https://github.com/shinji62/vault-tf-psc-test

	`default`	`use_private_ip`	`use_psc`
default postgres	✅	❌	❌
postgres with psc	❌	❌	✅
postgres with privateip	❌	✅	❌
default mysql	✅	❌	❌
mysql with psc	❌	❌	✅
mysql with private ip	❌	✅	❌

fairclothjm · 2024-08-15T20:25:06Z

sdk/database/helper/connutil/sql.go

+	UsePrivateIP             bool        `json:"use_private_ip" mapstructure:"use_private_ip" structs:"use_private_ip"`
+	UsePSC                   bool        `json:"use_psc" mapstructure:"use_psc" structs:"use_psc"`


These will be returned in the READ response for all database plugins that embed this SQLConnectionProducer struct. That is not ideal, since they are only relevant to a small subset of database plugins. It might be better to implement this per-plugin similar to mysql. Maybe there is some common code that could be reused by each plugin?

But this is already the case for

AuthType string `json:"auth_type" mapstructure:"auth_type" structs:"auth_type"` ServiceAccountJSON string `json:"service_account_json" mapstructure:"service_account_json" structs:"service_account_json"`

Which is only supported for Postgresql, but still field is available for hana, mssql, redshift

Oh actually I was able to remove from the read path, when auth_type is not gcp_iam

Test (did the same test with mysql as well)
Postgresql
Write and read using auth_typegcp_iam

vault write database/config/my-postgresql-database-psc \ plugin_name="postgresql-database-plugin" \ allowed_roles="my-role" \ connection_url="host=sej-tools-hashicorp:asia-northeast1:psc-enabled-main-instance [email protected] dbname=postgres sslmode=disable" \ auth_type="gcp_iam" use_psc=true Success! Data written to: database/config/my-postgresql-database-psc vault read database/config/my-postgresql-database-psc Key Value --- ----- allowed_roles [my-role] connection_details map[auth_type:gcp_iam connection_url:host=sej-tools-hashicorp:asia-northeast1:psc-enabled-main-instance [email protected] dbname=postgres sslmode=disable use_psc:true] password_policy n/a plugin_name postgresql-database-plugin plugin_version n/a root_credentials_rotate_statements []

Write and read using normal plugin and normal auth

vault write database/config/my-postgresql-database-psc-ip \ plugin_name=postgresql-database-plugin \ connection_url="postgresql://{{username}}:{{password}}@127.0.0.1/postgres?sslmode=disable" \ allowed_roles=readonly \ username="admin" \ password="XXXXXx" Success! Data written to: database/config/my-postgresql-database-psc-ip vault read database/config/my-postgresql-database-psc-ip Key Value --- ----- allowed_roles [readonly] connection_details map[connection_url:postgresql://{{username}}:{{password}}@127.0.0.1/postgres?sslmode=disable username:admin] password_policy n/a plugin_name postgresql-database-plugin plugin_version n/a root_credentials_rotate_statements []

I was wrong about this. See my comment here https://github.com/hashicorp/vault/pull/27889/files#r1720329947

fairclothjm · 2024-08-16T20:43:40Z

builtin/logical/database/path_config_connection.go

+		// If the auth_type is not gcp_iam we remove the field from the read response
+
+		if authType, ok := config.ConnectionDetails["auth_type"]; ok {
+			if authType != connutil.AuthTypeGCPIAM {
+				delete(config.ConnectionDetails, "use_private_ip")
+				delete(config.ConnectionDetails, "use_psc")
+			}
+		}


My apologies! This isn't actually needed after all. These won't be returned in the response because connection details will only contain what was set in the write request.

I just reverted my commit.

DrFaust92 · 2025-01-08T21:48:12Z

can this get some attention? we would love to get this released and supported

Added PSC Private Service Connect for GCP CloudSQL

49d9118

Added PrivateIP support for GCP MySQL

shinji62 requested review from a team as code owners July 29, 2024 08:32

shinji62 added 3 commits July 29, 2024 17:35

Added changelog

b90ab8d

Merge branch 'main' into feature/add_psc_google_database

acac415

Merge branch 'main' into feature/add_psc_google_database

c5fdb98

heatherezell added the hashicorp-contributed-pr If the PR is HashiCorp (i.e. not-community) contributed label Aug 5, 2024

schavis previously approved these changes Aug 6, 2024

View reviewed changes

schavis added the content-lgtm Content changes approved. Merge depends on code review label Aug 6, 2024

Merge branch 'main' into feature/add_psc_google_database

6743547

fairclothjm reviewed Aug 13, 2024

View reviewed changes

plugins/database/mysql/connection_producer.go Outdated Show resolved Hide resolved

sdk/database/helper/connutil/sql.go Outdated Show resolved Hide resolved

changelog/27889.txt Outdated Show resolved Hide resolved

shinji62 added 4 commits August 14, 2024 09:00

Merge branch 'main' into feature/add_psc_google_database

1c1e6ae

Update changelog

b6f8f77

Value need to be exported or will be false

4b26678

Exported variablee for MySQL as well

d862d0b

fairclothjm reviewed Aug 14, 2024

View reviewed changes

fairclothjm reviewed Aug 15, 2024

View reviewed changes

fairclothjm reviewed Aug 16, 2024

View reviewed changes

shinji62 force-pushed the feature/add_psc_google_database branch from a95ca68 to d862d0b Compare August 19, 2024 00:08

shinji62 added 3 commits August 30, 2024 09:35

Merge branch 'main' into feature/add_psc_google_database

1cc6299

Merge branch 'main' into feature/add_psc_google_database

9bdcb44

Merge branch 'main' into feature/add_psc_google_database

b377760

shinji62 dismissed schavis’s stale review via b377760 December 2, 2024 23:57

Merge branch 'main' into feature/add_psc_google_database

16b564d

shinji62 requested a review from a team as a code owner December 4, 2024 01:03

shinji62 requested a review from elliesterner December 4, 2024 01:03

Merge branch 'main' into feature/add_psc_google_database

f3c8579

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added PSC Private Service Connect for GCP CloudSQL #27889

Added PSC Private Service Connect for GCP CloudSQL #27889

shinji62 commented Jul 29, 2024

hashicorp-cla-app bot commented Jul 29, 2024 •

edited

Loading

shinji62 commented Aug 1, 2024

schavis left a comment

fairclothjm Aug 14, 2024

shinji62 Aug 15, 2024

shinji62 commented Aug 15, 2024

fairclothjm Aug 15, 2024

shinji62 Aug 16, 2024

shinji62 Aug 16, 2024

fairclothjm Aug 16, 2024

fairclothjm Aug 16, 2024

shinji62 Aug 19, 2024

DrFaust92 commented Jan 8, 2025

		usePrivateIP bool `json:"use_private_ip" mapstructure:"use_private_ip" structs:"use_private_ip"`
		usePSC bool `json:"use_psc" mapstructure:"use_psc" structs:"use_psc"`

Added PSC Private Service Connect for GCP CloudSQL #27889

Are you sure you want to change the base?

Added PSC Private Service Connect for GCP CloudSQL #27889

Conversation

shinji62 commented Jul 29, 2024

Description

TODO only if you're a HashiCorp employee

hashicorp-cla-app bot commented Jul 29, 2024 • edited Loading

shinji62 commented Aug 1, 2024

schavis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shinji62 commented Aug 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DrFaust92 commented Jan 8, 2025

hashicorp-cla-app bot commented Jul 29, 2024 •

edited

Loading