Auditing in Stackdriver

[MaxDiOrio]

So I enabled logging, which is supposed to be done via stdout:

vault audit enable file file_path=stdout

I can kubectl logs the container and see the audit logs. But for some reason, I see absolutely nothing in Stackdriver relating to GKE logs. No Container logs at all, they're blank even though GKE was deployed with Stackdriver enabled and I can see the Fluentd container.

Any thoughts?

[minupla]

By default the permissions at the GCP level don't have permissions to write to stackdriver. Add the logs writer role and you should be fine. Fixed it for me at any rate.

[MaxDiOrio]

Strange - none of the IAM permissions in my other GKE cluster have or need logs writer role.

Would that get added to the Compute Engine default service account?

[MaxDiOrio]

For a cluster created "manually" through the GCloud UI, you can see the K8S logs:

For the Vault cluster created through the Terraform, nothing.

[minupla]

Ya, typically GCP compute engine default has editor, which inherits a lot of privs, but the vault-on-GKE rolls its own permissions, which didn't include the log writer one on my system at least.

[MaxDiOrio]

This isn't the case anymore then. GCP Compute Engine does have the Editor role, which does allow logging. There are no permissions that were rolled by the Terraform that I can see.

logging.exclusions.get
logging.exclusions.list
logging.logEntries.create
logging.logEntries.list
logging.logMetrics.create
logging.logMetrics.delete
logging.logMetrics.get
logging.logMetrics.list
logging.logMetrics.update
logging.logServiceIndexes.list
logging.logServices.list
logging.logs.delete
logging.logs.list
logging.sinks.get
logging.sinks.list
logging.usage.get

[minupla]

Ah on my system the nodes run as [email protected] not GCP compute engine in the vault project. Maybe you have a different issue then, sorry!

[MaxDiOrio]

No word on this issue from anyone? Vault is unusable in this case - can't get audit logs.

I asked Google about it, and they said it's not their problem since Terraform was used and another of our clusters correctly shows the Pod, Node and Container logs.

[brettcurtis]

If you're running Seth's code it does create it's own service account for the cluster and does not use the default SA like @minupla mentioned.
https://github.com/sethvargo/vault-on-gke/blob/master/terraform/gcp.tf#L23

and the nodes uses it here:
https://github.com/sethvargo/vault-on-gke/blob/master/terraform/gcp.tf#L132

So I'm in line with @minupla - this account does not have the correct roles. Did you change the code?

If not can you show the roles the vault-server service account has?

I don't use this code exactly how it is but i suspect you just need to add the correct role here:

https://github.com/sethvargo/vault-on-gke/blob/master/terraform/variables.tf#L29

[MaxDiOrio]

I feel like an idiot. That was exactly right. Somehow I completely missed the service account every time I looked at it. I'm chalking it up to being new with Google Cloud. Yeah, that's it.

Thanks.

[brettcurtis]

No problem man, there is a lot to learn with this stuff. Especially taking something as big as this and "clicking go" so to speak. Glad you are all sorted out.

[sethvargo]

Hey @MaxDiOrio - glad you were able to sort it out.

I asked Google about it, and they said it's not their problem since Terraform was used ...

Can you share more information about this? Who told you this and when?