TemplateServer: render secrets with Consul Template #7621

catsby · 2019-10-10T20:03:09Z

This pull request adds a package template to Vault Agent. The new type template.Server (aka "TemplateServer") accepts Consul Template style template configurations, and then constructs and manages the lifecycle of a Consul Template Runner that will communicate with a Vault server.

The TemplateServer operates in the same manner as SinkServer, receiving a token from AuthHandler. When a new token is received the TemplateServer updates it's internal Vault client with the token updates the internal runner. The runner is responsible for querying the Vault server and rendering the template.

The TemplateServer needs to manage the lifecycle of the runner and report any errors encountered.

I'm opening this pull request as a draft for early review, to get feedback on the implementation.

There are caveats:

Limited test coverage. More tests will come in future commits and/or separate pull requests
No configuration hot-reloads. Originally planned to include this but found that Vault Agent does not support this at all yet, so it will require more engineering affecting more of Agent rather than just reloading the templates.
Consul Template does not provide a way to supply a pre-configured Vault client. Originally had planned to just share the client that's already configured and used by AuthHandler but there's no easy way to do this. As a result, the setupTemplateConfig method was added to read Vault configuration values from the command, which is already setup with values from the environment. Those values are then passed on to Consul Template which builds it's own internal Vault client.

This pull request is opened against the topic branch f-vault-agent-template

…ucture

…vironment

* f-vault-agent-template: Add agent template doc

tyrannosaurus-becks

Looks like a good start! The handling of parallelism is very idiomatic with how I've seen it in other locations within Vault.

command/agent/config/config_test.go

command/agent.go

command/agent/template/template.go

command/agent.go

kalafut · 2019-10-11T16:00:45Z

command/agent.go

+// create it's own api client internally.
+// TODO test setupTemplateConfig
+func (c *AgentCommand) setupTemplateConfig() *config.Vault {
+	cfg := new(config.Vault)


I think since you're just checking for the zero value, all of this can be a single literal:

return &config.Vault { Address: c.flagAddress, CACert: c.flagCACert, ... }

command/agent/template/template.go

kalafut · 2019-10-11T16:38:22Z

command/agent/template/template.go

+	// Always set these to ensure nothing is picked up from the environment
+	emptyStr := ""
+	conf.Vault.RenewToken = boolPtr(false)
+	conf.Vault.Token = &emptyStr


A strPtr() function seems useful but for consistent and to break the pattern where all of these are being assigned the same pointer. e.g. if somewhere later I do *conf.Vault.Token = "abc", then all of the config items would be suddenly be "abc".

…ormal chan, as a better practice to enforce reading only

Co-Authored-By: Jim Kalafut <[email protected]>

catsby · 2019-10-14T18:42:04Z

Got the approval to merge into feature branch and open new PR against master 👍

* add template config parsing, but it's wrong b/c it's not using mapstructure * parsing consul templates in agent config * add additional test to configuration parsing, to cover basics * another test fixture, rework simple test into table * refactor into table test * rename test * remove flattenKeys and add other test fixture * add template package * WIP: add runner * fix panic, actually copy templates, etc * rework how the config.Vault is created and enable reading from the environment * this was supposed to be a part of the prior commit * move/add methods to testhelpers for converting some values to pointers * use new methods in testhelpers * add an unblock channel to block agent until a template has been rendered * add note * unblock if there are no templates * cleanups * go mod tidy * remove dead code * simple test to starT * add simple, empty templates test * Update package doc, error logs, and add missing close() on channel * update code comment to be clear what I'm referring to * have template.NewServer return a (<- chan) type, even though it's a normal chan, as a better practice to enforce reading only * Update command/agent.go Co-Authored-By: Jim Kalafut <[email protected]>

* Vault Agent Template: parse templates (#7540) * add template config parsing, but it's wrong b/c it's not using mapstructure * parsing consul templates in agent config * add additional test to configuration parsing, to cover basics * another test fixture, rework simple test into table * refactor into table test * rename test * remove flattenKeys and add other test fixture * Update command/agent/config/config.go Co-Authored-By: Jim Kalafut <[email protected]> * return the decode error instead of swallowing it * Update command/agent/config/config_test.go Co-Authored-By: Jim Kalafut <[email protected]> * go mod tidy * change error checking style * Add agent template doc * TemplateServer: render secrets with Consul Template (#7621) * add template config parsing, but it's wrong b/c it's not using mapstructure * parsing consul templates in agent config * add additional test to configuration parsing, to cover basics * another test fixture, rework simple test into table * refactor into table test * rename test * remove flattenKeys and add other test fixture * add template package * WIP: add runner * fix panic, actually copy templates, etc * rework how the config.Vault is created and enable reading from the environment * this was supposed to be a part of the prior commit * move/add methods to testhelpers for converting some values to pointers * use new methods in testhelpers * add an unblock channel to block agent until a template has been rendered * add note * unblock if there are no templates * cleanups * go mod tidy * remove dead code * simple test to starT * add simple, empty templates test * Update package doc, error logs, and add missing close() on channel * update code comment to be clear what I'm referring to * have template.NewServer return a (<- chan) type, even though it's a normal chan, as a better practice to enforce reading only * Update command/agent.go Co-Authored-By: Jim Kalafut <[email protected]> * update with test * Add README and doc.go to the command/agent directory (#7503) * Add README and doc.go to the command/agent directory * Add link to website * address feedback for agent.go * updated with feedback from Calvin * Rework template.Server to export the unblock channel, and remove it from the NewServer function * apply feedback from Nick * fix/restructure rendering test * Add pointerutil package for converting types to their pointers * Remove pointer helper methods; use sdk/helper/pointerutil instead * update newRunnerConfig to use pointerutil and empty strings * only wait for unblock if template server is initialized * drain the token channel in this test * conditionally send on channel

* Vault Agent Template: parse templates (#7540) * add template config parsing, but it's wrong b/c it's not using mapstructure * parsing consul templates in agent config * add additional test to configuration parsing, to cover basics * another test fixture, rework simple test into table * refactor into table test * rename test * remove flattenKeys and add other test fixture * Update command/agent/config/config.go Co-Authored-By: Jim Kalafut <[email protected]> * return the decode error instead of swallowing it * Update command/agent/config/config_test.go Co-Authored-By: Jim Kalafut <[email protected]> * go mod tidy * change error checking style * Add agent template doc * TemplateServer: render secrets with Consul Template (#7621) * add template config parsing, but it's wrong b/c it's not using mapstructure * parsing consul templates in agent config * add additional test to configuration parsing, to cover basics * another test fixture, rework simple test into table * refactor into table test * rename test * remove flattenKeys and add other test fixture * add template package * WIP: add runner * fix panic, actually copy templates, etc * rework how the config.Vault is created and enable reading from the environment * this was supposed to be a part of the prior commit * move/add methods to testhelpers for converting some values to pointers * use new methods in testhelpers * add an unblock channel to block agent until a template has been rendered * add note * unblock if there are no templates * cleanups * go mod tidy * remove dead code * simple test to starT * add simple, empty templates test * Update package doc, error logs, and add missing close() on channel * update code comment to be clear what I'm referring to * have template.NewServer return a (<- chan) type, even though it's a normal chan, as a better practice to enforce reading only * Update command/agent.go Co-Authored-By: Jim Kalafut <[email protected]> * update with test * Add README and doc.go to the command/agent directory (#7503) * Add README and doc.go to the command/agent directory * Add link to website * address feedback for agent.go * updated with feedback from Calvin * Rework template.Server to export the unblock channel, and remove it from the NewServer function * apply feedback from Nick * fix/restructure rendering test * Add pointerutil package for converting types to their pointers * Remove pointer helper methods; use sdk/helper/pointerutil instead * update newRunnerConfig to use pointerutil and empty strings * only wait for unblock if template server is initialized * update test structure * some test cleanup * follow up tests * remove debugging, fix issue in replacing runner config * need to handle first render/token * Simplify the blocking logic to support exit after auth * fix channel name * expand TestAgent_Template to include multiple scenarios * cleanup * test cleanups after feedback

catsby added 24 commits September 24, 2019 15:38

add template config parsing, but it's wrong b/c it's not using mapstr…

6b80d5d

…ucture

parsing consul templates in agent config

0fe3f61

add additional test to configuration parsing, to cover basics

5d8dcf9

another test fixture, rework simple test into table

a4bd223

refactor into table test

d024060

rename test

8933841

remove flattenKeys and add other test fixture

1213f30

add template package

651f37a

merge config branch

cb57c4b

WIP: add runner

2680f68

fix panic, actually copy templates, etc

8e15a5d

rework how the config.Vault is created and enable reading from the en…

fdbd1dd

…vironment

this was supposed to be a part of the prior commit

ff66aab

Merge branch 'f-vault-agent-template' into f-vat-template-server

c2d06a3

* f-vault-agent-template: Add agent template doc

move/add methods to testhelpers for converting some values to pointers

268d869

use new methods in testhelpers

6d165c4

add an unblock channel to block agent until a template has been rendered

7245fc7

add note

054f394

unblock if there are no templates

370b874

cleanups

459354e

go mod tidy

28a6a51

remove dead code

a1e3c83

simple test to starT

541a77a

add simple, empty templates test

fa85039

tyrannosaurus-becks reviewed Oct 10, 2019

View reviewed changes

Update package doc, error logs, and add missing close() on channel

14daed7

kalafut reviewed Oct 11, 2019

View reviewed changes

catsby and others added 3 commits October 11, 2019 12:08

update code comment to be clear what I'm referring to

c5720ee

have template.NewServer return a (<- chan) type, even though it's a n…

429285a

…ormal chan, as a better practice to enforce reading only

Update command/agent.go

19d1f74

Co-Authored-By: Jim Kalafut <[email protected]>

briankassouf added this to the 1.3 milestone Oct 14, 2019

catsby marked this pull request as ready for review October 14, 2019 18:19

catsby merged commit f235b32 into f-vault-agent-template Oct 14, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TemplateServer: render secrets with Consul Template #7621

TemplateServer: render secrets with Consul Template #7621

catsby commented Oct 10, 2019 •

edited

Loading

tyrannosaurus-becks left a comment

kalafut Oct 11, 2019

kalafut Oct 11, 2019

catsby commented Oct 14, 2019

TemplateServer: render secrets with Consul Template #7621

TemplateServer: render secrets with Consul Template #7621

Conversation

catsby commented Oct 10, 2019 • edited Loading

tyrannosaurus-becks left a comment

Choose a reason for hiding this comment

kalafut Oct 11, 2019

Choose a reason for hiding this comment

kalafut Oct 11, 2019

Choose a reason for hiding this comment

catsby commented Oct 14, 2019

catsby commented Oct 10, 2019 •

edited

Loading