Add go-sockaddr functions

[jen20]

This is an issues to discuss the acceptability and design for a large feature which I think would be useful in gomplate - the ability to execute go-sockaddr pipelines.

The go-sockaddr library was written primarily by @sean- for use in HashiCorp runtime tools to allow for evaluation of network interfaces against heuristics to avoid hard-coding interface names, IP addresses and so forth during configuration. Consul and Nomad (and I believe soon Vault) all support the templates in configuration fields such as bind.

Templates take the form of a Go function pipeline, and provide functionality such as finding an IP address that:

• is attached to a default route (GetDefaultInterfaces())
• is contained within a CIDR block (IfByNetwork())
• is an RFC1918 address (IfByRFC("1918"))
• is ordered (OrderedIfAddrBy(args) where args includes AscIfType, AscNetworkSize)
• excludes all IPv6 addresses (IfByType("^(IPv4)$"))
• is larger than a /32 (IfByMaskSize(32))
• is not on a down interface (ExcludeIfs("flags", "down"))
• preferences an IPv6 address over an IPv4 address (SortIfByType() + ReverseIfAddrs()); and
• excludes any IP in RFC6890 address (IfByRFC("6890"))

I believe this would be a fairly straightforward lift-and-shift to support all of the functions supported by go-sockaddr in gomplate, and it would make rendering templates with networking aspects more flexible. Two possible approaches seem to exist:

1. Add the go-sockaddr-supplied functions to gomplate directly
2. Add a function named something like sockaddr which takes a template to render as a parameter.

I'm happy to take a look at doing this work, but want to confirm that the feature set is of interest prior to doing so.

[hairyhenderson]

Thanks @jen20 - on the surface, this looks like a great feature. Hadn't run into go-sockaddr until now, and I know I've run into situations that I'm sure would've been more elegantly solved with go-sockaddr. So yes, I'm interested in this.

Since this seems like it'd pull in a huge number of new functions, I want to make sure to design this in a way that'll make sense and won't paint us into a corner later.

Two possible approaches seem to exist:

1. Add the go-sockaddr-supplied functions to gomplate directly
2. Add a function named something like sockaddr which takes a template to render as a parameter.

On the face of it, option 1. seems like the most natural approach to me, because it would follow the existing convention of every function being in a big flat namespace. But I'm not married to that. This might also be the best option from a template author's perspective, since they could just have {{ getPrivateIp }} rather than {{ sockaddr "{{ GetPrivateIp }}" }} (unless you were thinking of a different usage?).

Documenting all these new functions would be a challenge -- it'll break my non-scalable shove-everything-in-README scheme, but that's probably something needing solving anyway. We could just point to https://godoc.org/github.com/hashicorp/go-sockaddr/template, but I'm not sure how consumable that'd be for the average gomplate user (though, I have no clue how Go-savvy the average user is to begin with...).

[jen20]

@hairyhenderson Thanks for the feedback. Hopefully I can pull in @sean- here for some further opinions.

A third option might be to introduce function name spacing, since many of the sockaddr functions are fairly generic. This could be done with a simple dot notation (and retrofitted to some other functions over time), so we'd end up eventually with:

• sockaddr.getPrivateInterfaces
• sockaddr.join
• aws.ec2tag
• triton.metadata (future?)

Leaving the string, array and list handling at the top level, unnamespaced level would be sensible however.

And so forth. As people inevitably find this program useful I can foresee them desiring cloud platforms other than EC2 to be added - Google Cloud is an obvious choice. A namespacing approach should prevent the top level becoming cluttered, and makes it obvious what each function will operate on without naming them in an obtuse manner.

With regards to documentation, I wonder if it is time to consider moving away from README documentation to something like a static site or ReadTheDocs - this is worth considering outside of the scope of these changes even? My guess is that relying on godoc will be insufficient for at least some future users.

[sean-]

Hello. A few things come to mind:

1. Have you thought about breaking gomplate into a sub-package like github.com/hairyhenderson/gomplate/cmd/gomplate so that the "gomplate" template library could be the meta-library used to aggregate a bunch of different APIs and helper functions? It's pretty easy for me to see that this would be useful to embed in standalone programs. "Dear gomplate, please render my string." Either gomplate(1) or the library github.com/hairyhenderson/gomplate.
2. go-sockaddr was written so that it could be easily used natively via the library github.com/hashicorp/go-sockaddr, have its templating functions cribbed or reused elsewhere by cribbing the helper functions in github.com/hashicorp/go-sockaddr/template, or as a standalone binary github.com/hashicorp/go-sockaddr/cmd/sockaddr that could eval strings. In the case of gomplate subsuming the sockaddr interface, I think all that is necessary is to inject a new template interface by adding the necessary functions to text/template before you create the context: https://github.com/hashicorp/go-sockaddr/blob/master/template/template.go#L28-L84 I think it should be extremely simple.
3. I had a huge lesson learned during go-sockaddr that I haven't acted on: notably that the template functions in go-sockaddr should not be strict wrappers on their lower-level library calls, instead I should type-assert in the template library with a wrapper function and then pass the value down to the lower-level library call. I started doing this in the yet-to-be-merged PR https://github.com/hashicorp/go-sockaddr/pull/11/files#diff-cbe4d12dc3bd0fa2adc428fafbfd4661R104 and realized I should be doing this everywhere so that I can have users submit naked numbers or strings and have everything "just work."
4. As function namespacing goes, use sockaddr. as the prefix (or sockaddr:). For example: sockaddr:GetPrivateInterface | sockaddr.Attr "address". Either looks reasonable to me.
5. If you added a "namespace search path" to gomplate, you could preserve your existing semantics for a while. Something like namespaces "env,sockaddr" if you want to have all of the env and sockaddr methods available by default without having to fully-qualify the function within a package.

Let me know what I/we can do to help!

[hairyhenderson]

Thanks for the thoughtful replies, @jen20 and @sean-. I really appreciate it!

First up, I'll address @jen20's comments:

A third option might be to introduce function name spacing, since many of the sockaddr functions are fairly generic. This could be done with a simple dot notation (and retrofitted to some other functions over time)

Yup - I agree. I'd started writing something to that effect but I couldn't really wrap my brain around it late last night. My initial concern was that I would like all of the functions to be eventually namespaced, but compatibility would be an issue. I'm not sure why I didn't think of it last night, but the obvious answer is to transition relatively slowly and support both versions (i.e. support both aws.ec2region and ec2region for a while).

There's already a bit of a namespace going on code-wise with a couple packages (vault and aws), but some shuffling should probably happen for other functions.

As people inevitably find this program useful I can foresee them desiring cloud platforms other than EC2 to be added - Google Cloud is an obvious choice.

Absolutely. I use mostly AWS these days so I haven't had any real need for other platforms yet, but I'm totally open to broadening this.

With regards to documentation, I wonder if it is time to consider moving away from README documentation to something like a static site or ReadTheDocs - this is worth considering outside of the scope of these changes even?

I've been having the same thoughts. I've logged #146 to track that effort.

My guess is that relying on godoc will be insufficient for at least some future users.

Yeah, that's my main worry.

Ok, now @sean-:

Have you thought about breaking gomplate into a sub-package like github.com/hairyhenderson/gomplate/cmd/gomplate so that the "gomplate" template library could be the meta-library used to aggregate a bunch of different APIs and helper functions?

I almost ended up doing this when I moved to cobra recently. But the changes would have been too drastic at the time. Plus I didn't really think that gomplate would ever be used as anything but a CLI tool. I am aware however of at least one case where someone's imported the vault package as part of a private utility... So, this'd probably be wise. I've logged #147 around this.

I think all that is necessary is to inject a new template interface by adding the necessary functions to text/template before you create the context

Yup, totally. The change itself seems pretty minimal, but I want there to be good tests and documentation accompanying it, so that's where things get a bit more complex.

notably that the template functions in go-sockaddr should not be strict wrappers on their lower-level library calls, instead I should type-assert in the template library with a wrapper function and then pass the value down to the lower-level library call

Yup. I've had to resort to all kinds of shady use of interface{} lately, and having lower-level funcs with strict(er) signatures would make things way easier to deal with in lots of ways. I think this fits into the context of #147.

As function namespacing goes, use sockaddr. as the prefix (or sockaddr:). For example: sockaddr:GetPrivateInterface | sockaddr.Attr "address". Either looks reasonable to me.

I have a slight preference for .... I must confess I'm having a little trouble figuring out exactly how this could be implemented, though. I can't just do t.Funcs(&template.FuncMap{ "sockaddr.Attr": sockaddr.Attr}), since I get panic: function name sockaddr.Attr is not a valid identifier. Same deal for :. Am I missing something obvious!?

Again - thanks for all the feedback. Overall, I think this proposal is great, and I'm in favour of getting it in. The only hurdle is to complete #146 first (thought this effort can certainly start in parallel).

[hairyhenderson]

I must confess I'm having a little trouble figuring out exactly how this could be implemented, though.

Scratch that - I mocked something up and it seems to work pretty well - see b37303d. Unless there are any better ideas... 😉

[sean-]

Ha, b37303d is clever. :)

[hairyhenderson]

@sean- maybe, but I'm not 100% sold on it - I threw it in #148 and complained a bit about it there 😉

[hairyhenderson]

@jen20 @sean- seems this issue got somewhat lost in the sands of time... Were you still planning to put go-sockaddr support into gomplate? Otherwise I may put it in myself eventually...

[sean-]

@hairyhenderson Go for it (no pun intended). We're both respectively 100% committed w/ our time atm and will be for the next month or two. ESTAYTUNED If you want us to review something, please feel free to ping. I'm very interested in this but it's not at the top of my priority queue (sadly).

[hairyhenderson]

Go for it

😂

Ok, I'll probably take a stab at this at some point - I'll @mention the two of you for reviews. Thanks!