Allow persisting discovery bundles

[anderseknert]

With systems configured with discovery, it would be great if the new persist option was available to persist both the discovery bundle itself, as well as any from discovery config included bundles, in case the remote service was unreachable when re-starting an OPA instance.

For the configured policy bundles, I suppose one could add persist: true in the config provided by the discovery bundle, but whether to persist bundles feels more like a decision to make locally (like log level) than something that should be centrally managed. Having persist: true in the bootstrap configuration would thus be desirable.

[tsandall]

We should support persistence of discovery bundles because otherwise the persistence feature doesn't work when discovery is used (e.g., the OPA would start, attempt to download a discovery bundle, and then fail and remain inoperable.) The implementation should just persist the discovery bundle like any other one (i.e., there's no need to persist the configuration produced by the discovery bundle.)

[tsandall]

@ashutosh-narkar could you take a look and see how much work this would be?

[anderseknert]

The implementation should just persist the discovery bundle like any other one (i.e., there's no need to persist the configuration produced by the discovery bundle.)

Right, what I meant was that if the discovery bundle has configuration to download other bundles those should also be persisted if the discovery bundle bootstrap config has the persist: true attribute. Does that make sense?

[tsandall]

I'd just have the discovered configuration specify the persist option on each of the configured bundles rather than trying to bake in some kind of inheritance.

[anderseknert]

The discovered configuration is not necessarily even managed by the OPA admin, but rather the "bundle server admin". If I configure my OPA bootstrap config with persist: true it should be pretty clear that the intent is for the server to persist whatever it needs in order to be able to start with the remote system temporarily unavailable. If this is not an exclusively local decision we'd have no idea about whether what was persisted would actually help in that goal. On the other side of the this - if we don't care for this option, would like to disable it and run our OPAs on read-only file systems, the bundle server admin should still be able to configure our bundles to be written to disk?

Even when these two admins are the same person, having to administer this in two disparate systems sounds less than ideal.

(..additionally, our current bundle server of choice does not provide us with options to meddle with the configuration provided in discovery bundles, so unless that feature was added in that product too, this would not be available to us)

[tsandall]

That sounds like a limitation in the bundle server of choice that ought to be addressed...

RE: the implementation, I'm advocating for the option with the least amount of magic/implicit behaviour. I don't know how common the separation of duty between "OPA admin" and "bundle server admin" is. It sounds especially odd when Discovery is enabled since that means that all the "OPA admin" is doing is enabling Discovery--the "bundle server admin" would be in complete control of the discovered configuration so the "OPA admin" is really not in control. My guess is that this stems from the limitation in the bundle service you mentioned...

[anderseknert]

I can kinda get behind the no magic / implicit behavior from a developers point of view, but as an "OPA admin" I'm mainly interested in knowing that I can trust the config I provide will do what I intended, and if I say "persist my bundles" in hopes I can start OPA in a last known working state, it's hard to see the appeal of having that option scattered over many places - especially so given how saving something to local disk feels like a decision to make.. well, locally. The bundle server admin does not control the level of which logs should be written to console, and to me this feels like it's in the same category.

Is there even a use case for configuring say 2 out of 3 bundles with persist: true but leaving it off for the third? Unless there is, this just seems like an easy mistake to make - and one you won't probably won't notice until you really would have needed the "offline restore", and then it's of course gonna be too late. If some "magic" can help me avoid that, I'll take it ;)

Oh well, if that's not the direction this will take, I'll make sure to at least file a feature request to have this supported in our bundle server.

[tsandall]

I can kinda get behind the no magic / implicit behavior from a developers point of view, but as an "OPA admin" I'm mainly interested in knowing that I can trust the config I provide will do what I intended, and if I say "persist my bundles" in hopes I can start OPA in a last known working state, it's hard to see the appeal of having that option scattered over many places - especially so given how saving something to local disk feels like a decision to make.. well, locally. The bundle server admin does not control the level of which logs should be written to console, and to me this feels like it's in the same category.

We could add a global "persist my bundles" option to the bootstrap config however the same could be said for any of the other config setting (e.g., verification keys). I'd like to avoid the problem of merging config/desired state from multiple sources (in this case the bootstrap config and the discovered config) because that always causes headaches. Putting as little information as possible in the bootstrap config helps ensure that a central service can manage as much of the OPA instance as possible.

As far as logging and other server settings go...there is no good reason for them being outside of the rest of the config.

[anderseknert]

Thanks Torin! This isn't the hill I'm planning to die on ( 😄 ), so I'll have a chat with our dear bundle server provider to have this feature taken into account from their end, and I'm looking forward to seeing the discovery bundle included in the persist options on the OPA side 👍