Fix policy creation race

[jefferai]

Calling policy creates on first use, so if your first access (even a
read) of the plugin comes from a secondary you get a read only error
message. This runs policy creation as a part of the upgrade path and
adds a check to ensure it's been run.

This was tripping during the existence check, since to read metadata it creates a path encryptor except we don't forward existence checks due to read only errors.

[kalafut]

I see a warning about "The caller must have the backend lock." in policy(). Applies in the upgrade case too?

[jefferai]

That's a good point -- It's fine in the non-goroutine case because it's synchronous during backend factory creation. But if we don't run it then, and end up running it in the goroutine, then we don't have the lock at that time. This the presents an issue, because if we aren't upgraded, if we write the policy it'll be considered by the upgrade logic to be existing data and will be moved. So it has to be done after upgrading. But in the meantime clients can make calls that can then indirectly call policy. So I may indeed have to wrap the second invocation (in the goroutine) in a lock and it's just best effort. That said since secondaries will wait for the write of the upgrade done canary anyways probably they'll block client access until then so it'll be fine just moving that block up a bit.