Add support for metadata API

[mateiidavid]

Motivation

In order to reduce memory and I/O pressure, we can use the (undocumented) metadata API. Instead of retrieving an entire object, we can retrieve only the metadata. This is useful in lots of scenarios where users might not be interested in the spec or status.

Solution

Metadata-only requests are made possible through the REST API by including an extended Accept header. The header will notify the API Server a JSON response is expected, and it will encode an additional 'as=PartialObjectMetadata' k=v pair to receive only the metadata of the object. This is more efficient when only the metadata needs to be parsed (as opposed to the whole spec).

To make it possible to use the API, this change adds new (metadata specific) request impls in kube-core, and new metadata methods on the Api<K> type. Unfortunately, the methods on the Api type get complicated really quickly when attempting to preserve the function signatures originally posted in #1027. Deserializing the wire format we have in the response directly into an ObjectMeta type works if we first deserialize into a Value and then into the concrete type. For wrapper types (such as in list_metadata or watch_metadata) this gets even more complex.

To get around it, I've added a new PartialObjectMeta type that is supposed to mirror the Go counterpart. A wrapper list has also been added. Both of these support conversions to ObjectMeta (or ObjectList<ObjectMeta>). This makes it a bit neater to implement everything.

This is still WIP (so the code is not super tidy), I wanted to show the progress and find a consensus moving forward (or even some ideas on how we could deserialize this better if we're not happy with the PartialObject types).

Sending requests through curl

Thought it might be helpful. This helped me look at the format a bit more closely.

First, run kubectl proxy & and then send requests with curl

# Get a deployment's metadata
curl -H "Accept: application/json;as=PartialObjectMetadata;g=meta.k8s.io;v=v1" http://localhost:8001/apis/apps/v1/namespaces/kube-system/d
eployments/foo

# List a deployment(s) metadata
 curl -H "Accept: application/json;as=PartialObjectMetadataList;g=meta.k8s.io;v=v1" http://localhost:8001/apis/apps/v1/namespaces/kube-syst
em/deployments

# Watch
curl -H "Accept: application/json;as=PartialObjectMetadata;g=meta.k8s.io;v=v1" "http://localhost:8001/api/v1/namespaces/kube-system/pods?&
watch=true&resourceVersion=0&timeoutSeconds=290&allowWatchBookmarks=true"

Closes #1027

Signed-off-by: Matei David [email protected]

[clux]

This is great! Don't worry that the type signatures doesn't match what we originally set out. Matching the api is the actual goal. Left some minor comments on testing and formatting, but beyond that I think it's basically only docs that's needed here - EDIT: provided we can get CI green :-)

Watcher extensions in runtime is also an essential down the line, but we can do this here first.

[clux]

if this works, then this is perfect; a mirror of what the api does (what the crate is meant to do).

..it will present some extra logic to watcher to actually set up a persistent watch stream (watcher assumes the input type is the same as the output type, and now we need to go through conversion first), but hopefully we can do something similar to watch_object and specialize for that down the line (maybe some Into<K> bounds can do it).

[mateiidavid]

By persistent here we mean watches that do not terminate on errors? I have to be honest, haven't spent a lot of time looking and understanding how they're implemented in kube-rs. I looked at watch_object, think it might be work with some bounds. Happy to take a stab at it once this wraps up. Will try to do it before a release is out 🤞🏻

[clux]

Yeah, most informer-style(watcher)/reflector-style/controller based workloads will want to use an infinite watch that retries on de-syncs. Regular Api::watch is basically awful for real-world use-cases because of how temperamental the api is, so we forward people to watcher or the more specific watch_object.

Happy to take a stab at it once this wraps up. Will try to do it before a release is out 🤞🏻

That would be great ❤️

[mateiidavid]

Thanks a lot for the review! Added docs, might need some wordsmithing though. Thanks for the nightly toolchain and just recipe tip (yay for more just! makes my life easier). Fixed a unit test, put up docs, expanded the integration test. Let's hope we're all green :D

[codecov-commenter]

Codecov Report

Merging #1137 (8b6bcb9) into main (4c26615) will increase coverage by 0.29%.
The diff coverage is 77.24%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1137      +/-   ##
==========================================
+ Coverage   72.48%   72.78%   +0.29%     
==========================================
  Files          65       66       +1     
  Lines        4896     5063     +167     
==========================================
+ Hits         3549     3685     +136     
- Misses       1347     1378      +31     

Impacted Files	Coverage Δ
kube-core/src/metadata.rs	15.38% <15.38%> (ø)
kube-client/src/api/core_methods.rs	71.83% <46.15%> (-14.84%)	⬇️
kube-core/src/request.rs	94.41% <88.77%> (-1.99%)	⬇️
kube-client/src/lib.rs	93.88% <93.33%> (-0.12%)	⬇️
kube-core/src/resource.rs	53.57% <0.00%> (+8.33%)	⬆️

[clux]

This is all sensible to me. As discussed on discord, a watcher setup would be nice, but it's non-trivial without any further indirection so let's put a clear break here for now.

CI is green (EDIT: github was lagging, hopefully still after the approval) apart from the unrelated deny issue, so all good for me. Can merge tomorrow unless you have objections.

[mateiidavid]

This is all sensible to me. As discussed on discord, a watcher setup would be nice, but it's non-trivial without any further indirection so let's put a clear break here for now.

CI is green (EDIT: github was lagging, hopefully still after the approval) apart from the unrelated deny issue, so all good for me. Can merge tomorrow unless you have objections.

No objections :) will start playing with the code to see how we can get a watcher setup working