Using kube-openapi + 'kubectl explain' on Custom Resources?

[devdattakulkarni]

I was wondering if 'kubectl explain' can be used to find out information about Custom Resources?

Based on my understanding of kube-openapi and 'kubectl explain' workflow, it seems to me that this is currently not possible. Is this correct understanding?

I have written up my analysis here:
https://medium.com/@cloudark/understanding-kubectl-explain-9d703396cc8

Could you take a look and let me know if it is correct?

cc: @sttts

[sttts]

We don't publish the validation OpenAPI schema of CRD as part of the swagger.json. We should do that (with some loss in expressivity because we are still on OpenAPI v2 there, but on v3 for CRD validation). I am sure we also have an issue for that. This isn't too hard to do, just nobody did it.

[sttts]

I don't agree with the kubectl get --raw /apis/kubediscovery.cloudark.io/v1/namespaces/default/Postgres/ explain approach from the linked medium post. The OpenAPI spec published by the API server has to be updated dynamically in-memory with CRD validation schemata. This is possible and the whole apiserver architecture is build towards that.

[devdattakulkarni]

Thanks @sttts for the inputs.

I will be happy to contribute on this issue. Can you share the link to the issue
and any other relevant pointers?

[mingfang]

is there a plan for this?

[devdattakulkarni]

@mingfang I haven't heard anything on it yet.

I the mean while we have implemented the solution outlined in the blog post.
If you are interested in checking it out, try the steps here:
https://github.com/cloud-ark/kubeplus/blob/master/kubeplus-steps.txt

OpenAPI Spec for Postgres custom resource can be queried as shown in line:
https://github.com/cloud-ark/kubeplus/blob/master/kubeplus-steps.txt#L47

[mingfang]

This issue kubernetes/kubectl#546 may be useful

[mingfang]

It looks like progress relating to this is being made in kubernetes v1.13.
kubernetes/kubernetes#67205

[roycaihw]

kubernetes/kubernetes#67205 was moved out of 1.13 because we hit some snags. We opened kubernetes/kubernetes#71192 to continue the development. It will address the kubectl explain CR issue.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[roycaihw]

the publish CRD openapi feature has been merged in 1.14 kubernetes/kubernetes#71192

website documentation on how to enable the feature is to be merged in kubernetes/website#12910

/close

[k8s-ci-robot]

@roycaihw: Closing this issue.

In response to this:

the publish CRD openapi feature has been merged in 1.14 kubernetes/kubernetes#71192

website documentation on how to enable the feature is to be merged in kubernetes/website#12910

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mingfang]

@roycaihw It looks like 1.14 does not fully address this issue.
Given this CRD

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
  - name: v1
    served: true
    storage: true
  version: v1
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            cronSpec:
              type: string
              pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
            replicas:
              type: integer
              minimum: 1
              maximum: 10

kubectl explain only shows the metadata but not the spec.

kubectl explain crontabs --recursive
KIND:     CronTab
VERSION:  stable.example.com/v1

DESCRIPTION:
     <empty>

FIELDS:
   apiVersion	<string>
   kind	<string>
   metadata	<Object>
      annotations	<map[string]string>
      labels	<map[string]string>
      name	<string>
      namespace	<string>
      <removed non-relevant fields>
   spec	<>

I expected the spec to be filled in.
Using the openapi's visitor pattern, I'm getting called on VisitArbitrary() for the spec.
This call does not have any information about the spec, such as cronSpec in this case.

How does kubectl handle this?

[sttts]

You forgot to specify type: object for the spec. For backwards compatibility reasons we cannot publish properties "alone".

[mingfang]

@stts Thanks. I confirm that setting spec.type = object works.

[mingfang]

I ran a few tests with variations of the crontab CRD.
First is the basic CRD without any openAPIV3Schema, second is with openAPIV3Schema but not spec type(above) and finally with spec.type = object.

In all cases, the request sent bykubectl create -f crontab.yaml --v=8 where crontab is

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image

is the same

{"apiVersion":"stable.example.com/v1","kind":"CronTab",
"metadata":{"name":"my-new-cron-object","namespace":"default"},
"spec":{"cronSpec":"* * * * */5","image":"my-awesome-cron-image"}}

Can I assume from this that kubectl and api-server always expects the spec to be an object regardless of whether there is an openAPIV3Schema and spec.type?
and that the openAPIV3Schema only changes the published openapi?

[sttts]

@mingfang if you don't specifiy it, the type of spec is not checked. The create call always does the same.

[mingfang]

I think this needs to be documented.
The crontab crd example I used was from this document https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/ and it does not set spec.type to object.

[devdattakulkarni]

+1 about documentation

@mingfang
Do you have a github gist or something of your exact setup and steps.
I would like to try it out as well.

@sttts I guess can suggest where the documentation can go eventually.

[mingfang]

@devdattakulkarni
I created a repo with tests here https://github.com/mingfang/kubernetes-crd-test

[devdattakulkarni]

@mingfang: Thanks!!

[devdattakulkarni]

@sttts Reading through https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#publish-validation-schema-in-openapi-v2 and the examples by @mingfang at https://github.com/mingfang/kubernetes-crd-test
I was wondering why is explain functionality for CRDs tied with OpenAPI validation? Not all CRDs will necessarily have Spec validation defined. But every CRD will benefit from 'explain' functionality. So I am wondering what is the reason to couple these two things?

[mingfang]

@devdattakulkarni The reason is because kubectl explain works by loading the openapi published by the apiserver. The new feature in 1.14 is to publish CRDs with validation schema as part fo the openapi, which enables my test case #2 and #3 to work.

You are correct that most CRDs fall in the test case #1 scenario and do not have openapi validation. Without it, there's really nothing to "explain". Hopefully more CRDs will include the validation going forward.

[devdattakulkarni]

@mingfang I understand the implementation. But I don't understand why there is no need to "explain" a CRD that does not have any validation defined. To me, "explain" is about showing the fields that are available in Custom Kind Spec and their descriptions - what do they do, how to use them, etc. This has nothing to do with validation -- at least I don't see how validation can be a dependency for this.

[mingfang]

@devdattakulkarni Without openAPIV3Schema, the only fields to explain are the basic metadata fields.

Basically the CRD here https://github.com/mingfang/kubernetes-crd-test#test-case-1 can have the same explain output here https://github.com/mingfang/kubernetes-crd-test#test-case-2.

I made the same comment here kubernetes/kubernetes#71159 (comment) but apparently that can't be done due to backwards compatibility issue with kubectl.

In fact my Terraform plugin requires the metadata fields and I had to hack it here https://github.com/mingfang/terraform-provider-k8s/blob/e04774a198874111e505c40fdaeef4149db15f82/k8s/k8sconfig.go#L200

[devdattakulkarni]

@mingfang Thanks for the pointer to #71159. Reading through the comments in there, I agree with you that not everyone will necessarily provide validation schema in the CRD definition. The backward compatibility issue with kubectl seems to be the blocker. But even after that is resolved, I am wondering how would explain behave for spec properties for which validation is not defined in the schema. Disclaimer: I haven't experimented with schema validation, so don't know whether validation needs to be defined for all the Spec properties or if it okay to define it only for some subset.