Is it possible to create a CRD controller?

[omerlh]

I'm using the dotnet client in Kamus, and I really enjoyed it so far. I want to add a CRD (see Soluto/kamus#13), and ideally, I'll be able to write it in dotnet like the rest of the service. By looking at golang samples, look like I need to be able to listen to CLI events. I couldn't find the relevant methods in the SDK - am I missing something? I find only WatchCustomResourceDefinitionAsync, but I'm not sure this is the same as NewListWatchFromClient or NewInformer that are used on this sample.
Can you please help me and point me in the right direction?

[brendandburns]

Yes, you can definitely write a CRD controller in csharp.

ListWatch and Informer are higher-level constructs/utilities that are built into the golang client.

Equivalent capabilities haven't been built into the dotnet client (yet) but that doesn't mean that you can't build a CRD controller, it just means you'll have to do a little more work.

There's a ListWatch cache that has been built for Javascript here:

https://github.com/kubernetes-client/javascript/blob/master/src/cache.ts

It wouldn't be hard to implement something similar for csharp (and we'd love a PR), or you can just use List and Watch directly.

[omerlh]

So just to make sure I understand - I need to write a listWatch CRD using the function WatchCustomResourceDefinitionAsync and ListCustomResourceDefintionAsync? Because I tried using the watch and didn't get evens although I created CRDs object (not the definition).

[brendandburns]

Ah, I looked, you want WatchCustomObject not WatchCustomResource but in looking I realized that hasn't been generated for C#. I need to run a regeneration pass to get it in. I'll try to send a PR this evening....

Apologies!

[omerlh]

Thank you! I need also advice on a design: Is there a doc on how to write HA controller? I understand I need to listen for CLI events, so I guess it means having one pod listening for events on the cluster. How can I handle pod stopping/starting and ensuring I'm losing events?
Also - I wanted to generate the new API, but couldn't figure out why. Will be happy to send a PR :)

[omerlh]

Find my answer (regarding the controller) on the docs:

A given Kubernetes server will only preserve a historical list of changes for a limited time. Clusters using etcd3 preserve changes in the last 5 minutes by default. When the requested watch operations fail because the historical version of that resource is not available, clients must handle the case by recognizing the status code 410 Gone, clearing their local cache, performing a list operation, and starting the watch from the resourceVersion returned by that new list operation. Most client libraries offer some form of standard tool for this logic. (In Go this is called a Reflector and is located in the k8s.io/client-go/cache package.)

[omerlh]

One finall update - I managed to understand how to do it by hand using:

var watch = await kubernetes.WatchObjectAsync<object>("apis/drud.com/v1/apps");

But recevied the following execption:

System.NullReferenceException: Object reference not set to an instance of an object.
   at k8s.Watcher`1.WatcherLoop(CancellationToken cancellationToken)
closed

I guess it's somewhere inside the watch but I couldn't understand where...

[brendandburns]

It's going to try to deserialize the objects into a JSON parsed object here:

csharp/src/KubernetesClient/Watcher.cs

Line 120 in 9372e32

var @event = SafeJsonConvert.DeserializeObject<k8s.Watcher<T>.WatchEvent>(line);

So instead of passing object I think you should pass the object you should expect with the proper JSON field names.

Hope that helps. This is definitely harder than it should be...

[omerlh]

Got the same exception, trying:

await kubernetes.WatchObjectAsync<KamusSecret>("apis/soluto.com/alpha/kamussecrets");

This is KamusSecret:

    class KamusSecret : KubernetesObject
    {
        public Dictionary<string,string> Data { get; set; }
        public string Type { get; set; }
        public V1ObjectMeta Metadata { get; set; }
    }

And when trying the following URL http://localhost:8001/apis/soluto.com/alpha/kamussecrets?watch=1 I get valid results:

{"type":"ADDED","object":{"apiVersion":"soluto.com/alpha","data":{"key":"J9NYLzTC/O44DvlCEZ+LfQ==:Cc9O5zQzFOyxwTD5ZHseqg=="},"kind":"KamusSecret","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"soluto.com/alpha\",\"data\":{\"key\":\"J9NYLzTC/O44DvlCEZ+LfQ==:Cc9O5zQzFOyxwTD5ZHseqg==\"},\"kind\":\"KamusSecret\",\"metadata\":{\"annotations\":{},\"name\":\"my-drupal-site\",\"namespace\":\"default\"},\"type\":\"TlsSecret\"}\n"},"clusterName":"","creationTimestamp":"2019-02-12T06:58:09Z","generation":1,"name":"my-drupal-site","namespace":"default","resourceVersion":"1162182","selfLink":"/apis/soluto.com/alpha/namespaces/default/kamussecrets/my-drupal-site","uid":"8e3f5bac-2e93-11e9-9566-025000000001"},"type":"TlsSecret"}}
{"type":"DELETED","object":{"apiVersion":"soluto.com/alpha","data":{"key":"J9NYLzTC/O44DvlCEZ+LfQ==:Cc9O5zQzFOyxwTD5ZHseqg=="},"kind":"KamusSecret","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"soluto.com/alpha\",\"data\":{\"key\":\"J9NYLzTC/O44DvlCEZ+LfQ==:Cc9O5zQzFOyxwTD5ZHseqg==\"},\"kind\":\"KamusSecret\",\"metadata\":{\"annotations\":{},\"name\":\"my-drupal-site\",\"namespace\":\"default\"},\"type\":\"TlsSecret\"}\n"},"clusterName":"","creationTimestamp":"2019-02-12T06:58:09Z","generation":1,"name":"my-drupal-site","namespace":"default","resourceVersion":"1162511","selfLink":"/apis/soluto.com/alpha/namespaces/default/kamussecrets/my-drupal-site","uid":"8e3f5bac-2e93-11e9-9566-025000000001"},"type":"TlsSecret"}}

[brendandburns]

Yeah, it looks like you need to add warch=1

There's a comment here:

https://github.com/kubernetes-client/csharp/blob/master/src/KubernetesClient/Kubernetes.Watch.cs#L48

That explains why.

But honestly, looking at the code (I didn't write most of it) I feel like there are two competing watch implementations...

Anyway, I think that what you need to do is parse out that "object" field and that will be your KamusSecret.

Alteranately you could create:

class KamuSecretWatchContainer {
  // The type of Watch event (ADDED, DELETED,etc)
   public string type

  // The object itself
  public KamusSecret object
}

And pass that to the Watch.

Sorry this is so complicated. we should make it easier. Thanks for sticking with it!

[omerlh]

I tried adding the query param:

await kubernetes.WatchObjectAsync<KamusSecret>("apis/soluto.com/alpha/kamussecrets?watch=1");

But the requests failed with NotFound error from Kuberentes. Which is weird, because

GET http://localhost:8001/apis/soluto.com/alpha/kamussecrets?watch=1

Hang because there are no objects right now. So it exists. I have no idea why it's not working. I looked on the URL in the exception and it looks valid...

[brendandburns]

Hrm, ok at this point I think I just need to reproduce the error on my local box and debug more :)

Apologies!

[omerlh]

Sure, thanks!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.