Proposal: Allow opting out of the router (user brings their own ingress controller instead)

[krancour]

Support for k8s ingresses and Let's Encrypt are two frequently requested features for the router. Enabling either of these without incurring breaking changes seems challenging, if not impossible... unless... we make these opt-in by hiding them behind a feature flag!

I've recently submitted helm/charts#146, which adds a Helm chart that utilizes Traefik as a Kubernetes ingress contoller / lightweight router with opt-in support for Let's Encrypt.

This chart would work very well with Workflow if, depending upon the value of a feature flag, the controller were to manage k8s ingress resources instead of "routable services."

Caveats:

• Router does tons of web-servery stuff as well as routing. This includes features like gzip compression (for example). Traefik is just a router / proxy. It doesn't do some of these web-servery kind of things. (imo, it's a good thing that Traefik is simpler-- does one thing and does it well.) If your application needs any of these missing features, you need to implement them within your application. (Ordinarily, this is a good use case for a "sidecar" container, although Workflow doesn't have any specific support for deploying multi-container applications.) Bottom line: be aware of what features you sacrifice when you opt for this.
• Like the router, when SSL is enabled, Traefik terminates SSL. There is currently no ability to achieve e2e SSL. This might make this a poor choice for routing in certain industries, i.e. healthcare, banking, etc. But this might remain a very attractive option to everyone else.
• Like the router, Traefik won't handle non-HTTP/S traffic.

[cdupont]

Hi guys, what is the status of this?

[bacongobbler]

Our system is relatively simple for bug triage: this issue is labeled as a proposal and there is no milestone and nobody assigned to it, so nobody is working on this proposal. :)

[krancour]

fwiw, I'd like to amend the original idea here to not be Traefik specific. I'd propose that what's better to is just let people opt out of the router (instead of opting in to Traefik). The idea would then be that they could bring along their own preferred ingress controller-- whatever that happens to be.

I'll update the issue's title to reflect that.

[krancour]

@bacongobbler I haven't been in Workflow planning sessions lately due to other work... I'm curious whether there's at least been any discussion of this.

[bacongobbler]

No discussion on this front has occurred AFAIK. We're thinking of tackling webhooks and deis run bash for the next milestone but nothing on the router side of things.

[krancour]

fwiw, I feel this is something a lot of people have expressed interest in.

[cdupont]

HI again,
is there a workaround?
At the moment my platform is not usable, because an administrator need to add an ingress in kubernetes manually after a user pushed an application in Deis.

[bacongobbler]

You can fork the controller and make the scheduler create an ingress route after deploying the application, but there is no existing feature flag/workaround for that use case.

[cdupont]

@bacongobbler I can try. Is this the location where I should add the ingress route?
https://github.com/cdupont/controller/blob/master/rootfs/scheduler/resources/deployment.py#L106

I could add in the manifest generated an ingress route:

 apiVersion: extensions/v1beta1
 kind: Ingress
 metadata:
   name: <app name>
   namespace: <namespace>
 spec:
   rules:
   - host: <app name>.domain
     http:
       paths:
       - path: /
         backend:
           serviceName: <app name>
           servicePort: <entrypoint>

[bacongobbler]

We'd likely accept a PR if someone wants to tackle this.

[bacongobbler]

@cdupont yes.

[krancour]

@cdupont are you actively working on this? I'm asking because Deis has brought on a new resource who is probably ready to tackle this and I'm working up a more thorough design proposal. I just want to minimize any overlapping efforts on your part or ours.

[cdupont]

Hi,
I just added this to https://github.com/cdupont/controller/blob/master/rootfs/scheduler/resources/deployment.py#L106

        ingress_rules = {
            'host': name + '.waziup.io',
            'http': {
                'paths': [{
                     'path': '/',
                     'backend': {
                           'serviceName': name,
                           'servicePort': entrypoint
                     }
                }]
            }
        }
        ingress = {
            'kind': 'Ingress',
            'apiVersion': 'extensions/v1beta1',
            'metadata': {
                'name': name,
                'labels': labels,
                'annotations': {
                    'kubernetes.io/change-cause': kwargs.get('release_summary', '')
                }
            },
            'spec': {
                'rules': [ingress_rules]
            }
        }

But didn't tested. Also, I think it would be better to add it to its own file (like services.py).
So, feel free to take it up from there :)
I will have very little to no time next weeks to work on it.
Just to know, when do you think this could land?

[cdupont]

BTW, I cannot compile the controller:

$ make setup-venv
python3 -m venv venv
venv/bin/pip3 install --disable-pip-version-check -q -r rootfs/requirements.txt -r rootfs/dev_requirements.txt
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-ysm9qovl/psycopg2/
Makefile:54: recipe for target 'setup-venv' failed
make: *** [setup-venv] Error 1

[mboersma]

@cdupont do you have PostgreSQL libraries & headers installed? On macOS brew install postgresql is necessary to do pip3 install for controller.

[krancour]

@cdupont

I just added...

After digging into this pretty deep, I know of a considerable bit more that needs to be done. This list is non-exhaustive, but will give you an idea. We have an internal document that goes into a nauseating level of detail.

• Workflow chart needs to be modified to include or not include router
• Need to manage an ingress resource per custom / "vanity" domain
  • Need to manage TLS for ingresses whenever a certificate is attached to one of these domains via deis CLI (depending on which ingress controller you use and whether it supports ACME; you may or may not continue to use deis certs:add and deis certs:attach).
• Changes are required to builder's service if router no longer is in play (router currently implements a TCP pass-thru for the builder)

We welcome contributions, as always, but knowing how involved this is, I believe @mboersma is going to put a dedicated resource on this task. I just want you to be aware before you put too much more effort into this.

[cdupont]

@krancour feel free to move forward, I don't think I will be able to go much further anyway.

[cdupont]

Any update on this :)

[krancour]

@kris-nova is actively working on it.

[krisnova]

@cdupont

There are 2 PRs out for the feature

• feat(ingress) Experimental Native Ingress controller#1243
• feat(ingress) Experimental Native Ingress #732

They are in review now, so we are waiting to hear back from the team. But I imagine they should be merged in shortly.

Is there anyway you can spare a few minutes and see if these changes are in alignment with what you imagined the feature to feel like?

[cdupont]

Hi Kris,
I looked at the code, looks good!
Cant' wait to test it once it's merged.

[mboersma]

Phase 1 of bring-your-own-ingress is completed now and will be in the Deis Workflow v2.13 release. See https://deis.com/docs/workflow/installing-workflow/experimental-native-ingress/ for instructions.

We will track any remaining issues separately in the controller or workflow repositories, so please leave feedback there as appropriate.