Limit dashboard visibility to a subset of namespaces

[MohamedTalhaoui]

We have different teams and want give them different dashboard to manage their own pipelines.
Basically we are giving each team a namespace and deploy tekton pipeline and dashboard on it.
K8S RBAC provides isolation so team A cannot see team B pipelines.
The problem is that tekton dashboard needs cluster roles to list the pipelines in all namespaces.
We have tried to change the tekton-dashboard-minimal ClusterRole/ClusterRoleBinding to Role/RoleBinding, but that makes the dashboard failing to start.

Expected behavior

Tekton dashboard should work with only Role/RoleBinding and only show pipelines in its own namespace

Actual behavior

Tekton dashboard is failing because it needs Tekton ClusterRole/ClusterRoleBinding to list all pipelines in the cluster

Steps to reproduce the problem

1. Change Dashboard ClusterRole & ClusterRoleBinding to Role & RoleBinding
2. Deploy the dashboard
3. Dashboard pod fails to start

Environment

• Kubernetes Platform:
  GKE v1.14.10-gke.17

• Tekton Pipelines version:
  v0.10.0

• Tekton Triggers version:
  latest

• Tekton Dashboard version:
  latest

[a-roberts]

Yeah this is an interesting one, agree it'd be nice to have - so can you share the pod log details please or the errors you're seeing?

I wonder if in our UI we do an all namespaces lookup first and you're getting a displayed RBAC error on the page 🤔

[MohamedTalhaoui]

Thanks for the prompt reply.
I think your assumption is correct.

here are the logs I get:

E0206 16:03:06.958071       1 reflector.go:134] runtime/asm_amd64.s:1357: Failed to list *v1alpha1.ClusterTask: clustertasks.tekton.dev is forbidden: User "system:serviceaccount:team-a:tekton-dashboard" cannot list resource "clustertasks" in API group "tekton.dev" at the cluster scope
E0206 16:03:07.445879       1 reflector.go:134] runtime/asm_amd64.s:1357: Failed to list *v1alpha1.PipelineResource: pipelineresources.tekton.dev is forbidden: User "system:serviceaccount:team-a:tekton-dashboard" cannot list resource "pipelineresources" in API group "tekton.dev" at the cluster scope
E0206 16:03:07.751665       1 reflector.go:134] runtime/asm_amd64.s:1357: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:team-a:tekton-dashboard" cannot list resource "secrets" in API group "" at the cluster scope
E0206 16:03:07.961594       1 reflector.go:134] runtime/asm_amd64.s:1357: Failed to list *v1alpha1.ClusterTask: clustertasks.tekton.dev is forbidden: User "system:serviceaccount:team-a:tekton-dashboard" cannot list resource "clustertasks" in API group "tekton.dev" at the cluster scope
E0206 16:03:08.448351       1 reflector.go:134] runtime/asm_amd64.s:1357: Failed to list *v1alpha1.PipelineResource: pipelineresources.tekton.dev is forbidden: User "system:serviceaccount:team-a:tekton-dashboard" cannot list resource "pipelineresources" in API group "tekton.dev" at the cluster scope
E0206 16:03:08.753924       1 reflector.go:134] runtime/asm_amd64.s:1357: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:team-a:tekton-dashboard" cannot list resource "secrets" in API group "" at the cluster scope
E0206 16:03:08.972151       1 reflector.go:134] runtime/asm_amd64.s:1357: Failed to list *v1alpha1.ClusterTask: clustertasks.tekton.dev is forbidden: User "system:serviceaccount:team-a:tekton-dashboard" cannot list resource "clustertasks" in API group "tekton.dev" at the cluster scope
E0206 16:03:09.450402       1 reflector.go:134] runtime/asm_amd64.s:1357: Failed to list *v1alpha1.PipelineResource: pipelineresources.tekton.dev is forbidden: User "system:serviceaccount:team-a:tekton-dashboard" cannot list resource "pipelineresources" in API group "tekton.dev" at the cluster scope

[MohamedTalhaoui]

Any update or progress here ?

[a-roberts]

Any update or progress here ?

Hey @MohamedTalhaoui, sorry for taking so long to get back to you - nobody's working on this but still think it's a good idea! Not sure if you can see it but we have a ZenHub board so I've put it up the backlog as it's an interesting piece of work anyone could pick up (and most of the active developers on the Dashboard here at IBM look at said board when looking for something new to pick up).

For anyone who does pick this up, do a good analysis and explore what options we have.

[MohamedTalhaoui]

Hi,
thanks for the update.
Unfortunately I cannot see the ZenHub board but looking forward for more updates

[MohamedTalhaoui]

Hi,
any update ?

[eddycharly]

I made a step in this direction here: tektoncd/experimental#536

Pipelines don't need to be deployed in each namespace, a single instance of pipelines should be ok.
On the other hand, dashboard would need to be deployed multiple times.

It's probably not technically difficult to implement but how to make this available to the end user ?

[a-roberts]

Hey @MohamedTalhaoui and @eddycharly, I brought this up on the working group call today - it's something we should call out as a feature to add to our roadmap I think.

I agree, it'd need to be a "can have multiple installs of the Dashboard" implementation, so a good first step would probably be to have Roles and RoleBindings everywhere as a version (ideally through kustomize), or we somehow lay down those roles programatically based on an installation mode. I don't really want to see us having an increased test matrix (so, read-only * openshift or not * multiple install namespace mode or not), so design contributions would be awesome here

@AlanGreene ^^

[eddycharly]

@a-roberts yep I heard that ;)
We can still use ClusterRole but use RoleBinding instead of ClusterRoleBinding.

[eddycharly]

I wouldn’t create the roles programmatically, it would require the backend to have permissions to create roles, I would prefer to avoid that for security reasons.

[a-roberts]

I wouldn’t create the roles programmatically, it would require the backend to have permissions to create roles, I would prefer to avoid that for security reasons.

@a-roberts yep I heard that ;)
We can still use ClusterRole but use RoleBinding instead of ClusterRoleBinding.

Ah, didn't realise you were listening in! Doesn't ClusterRole require cluster-admin permissions to install? Happy to be mistaken and that would simplify things.

I'm wondering as this was brought up here (definitely a related issue): and it'd be great if addressing this issue removes any cluster admin necessities. +1 on avoiding doing things programatically.

[eddycharly]

Doesn't ClusterRole require cluster-admin permissions to install?

I think it's like other resources, RBAC decides if one is allowed or not to operate ClusterRoles.

Anyway i don't think the requirement here is to let users install the dashboard themself, it would probably be installed by someone in the ops team who has the necessary permissions to do so.

But once installed, it should show resources from a single namespace etc... not the whole cluster.

I could have misunderstood the issue though (@MohamedTalhaoui ?)

[eddycharly]

Issue #1355 is not related, it's a breaking change in Deployment label selectors (those are immutable).

I wrote it in the description of the PR (#1327)

This will introduce a breaking change when trying to update deployments as label selectors have changed.

But AFAIK this was not released yet, the user is probably trying to install master.

[a-roberts]

Issue #1355 is not related, it's a breaking change in Deployment label selectors (those are immutable).

I wrote it in the description of the PR (#1327)

This will introduce a breaking change when trying to update deployments as label selectors have changed.

But AFAIK this was not released yet, the user is probably trying to install master.

I see, yeah the reason I figured it's related is that they, by the sounds of it, don't have permissions to install the Dashboard in their cluster, but they do have permissions to install things into their own namespace, and so by implementing this feature, if we went down the On the other hand, dashboard would need to be deployed multiple times. route) then they'd benefit.

But indeed, everything we have is scoped to a namespace anyway, so they should be able to install into a particular namespace. Thanks for the feedback on this one and good to get the discussion going.

Yeah, But once installed, it should show resources from a single namespace etc... not the whole cluster. makes sense to me for the scope of this, let's see what @MohamedTalhaoui thinks

[AlanGreene]

#1355 is related, there are 2 issues discussed:

• 1 is a known breaking change related to recently added labels, the user has resolved this issue
• 2 is related to providing non-admin users access to the dashboard without giving them cluster-wide access

[MohamedTalhaoui]

Doesn't ClusterRole require cluster-admin permissions to install?

I think it's like other resources, RBAC decides if one is allowed or not to operate ClusterRoles.

Anyway i don't think the requirement here is to let users install the dashboard themself, it would probably be installed by someone in the ops team who has the necessary permissions to do so.

But once installed, it should show resources from a single namespace etc... not the whole cluster.

I could have misunderstood the issue though (@MohamedTalhaoui ?)

Exactely.
The goal is the ops provision the dashboards (1 per namespace, 1 namespace per team).
Then each team have access to its own dashboard showing its own pipelines.

[eddycharly]

I guess we’re talking about strong isolation here, not just ui presentation, otherwise the drop down in the side nav would be enough 😄

Such a setup will probably require admin rights at some points, chances are that it will require to configure an ingress with authentication, register dns records, ssl certificates...

[eddycharly]

2 is related to providing non-admin users access to the dashboard without giving them cluster-wide access

Do you mean preventing them from creating resources in any namespace of the cluster ?

[eddycharly]

The goal is the ops provision the dashboards (1 per namespace, 1 namespace per team).
Then each team have access to its own dashboard showing its own pipelines.

Ok so install is not an issue, Ops team has admin privileges in the cluster right ?

If that's correct, the only thing left is limiting the actions scope of the dashboard backend to a single namespace.

[MohamedTalhaoui]

I confirm Install is not an issue for my use case.

[eddycharly]

There are still some details to be worked on but it should be pretty straight forward.

My recommended setup would be as follows:

• create the tekton-pipelines namespace and install pipelines in it (the same instance will be shared by all teams, pipelines doesn’t support multi tenancy yet)
• create the tekton-dashboards namespace and install dashboards in it (one dashboard instance per team but all in the same namespace)
• create one namespace per team (team1, team2, etc...) to isolate resources per team (don’t necessarily grant them access to the namespace, especially if secrets and service accounts are managed by the ops team)
• bind each dashboard instance to a specific namespace

With this setup a specific dashboard instance would only access a specific namespace and therefore only the pipeline runs, secrets, service accounts etc of the team associated with the namespace.

The ingress/authentication part is out of scope here, it will depend on your setup/provider.

I have other ideas to make things more flexible and potentially cross namespaces/cross teams but we’re not quite there yet so the setup above would be more easily reachable for now.

[MohamedTalhaoui]

How would you bind a dashboard instance to a specific namespace ?
Also, to have strong isolation we are also deploying different EventListerner for each team to listen to git webhook, so a git push in team A repo does not trigger a pipeline on team B. What would be your advice here ?

[eddycharly]

How would you bind a dashboard instance to a specific namespace ?

Something along those lines dashboard --tenant-namespace team1, it's not currently supported but it's the idea.

Regarding your tiggers question i don't know triggers well enough, sorry.

[eddycharly]

I began working on this, splitting RBAC for now.
/assign

[AlanGreene]

/remove-lifecycle stale
/remove-lifecycle rotten
/reopen

[tekton-robot]

@AlanGreene: Reopened this issue.

In response to this:

/remove-lifecycle stale
/remove-lifecycle rotten
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[AlanGreene]

We need to review the current state of this feature and determine next steps. It seems like there may be some issues: https://tektoncd.slack.com/archives/CHTHGRQBC/p1602167113007100

Waiting for more details from the user.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[AlanGreene]

This is still something we'd like to do, freezing so it doesn't get auto-closed.
/lifecycle frozen

[DrissiReda]

Is it possible to have this work on multiple namespaces instead of a single one?

[AlanGreene]

The --tenant-namespace flag in the installer currently accepts a single namespace and is primarily used to apply the required RBAC on the Dashboard service account to restrict access to the specified namespace. You could customise the RBAC yourself to grant access to multiple namespaces however there are some caveats to this.

The other function of the --tenant-namespace flag (applied as the --namespace flag on the deployment) is to inform the UI whether it should display the 'All Namespaces' option in the namespace dropdown or just the single specified namespace. The way this is currently implemented doesn't support specifying multiple values but should be reasonably simple to extend.

Historically the reason for supporting a single namespace was due to some of the back end code which registered informers for resources in the specified namespace instead of registering cluster-wide. This code has since been removed and we now take a different approach so the original limitations no longer apply.

---

Current situation

If you were to provide custom RBAC today to restrict access to a subset of namespaces one of two things would happen in the UI (this is without specifying the --tenant-namespace or --namespace flag):

• if the Dashboard service account still has access to list namespaces, the user will be presented with the list, but will only be able to view resources in the allowed namespaces and will receive errors if they select other namespaces. Requests to list resources in 'all namespaces' would likely fail with a similar error (need to test this)
• if the Dashboard service account does not have access to list namespaces, the user would need to construct the namespaced URL themselves to view resources as the UI would not display a list of namespaces from which to select.

Future

I think that when we revisit the 'single namespace' restriction described in this issue it would actually make sense to extend it to allow specifying multiple values, and ensure we handle this correctly in the UI. Updating the title and description accordingly.

[DrissiReda]

I tried restricting the rbac to some namespaces by switching ClusterRoleBinding resources to RoleBinding ones. But I just get the error Forbidden and no logs in the pod.

[AlanGreene]

/area roadmap

[AlanGreene]

PR #3042 is open to replace the existing --namespace arg (single namespace visibility) with a new --namespaces arg which supports a comma-separated list of namespaces that will be visible via the Dashboard UI. It should be included in the next release due in ~1-2 weeks.

[AlanGreene]

This will be available in the next nightly release ~4am UTC tomorrow, and will be included in the v0.39.0 release in the next week or so.