Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

minimize the RBAC permissions for the pull mode cluster #5793

Merged
merged 1 commit into from
Nov 20, 2024

Conversation

zhzhuang-zju
Copy link
Contributor

@zhzhuang-zju zhzhuang-zju commented Nov 7, 2024

What type of PR is this?
/kind feature

What this PR does / why we need it:
image
Currently, pull-mode clusters interact with karmada-apiserver on the control plane via a mounted karmada-kubeconfig, where the certificate has Organization as system:nodes and CN as system:node:{cluser_name}, so karmada -agent has the RBAC permissions on the control plane of a user with Group as system:nodes and username as system:node:{cluser_name}.
Since the control plane binds the clusterrole to Group system:nodes, all karmada-agents actually have the same set of RBAC permissions.

The clusterrole bound to Group system:nodes has too many permissions, e.g. secret permissions, cluster permissions.

image
So in order to minimise RBAC permissions, each karmada-agent needs to be assigned separate RBAC permissions

Besides, in order to isolate the user group that karmada-agent uses to access the Karmada API Server from the kubelet, so that it is not interfered with by node authentication, the user group system:nodes used by karmada-agent is changed to system:agents, and the user system :node: to system:agent:.

This PR contains the following two elements:

  • assign each karmada-agent separate RBAC permissions
  • the user group system:nodes used by karmada-agent is changed to system:agents, and the user system :node: to system:agent:

Installation methods involved:

  • local-up
  • karmadactl init
  • helm

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

`Security`: minimize the RBAC permissions for the pull mode cluster

@karmada-bot karmada-bot added the kind/feature Categorizes issue or PR as related to a new feature. label Nov 7, 2024
@karmada-bot karmada-bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 7, 2024
@zhzhuang-zju
Copy link
Contributor Author

Local validation:

$ karmadactl register 172.18.0.6:32543 --token ak0bng.1mde96apjyi98dwz --discovery-token-ca-cert-hash sha256:c31ef70196f7b28545526c94a3c62ce74ad9c476971a818e16f49396f50b9430 --kubeconfig ~/.kube/member4.config --context member4  --karmada-agent-image docker.io/karmada/karmada-agent:latest --cluster-name member4 -v 4
I1107 20:23:59.964757 3098581 register.go:331] Registering cluster. cluster name: member4
I1107 20:23:59.964796 3098581 register.go:332] Registering cluster. cluster namespace: karmada-cluster
[preflight] Running pre-flight checks
I1107 20:23:59.964814 3098581 register.go:491] Validating the existence of file /etc/karmada/bootstrap-karmada-agent.conf
I1107 20:23:59.964827 3098581 register.go:491] Validating the existence of file /etc/karmada/karmada-agent.conf
I1107 20:23:59.964836 3098581 register.go:491] Validating the existence of file /etc/karmada/pki/ca.crt
[preflight] All pre-flight checks were passed
[karmada-agent-start] Waiting to perform the TLS Bootstrap
I1107 20:23:59.976730 3098581 register.go:1155] [discovery] Created cluster-info discovery client, requesting info from "172.18.0.6:32543"
I1107 20:23:59.988138 3098581 register.go:1193] [discovery] Requesting info from "172.18.0.6:32543" again to validate TLS against the pinned public key
I1107 20:23:59.997904 3098581 register.go:1210] [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.18.0.6:32543"
I1107 20:23:59.997940 3098581 register.go:506] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process
I1107 20:23:59.997958 3098581 register.go:517] [discovery] writing bootstrap karmada-agent config file at /etc/karmada/bootstrap-karmada-agent.conf
I1107 20:23:59.998535 3098581 register.go:526] [discovery] writing CA certificate at /etc/karmada/pki/ca.crt
I1107 20:24:00.461989 3098581 register.go:866] Waiting for the client certificate agent-rbac-generator-member4wcq4v to be issued
I1107 20:24:01.465729 3098581 register.go:874] Signing certificate of csr agent-rbac-generator-member4wcq4v successfully
[karmada-agent-start] Waiting to check cluster exists
[karmada-agent-start] Assign the necessary RBAC permissions to the agent
[karmada-agent-start] Waiting to construct karmada-agent kubeconfig
I1107 20:24:03.665759 3098581 register.go:866] Waiting for the client certificate member4-nn9h2 to be issued
I1107 20:24:04.669051 3098581 register.go:874] Signing certificate of csr member4-nn9h2 successfully
I1107 20:24:04.669084 3098581 register.go:908] writing bootstrap karmada-agent config file at /etc/karmada/karmada-agent.conf
[karmada-agent-start] Waiting the necessary secret and RBAC
I1107 20:24:04.680320 3098581 idempotency.go:68] Secret karmada-system/karmada-kubeconfig has been created or updated.
I1107 20:24:04.683534 3098581 idempotency.go:188] ClusterRole karmada-agent has been created or updated.
I1107 20:24:04.691452 3098581 idempotency.go:212] ClusterRolebinding karmada-agent has been created or updated.
[karmada-agent-start] Waiting karmada-agent Deployment

cluster(member4) is joined successfully

$ kubectl get cluster
NAME      VERSION   MODE   READY   AGE
member1   v1.31.0   Push   True    3d10h
member2   v1.31.0   Push   True    6d6h
member3   v1.31.0   Pull   True    32h
member4   v1.31.0   Pull   True    34s

$ karmadactl apply -f samples/nginx/deployment.yaml --cluster member4
deployment.apps/nginx created
propagationpolicy.policy.karmada.io/nginx-6d7f8d5f5b unchanged

$ kubectl --kubeconfig ~/.kube/member4.config get deployments.apps 
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   2/2     2            2           11s

$ karmadactl unregister member4  --cluster-kubeconfig ~/.kube/member4.config -v=4                                                                           
I1107 20:26:53.269394 3100053 unregister.go:186] Unregistering cluster. cluster name: member4
I1107 20:26:53.269597 3100053 unregister.go:187] Unregistering cluster. karmada-agent deployed in namespace: karmada-system
I1107 20:26:53.269608 3100053 unregister.go:188] Unregistering cluster. member cluster secrets stored in namespace: karmada-cluster
I1107 20:26:56.314363 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:26:57.317515 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:26:58.310547 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:26:59.310182 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:27:00.309665 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:27:01.314053 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:27:02.311988 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:27:03.309908 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:27:04.310080 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:27:05.309887 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:27:06.312824 3100053 unregister.go:333] Successfully delete cluster object (member4) from control plane.
W1107 20:27:06.312863 3100053 unregister.go:342] The RBAC resources on the control plane need to be manually cleaned up, including the following resources:
ClusterRole/system:karmada:member4:agent
ClusterRoleBinding/system:karmada:member4:agent
Role/karmada-cluster/system:karmada:member4:agent-secret
Role/karmada-es-member4/system:karmada:member4:agent-work
RoleBinding/karmada-cluster/system:karmada:member4:agent-secret
RoleBinding/karmada-es-member4/system:karmada:member4:agent-work
I1107 20:27:06.316944 3100053 unregister.go:366] Successfully delete resource ({ServiceAccount karmada-member4 karmada-cluster}) from member cluster (member4).
I1107 20:27:06.320891 3100053 unregister.go:366] Successfully delete resource ({ClusterRole karmada-controller-manager:karmada-member4 }) from member cluster (member4).
I1107 20:27:06.323882 3100053 unregister.go:366] Successfully delete resource ({ClusterRoleBinding karmada-controller-manager:karmada-member4 }) from member cluster (member4).
I1107 20:27:06.325671 3100053 unregister.go:366] Successfully delete resource ({Secret karmada-member4 karmada-cluster}) from member cluster (member4).
I1107 20:27:06.328792 3100053 unregister.go:366] Successfully delete resource ({ServiceAccount karmada-impersonator karmada-cluster}) from member cluster (member4).
I1107 20:27:06.332187 3100053 unregister.go:366] Successfully delete resource ({Secret karmada-impersonator karmada-cluster}) from member cluster (member4).
I1107 20:27:06.335267 3100053 unregister.go:366] Successfully delete resource ({Namespace karmada-cluster }) from member cluster (member4).
I1107 20:27:06.339246 3100053 unregister.go:366] Successfully delete resource ({Deployment karmada-agent karmada-system}) from member cluster (member4).
I1107 20:27:06.342516 3100053 unregister.go:366] Successfully delete resource ({ServiceAccount karmada-agent-sa karmada-system}) from member cluster (member4).
I1107 20:27:06.346048 3100053 unregister.go:366] Successfully delete resource ({ClusterRole karmada-agent }) from member cluster (member4).
I1107 20:27:06.350296 3100053 unregister.go:366] Successfully delete resource ({ClusterRoleBinding karmada-agent }) from member cluster (member4).
I1107 20:27:06.354519 3100053 unregister.go:366] Successfully delete resource ({Secret karmada-kubeconfig karmada-system}) from member cluster (member4).
I1107 20:27:06.354622 3100053 unregister.go:382] Successfully delete local file (/etc/karmada/karmada-agent.conf) in current node.
I1107 20:27:06.354666 3100053 unregister.go:382] Successfully delete local file (/etc/karmada/pki/ca.crt) in current node.

@zhzhuang-zju zhzhuang-zju force-pushed the register branch 2 times, most recently from afd687a to 5f781cd Compare November 8, 2024 02:51
@codecov-commenter
Copy link

codecov-commenter commented Nov 8, 2024

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 58.79518% with 171 lines in your changes missing coverage. Please review.

Project coverage is 46.10%. Comparing base (6795dba) to head (ae161fd).

Files with missing lines Patch % Lines
pkg/karmadactl/register/register.go 65.16% 108 Missing ⚠️
pkg/util/rbac.go 0.00% 28 Missing ⚠️
pkg/karmadactl/util/work.go 0.00% 18 Missing ⚠️
pkg/karmadactl/unregister/unregister.go 61.53% 8 Missing and 2 partials ⚠️
...dactl/cmdinit/bootstraptoken/agent/tlsbootstrap.go 73.91% 4 Missing and 2 partials ⚠️
pkg/karmadactl/cmdinit/karmada/deploy.go 0.00% 1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5793      +/-   ##
==========================================
+ Coverage   46.02%   46.10%   +0.08%     
==========================================
  Files         660      661       +1     
  Lines       53987    54307     +320     
==========================================
+ Hits        24848    25040     +192     
- Misses      27521    27644     +123     
- Partials     1618     1623       +5     
Flag Coverage Δ
unittests 46.10% <58.79%> (+0.08%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


🚨 Try these New Features:

@karmada-bot karmada-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 16, 2024
@zhzhuang-zju
Copy link
Contributor Author

cc @RainbowMango

Copy link
Member

@RainbowMango RainbowMango left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/assign

@zhzhuang-zju
Copy link
Contributor Author

[FAIL] remedy testing remedy.spec.decisionMatches is not empty [It] Cluster domain name resolution function encounters an exception, then remove the remedy resource
https://github.com/karmada-io/karmada/actions/runs/11884396791/job/33112661676?pr=5793

/retest

@zhzhuang-zju
Copy link
Contributor Author

zhzhuang-zju commented Nov 19, 2024

This PR decides to standardize the naming of Karmada system roles by prefixing them with system:karmada, to distinguish them from Kubernetes system roles prefixed with system:.
Besides, rename label kubernetes.io/bootstrapping: rbac-defaults to karmada.io/bootstrapping: rbac-defaults, and remove annotations rbac.authorization.kubernetes.io/autoupdate: "true" as its target objects are the default roles of Kubernetes, not the default roles of Karmada.
to summarization:

  • standardize the naming of Karmada system roles by prefixing them with system:karmada
  • rename label kubernetes.io/bootstrapping: rbac-defaults to karmada.io/bootstrapping: rbac-defaults
  • remove annotation rbac.authorization.kubernetes.io/autoupdate: "true"

Group for bootstrapping bearer tokens must match \Asystem:bootstrappers:[a-z0-9:-]{0,255}[a-z0-9]\z, so the group for karmada bootstrapping bearer tokens can be prefixed with system:bootstrappers:karmada.

  • standardize the group for karmada bootstrapping bearer tokens by prefixing them with system:bootstrappers:karmada

wdyt? @RainbowMango

@zhzhuang-zju
Copy link
Contributor Author

/retest

1 similar comment
@zhzhuang-zju
Copy link
Contributor Author

/retest

@zhzhuang-zju
Copy link
Contributor Author

all done~ cc @RainbowMango

Copy link
Member

@RainbowMango RainbowMango left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@karmada-bot karmada-bot added the lgtm Indicates that a PR is ready to be merged. label Nov 20, 2024
@karmada-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RainbowMango

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/feature Categorizes issue or PR as related to a new feature. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants