Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use annotation for exposed affinity for dataflow #4277

Merged
merged 1 commit into from
Aug 27, 2024

Conversation

xliuqq
Copy link
Collaborator

@xliuqq xliuqq commented Aug 15, 2024

Ⅰ. Describe what this PR does

user should define annotation for exposed dataflow affinity. see the usage in the doc.

Ⅱ. Does this pull request fix one issue?

fixes #XXXX

Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

@cheyang cheyang requested a review from TrafalgarZZZ August 16, 2024 09:36
@TrafalgarZZZ
Copy link
Member

@xliuqq @cheyang Is it possible to store affinity information in annotations instead of labels? From my understanding, annotations indicate which labels should be exposed, and dataflow-controller injects key=value pair to DataOperation job's labels. Can we also injecting key=value pair to DataOperation job's annotations?

@@ -155,47 +154,28 @@ func (f *DataOpJobReconciler) injectPodNodeLabelsToJob(job *batchv1.Job) error {
}

// customized labels
if pod.Spec.Affinity != nil && pod.Spec.Affinity.NodeAffinity != nil {
fillCustomizedNodeAffinity(pod.Spec.Affinity.NodeAffinity, injectLabels, node)
exposedLabels := pod.Annotations[common.AnnotationDataFlowAffinityLabelsName]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should clearly comment here what does "exposed" mean for better code readabiility

pkg/common/label.go Outdated Show resolved Hide resolved
@xliuqq
Copy link
Collaborator Author

xliuqq commented Aug 22, 2024

@xliuqq @cheyang Is it possible to store affinity information in annotations instead of labels? From my understanding, annotations indicate which labels should be exposed, and dataflow-controller injects key=value pair to DataOperation job's labels. Can we also injecting key=value pair to DataOperation job's annotations?

do you mean only store the affinity information in annotations instead of labels for DataOperation Job when it finished ? I agree with this.

@TrafalgarZZZ
Copy link
Member

@xliuqq @cheyang Is it possible to store affinity information in annotations instead of labels? From my understanding, annotations indicate which labels should be exposed, and dataflow-controller injects key=value pair to DataOperation job's labels. Can we also injecting key=value pair to DataOperation job's annotations?

do you mean only store the affinity information in annotations instead of labels for DataOperation Job when it finished ? I agree with this.

Yes. All the affinity info DataFlow needs can be injected to annotations.

@cheyang
Copy link
Collaborator

cheyang commented Aug 22, 2024

@xliuqq @cheyang Is it possible to store affinity information in annotations instead of labels? From my understanding, annotations indicate which labels should be exposed, and dataflow-controller injects key=value pair to DataOperation job's labels. Can we also injecting key=value pair to DataOperation job's annotations?

I also think annotation is better than labels. Thanks.

@cheyang
Copy link
Collaborator

cheyang commented Aug 23, 2024

@xliuqq Could you please fix the conflict of source file? Thanks.

@cheyang cheyang self-requested a review August 26, 2024 02:20
@cheyang
Copy link
Collaborator

cheyang commented Aug 26, 2024

/test fluid-e2e

@cheyang cheyang requested a review from TrafalgarZZZ August 26, 2024 02:43
sdk/python Outdated Show resolved Hide resolved
Copy link

Copy link
Collaborator

@cheyang cheyang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@cheyang
Copy link
Collaborator

cheyang commented Aug 27, 2024

/test fluid-e2e

@TrafalgarZZZ
Copy link
Member

fluid-e2e failed because the port allocator randomly picked the same port for two runtimes. It's a temporary failure
/retest

Copy link
Member

@TrafalgarZZZ TrafalgarZZZ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

Copy link

fluid-e2e-bot bot commented Aug 27, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cheyang, TrafalgarZZZ

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [TrafalgarZZZ,cheyang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@fluid-e2e-bot fluid-e2e-bot bot merged commit b2e4e61 into fluid-cloudnative:master Aug 27, 2024
14 checks passed
@xliuqq xliuqq deleted the exposed-dataflow branch September 11, 2024 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants