Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup: Use ResourceFlavor.metadata.labels #353

Closed
wants to merge 1 commit into from
Closed

Cleanup: Use ResourceFlavor.metadata.labels #353

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 0 additions & 8 deletions apis/kueue/v1alpha2/resourceflavor_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,6 @@ type ResourceFlavor struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`

// labels associated with this flavor. They are matched against or
// converted to node affinity constraints on the workload’s pods.
// For example, cloud.provider.com/accelerator: nvidia-tesla-k80.
// More info: http://kubernetes.io/docs/user-guide/labels
//
// labels can be up to 8 elements.
Comment on lines -32 to -37
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can something similar be added above ObjectMeta? Would that be included in the CRD yaml?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding single or multi-line comment does not generate description for ObjectMeta in CRD yaml using make manifests command.
metav1.ObjectMeta struct is embedded inside ResourceFlavor struct. It's not a separate field defined. Don't know of any other way to add for .metadata.labels.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. That's fine then.

Labels map[string]string `json:"labels,omitempty"`

// taints associated with this flavor that workloads must explicitly
// “tolerate” to be able to use this flavor.
// For example, cloud.provider.com/preemptible="true":NoSchedule
Expand Down
7 changes: 0 additions & 7 deletions apis/kueue/v1alpha2/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 0 additions & 9 deletions config/components/crd/bases/kueue.x-k8s.io_resourceflavors.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,6 @@ spec:
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
labels:
additionalProperties:
type: string
description: "labels associated with this flavor. They are matched against
or converted to node affinity constraints on the workload’s pods. For
example, cloud.provider.com/accelerator: nvidia-tesla-k80. More info:
http://kubernetes.io/docs/user-guide/labels \n labels can be up to 8
elements."
type: object
metadata:
type: object
taints:
Expand Down
6 changes: 3 additions & 3 deletions docs/concepts/cluster_queue.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,8 +163,8 @@ apiVersion: kueue.x-k8s.io/v1alpha1
kind: ResourceFlavor
metadata:
name: spot
labels:
instance-type: spot
labels:
instance-type: spot
taints:
- effect: NoSchedule
key: spot
Expand All @@ -177,7 +177,7 @@ ClusterQueue in the `.spec.resources[*].flavors[*].name` field.
### ResourceFlavor labels

To associate a ResourceFlavor with a subset of nodes of you cluster, you can
configure the `.labels` field with matching node labels that uniquely identify
configure the `.metadata.labels` field with matching node labels that uniquely identify
the nodes. If you are using [cluster autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler)
(or equivalent controllers), make sure it is configured to add those labels when
adding new nodes.
Expand Down
8 changes: 4 additions & 4 deletions docs/tasks/administer_cluster_quotas.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,8 +143,8 @@ apiVersion: kueue.x-k8s.io/v1alpha1
kind: ResourceFlavor
metadata:
name: x86
labels:
cpu-arch: x86
labels:
cpu-arch: x86
```

```yaml
Expand All @@ -153,8 +153,8 @@ apiVersion: kueue.x-k8s.io/v1alpha1
kind: ResourceFlavor
metadata:
name: arm
labels:
cpu-arch: arm
labels:
cpu-arch: arm
```

To create the ResourceFlavors, run the following command:
Expand Down
18 changes: 12 additions & 6 deletions pkg/cache/cache_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,8 +103,10 @@ func TestCacheClusterQueueOperations(t *testing.T) {
}
setup := func(cache *Cache) {
cache.AddOrUpdateResourceFlavor(&kueue.ResourceFlavor{
ObjectMeta: metav1.ObjectMeta{Name: "default"},
Labels: map[string]string{"cpuType": "default"},
ObjectMeta: metav1.ObjectMeta{
Name: "default",
Labels: map[string]string{"cpuType": "default"},
},
})
for _, c := range initialClusterQueues {
if err := cache.AddClusterQueue(context.Background(), &c); err != nil {
Expand Down Expand Up @@ -189,8 +191,10 @@ func TestCacheClusterQueueOperations(t *testing.T) {
}
}
cache.AddOrUpdateResourceFlavor(&kueue.ResourceFlavor{
ObjectMeta: metav1.ObjectMeta{Name: "default"},
Labels: map[string]string{"cpuType": "default"},
ObjectMeta: metav1.ObjectMeta{
Name: "default",
Labels: map[string]string{"cpuType": "default"},
},
})
},
wantClusterQueues: map[string]*ClusterQueue{
Expand Down Expand Up @@ -302,8 +306,10 @@ func TestCacheClusterQueueOperations(t *testing.T) {
}
}
cache.AddOrUpdateResourceFlavor(&kueue.ResourceFlavor{
ObjectMeta: metav1.ObjectMeta{Name: "default"},
Labels: map[string]string{"cpuType": "default", "region": "central"},
ObjectMeta: metav1.ObjectMeta{
Name: "default",
Labels: map[string]string{"cpuType": "default", "region": "central"},
},
})
},
wantClusterQueues: map[string]*ClusterQueue{
Expand Down
26 changes: 16 additions & 10 deletions pkg/cache/snapshot_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,16 +152,19 @@ func TestSnapshot(t *testing.T) {
}
flavors := []kueue.ResourceFlavor{
{
ObjectMeta: metav1.ObjectMeta{Name: "demand"},
Labels: map[string]string{"foo": "bar", "instance": "demand"},
ObjectMeta: metav1.ObjectMeta{
Name: "demand",
Labels: map[string]string{"foo": "bar", "instance": "demand"},
},
},
{
ObjectMeta: metav1.ObjectMeta{Name: "spot"},
Labels: map[string]string{"baz": "bar", "instance": "spot"},
ObjectMeta: metav1.ObjectMeta{
Name: "spot",
Labels: map[string]string{"baz": "bar", "instance": "spot"},
},
},
{
ObjectMeta: metav1.ObjectMeta{Name: "default"},
Labels: nil,
},
}

Expand Down Expand Up @@ -381,15 +384,18 @@ func TestSnapshot(t *testing.T) {
ResourceFlavors: map[string]*kueue.ResourceFlavor{
"default": {
ObjectMeta: metav1.ObjectMeta{Name: "default"},
Labels: nil,
},
"demand": {
ObjectMeta: metav1.ObjectMeta{Name: "demand"},
Labels: map[string]string{"foo": "bar", "instance": "demand"},
ObjectMeta: metav1.ObjectMeta{
Name: "demand",
Labels: map[string]string{"foo": "bar", "instance": "demand"},
},
},
"spot": {
ObjectMeta: metav1.ObjectMeta{Name: "spot"},
Labels: map[string]string{"baz": "bar", "instance": "spot"},
ObjectMeta: metav1.ObjectMeta{
Name: "spot",
Labels: map[string]string{"baz": "bar", "instance": "spot"},
},
},
},
InactiveClusterQueueSets: sets.String{"flavor-nonexistent-cq": {}},
Expand Down
24 changes: 16 additions & 8 deletions pkg/scheduler/scheduler_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -885,20 +885,28 @@ func TestEntryAssignFlavors(t *testing.T) {
ObjectMeta: metav1.ObjectMeta{Name: "default"},
},
"one": {
ObjectMeta: metav1.ObjectMeta{Name: "one"},
Labels: map[string]string{"type": "one"},
ObjectMeta: metav1.ObjectMeta{
Name: "one",
Labels: map[string]string{"type": "one"},
},
},
"two": {
ObjectMeta: metav1.ObjectMeta{Name: "two"},
Labels: map[string]string{"type": "two"},
ObjectMeta: metav1.ObjectMeta{
Name: "two",
Labels: map[string]string{"type": "two"},
},
},
"b_one": {
ObjectMeta: metav1.ObjectMeta{Name: "b_one"},
Labels: map[string]string{"b_type": "one"},
ObjectMeta: metav1.ObjectMeta{
Name: "b_one",
Labels: map[string]string{"b_type": "one"},
},
},
"b_two": {
ObjectMeta: metav1.ObjectMeta{Name: "b_two"},
Labels: map[string]string{"b_type": "two"},
ObjectMeta: metav1.ObjectMeta{
Name: "b_two",
Labels: map[string]string{"b_type": "two"},
},
},
"tainted": {
ObjectMeta: metav1.ObjectMeta{Name: "tainted"},
Expand Down
4 changes: 2 additions & 2 deletions pkg/util/testing/wrappers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -385,9 +385,9 @@ type ResourceFlavorWrapper struct{ kueue.ResourceFlavor }
func MakeResourceFlavor(name string) *ResourceFlavorWrapper {
return &ResourceFlavorWrapper{kueue.ResourceFlavor{
ObjectMeta: metav1.ObjectMeta{
Name: name,
Name: name,
Labels: map[string]string{},
},
Labels: map[string]string{},
}}
}

Expand Down
11 changes: 8 additions & 3 deletions test/integration/webhook/v1alpha2/resourceflavor_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,11 @@ limitations under the License.
package v1alpha2

import (
"fmt"

"github.com/onsi/ginkgo/v2"
"github.com/onsi/gomega"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
Expand Down Expand Up @@ -68,7 +69,7 @@ var _ = ginkgo.Describe("ResourceFlavor Webhook", func() {
resourceFlavor := testing.MakeResourceFlavor("resource-flavor").Label("foo", "@abcd").Obj()
err := k8sClient.Create(ctx, resourceFlavor)
gomega.Expect(err).Should(gomega.HaveOccurred())
gomega.Expect(errors.IsForbidden(err)).Should(gomega.BeTrue(), "error: %v", err)
gomega.Expect(err.Error()).To(gomega.ContainSubstring(errorInvalidLabelValue("@abcd")))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error returned is not of type forbidden. Hence, added the error substring. Any other way to assert the error?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there should be an error type that matches in the package.

The substrings could change upstream, so I rather not pay attention to that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably errors.IsBadRequest

})
})

Expand All @@ -92,7 +93,11 @@ var _ = ginkgo.Describe("ResourceFlavor Webhook", func() {
ginkgo.By("Updating the resourceFlavor with invalid labels")
err := k8sClient.Update(ctx, &created)
gomega.Expect(err).Should(gomega.HaveOccurred())
gomega.Expect(errors.IsForbidden(err)).Should(gomega.BeTrue(), "error: %v", err)
gomega.Expect(err.Error()).To(gomega.ContainSubstring(errorInvalidLabelValue("@abcd")))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert

})
})
})

func errorInvalidLabelValue(labelValue string) string {
return fmt.Sprintf("ResourceFlavor.kueue.x-k8s.io \"resource-flavor\" is invalid: metadata.labels: Invalid value: \"%s\"", labelValue)
}