Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(sdk): add qos limits capability to cloudian sdk #122

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions internal/sdk/cloudian/datasize.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
package cloudian

import (
"fmt"
)

// Construct by e.g. 3 * TB
type ByteSize uint64
Copy link
Member

@erikgb erikgb Jan 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this custom type in the SDK. To provide better usability, we could use https://pkg.go.dev/k8s.io/apimachinery/pkg/util/intstr#IntOrString in the provider API (Crossplane).

Or https://pkg.go.dev/k8s.io/apimachinery/pkg/api/resource#Quantity


const (
KB ByteSize = 1
MB = KB << 10
GB = MB << 10
TB = GB << 10
)

func (b ByteSize) KB() uint64 {
return uint64(b)
}

func (b ByteSize) MB() float64 {
v := b / MB
r := b % MB
return float64(v) + float64(r)/float64(MB)
}

func (b ByteSize) GB() float64 {
v := b / GB
r := b % GB
return float64(v) + float64(r)/float64(GB)
}

func (b ByteSize) TB() float64 {
v := b / TB
r := b % TB
return float64(v) + float64(r)/float64(TB)
}

func (b ByteSize) KBString() string {
if b == 0 {
return "0"
}
return fmt.Sprintf("%d", b/KB)
}
14 changes: 14 additions & 0 deletions internal/sdk/cloudian/datasize_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
package cloudian

import (
"fmt"
"testing"
)

func TestRenderTerraBytesAsKiloBytes(t *testing.T) {
expected := fmt.Sprintf("%d", 3*1024*1024*1024)

if actual := (3 * TB).KBString(); actual != expected {
t.Errorf("Expected 3 TB expressed in KB to be %s, got %s", expected, actual)
}
}
58 changes: 58 additions & 0 deletions internal/sdk/cloudian/sdk.go
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,30 @@ type SecurityInfo struct {
SecretKey string `json:"secretKey"`
}

// QoS is the Cloudian API's term for limits on quotas, counts and rates enforced on a `User`
type QoS struct {
// Max storage quota
StorageQuota ByteSize
// Warning limit storage quota
StorageQuotaWarning ByteSize
// Max storage quota in number of objects
StorageQuotaCount uint64
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To disable the limits, we need to support -1

Suggested change
StorageQuotaCount uint64
StorageQuotaCount int64

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could also make them pointer and transform nil to -1 and positive values to strings. Zero value would then be "unlimited", which is kinda nice

// Warning limit storage quota in number of objects
StorageQuotaCountWarning uint64
// Max nr of HTTP requests per minute
RequestRatePrMin uint64
// Warning limit nr of HTTP requests per minute
RequestRatePrMinWarning uint64
// Max inbound datarate in ByteSize per minute
DataRatePrMinInbound ByteSize
// Warning limit inbound datarate in ByteSize per minute
DataRatePrMinInboundWarning ByteSize
// Max outbound datarate in ByteSize per minute
DataRatePrMinOutbound ByteSize
// Warning limit outbound datarate in ByteSize per minute
DataRatePrMinOutboundWarning ByteSize
Copy link
Member

@tenstad tenstad Jan 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's long already, I think we can allow the e in Per

Suggested change
DataRatePrMinOutboundWarning ByteSize
DataRatePerMinOutboundWarning ByteSize

And maybe start with Outbound to make it more readable sentence rather than a postfix?

Suggested change
DataRatePrMinOutboundWarning ByteSize
OutboundDataRatePerMinWarning ByteSize

}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't you use the same approach as Kubernetes uses - with a map?

image

image


var ErrNotFound = errors.New("not found")

// WithInsecureTLSVerify skips the TLS validation of the server certificate when `insecure` is true.
Expand Down Expand Up @@ -405,6 +429,40 @@ func (client Client) GetGroup(ctx context.Context, groupID string) (*Group, erro
}
}

func qosQueryMap(user User, qos QoS) map[string]string {
return map[string]string{
"userId": user.UserID,
"groupId": user.GroupID,
"hlStorageQuotaKBytes": qos.StorageQuota.KBString(),
"wlStorageQuotaKBytes": qos.StorageQuotaWarning.KBString(),
"hlStorageQuotaCount": fmt.Sprintf("%d", qos.StorageQuotaCount),
"wlStorageQuotaCount": fmt.Sprintf("%d", qos.StorageQuotaCountWarning),
"hlRequestRate": fmt.Sprintf("%d", qos.RequestRatePrMin),
"wlRequestRate": fmt.Sprintf("%d", qos.RequestRatePrMinWarning),
Comment on lines +438 to +441
Copy link
Member

@tenstad tenstad Jan 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More readable to just use strconv.Itoa for these?

"hlDataKBytesIn": qos.DataRatePrMinInbound.KBString(),
"wlDataKBytesIn": qos.DataRatePrMinInboundWarning.KBString(),
"hlDataKBytesOut": qos.DataRatePrMinOutbound.KBString(),
"wlDataKBytesOut": qos.DataRatePrMinOutboundWarning.KBString(),
}
}

// CreateQuota sets the QoS limits for a `User`. To change QoS limits, a delete and recreate is necessary.
func (client Client) CreateQuota(ctx context.Context, user User, qos QoS) error {
resp, err := client.newRequest(ctx).
SetQueryParams(qosQueryMap(user, qos)).
Post("/qos/limits")
if err != nil {
return err
}

switch resp.StatusCode() {
case 200:
return nil
default:
return fmt.Errorf("SET quota unexpected status: %d", resp.StatusCode())
}
}

func (client Client) newRequest(ctx context.Context) *resty.Request {
return client.client.R().
SetContext(ctx).
Expand Down
37 changes: 37 additions & 0 deletions internal/sdk/cloudian/sdk_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,43 @@ func TestListUsers(t *testing.T) {

}

func TestQosQueryMap(t *testing.T) {
expected := map[string]string{
"userId": "1",
"groupId": "1",
"hlStorageQuotaKBytes": fmt.Sprintf("%d", 1*1024*1024),
"wlStorageQuotaKBytes": fmt.Sprintf("%d", 750*1024),
"hlStorageQuotaCount": "1000",
"wlStorageQuotaCount": "750",
"hlRequestRate": "200",
"wlRequestRate": "150",
"hlDataKBytesIn": fmt.Sprintf("%d", 1*1024*1024),
"wlDataKBytesIn": fmt.Sprintf("%d", 750*1024),
"hlDataKBytesOut": fmt.Sprintf("%d", 1*1024*1024),
"wlDataKBytesOut": fmt.Sprintf("%d", 750*1024),
}
actual := qosQueryMap(
User{
UserID: "1",
GroupID: "1",
},
QoS{
StorageQuota: 1 * GB,
StorageQuotaWarning: 750 * MB,
StorageQuotaCount: 1000,
StorageQuotaCountWarning: 750,
RequestRatePrMin: 200,
RequestRatePrMinWarning: 150,
DataRatePrMinInbound: 1 * GB,
DataRatePrMinInboundWarning: 750 * MB,
DataRatePrMinOutbound: 1 * GB,
DataRatePrMinOutboundWarning: 750 * MB,
})
if diff := cmp.Diff(expected, actual); diff != "" {
t.Errorf("qosQueryMap() mismatch (-want +got):\n%s", diff)
}
}

func mockBy(handler http.HandlerFunc) (*Client, *httptest.Server) {
mockServer := httptest.NewServer(handler)
return NewClient(mockServer.URL, ""), mockServer
Expand Down
Loading