Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add efficient search logic for large top-k users #2491

Merged

Conversation

kpango
Copy link
Collaborator

@kpango kpango commented May 2, 2024

Description

In the current Vald search architecture, each agent receives and processes queries broadcasted by the load balancer (LB) exactly as they are received. For example, with a configuration of Top-K=10 and Agent Replica=5, the LB reduces the results from 50 (produced collectively by the agents) down to the top 10.

However, when the Top-K value is extremely high, such as 100,000, it is inefficient to search across all K multiplied by the number of replicas. To address this, we have developed a feature that adjusts the number of requests forwarded to the agents based on a configurable ratio.

This pull request introduces a 'Ratio' setting within the Search Config. This setting allows for the dynamic adjustment of the proportion of the Top-K results that are propagated to the agents. If the 'Ratio' is not configured or set to 1, the system will maintain the current behavior of forwarding 100% of the Top-K results. If set to 0.5, only 50% of the Top-K will be searched. Conversely, setting the Ratio to 1.2 will result in 120% of the Top-K being searched, effectively increasing the breadth of the search under certain conditions.

Related Issue

Versions

  • Go Version: 1.22.2
  • Rust Version: 1.77.2
  • Docker Version: 20.10.8
  • Kubernetes Version: v1.29.3
  • NGT Version: 2.2.1

Checklist

Special notes for your reviewer

Summary by CodeRabbit

  • New Features

    • Introduced a new Dockerfile and README for the core agent.
    • Updated Kubernetes version across various GitHub templates to v1.30.0.
    • Enhanced download commands with -fsS and -fsSL options across multiple Makefiles for improved security and reliability.
    • Added new documentation images and updated installation commands in tutorials for Helm using secure curl options.
  • Bug Fixes

    • Fixed protocol issues in URLs for downloading tools like golangci-lint, reviewdog, kubectl, and buf.
  • Documentation

    • Updated Go version in example client module from 1.22.2 to 1.22.3.
    • Enhanced existing documentation to reflect updated download commands and Kubernetes version changes.
  • Refactor

    • Modified internal gRPC configuration to conditionally append health check durations, enhancing efficiency.

Copy link
Contributor

coderabbitai bot commented May 2, 2024

Warning

Rate Limit Exceeded

@kpango has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 1 minutes and 1 seconds before requesting another review.

How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.
Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.
Please see our FAQ for further information.

Commits Files that changed from the base of the PR and between 91475b4 and 6dee899.

Walkthrough

The repository has undergone extensive updates, primarily focused on enhancing security and reliability in file downloads across various Makefiles, with the addition of -fsSL flags to curl commands. This update also includes Kubernetes version bumps in GitHub templates and the introduction of new Dockerfiles and Rust components. Changes in GRPC configurations and dependency updates in Go modules are also notable, alongside modifications to improve logging and error handling in script executions.

Changes

Files Change Summary
.github/ISSUE_TEMPLATE/*, .github/PULL_REQUEST_TEMPLATE.md Updated Kubernetes version from v1.29.3 to v1.30.0.
Makefile, Makefile.d/* Added -fsSL options to curl commands for enhanced security and reliability. Introduced changes in version file logging and handling in scripts.
docs/tutorial/*, docs/user-guides/deployment.md Updated curl commands in Helm installation instructions to use -fsSL.
example/client/go.mod.default, hack/go.mod.default Updated Go and Kubernetes related dependencies to newer versions.
internal/config/*, internal/net/grpc/* Adjustments in GRPC configuration methods and test cases, including error handling and ticker management for health checks.
dockers/*, rust/* Added new Dockerfiles and initiated Rust components with necessary configurations and source files.
Various .md and .mk files General enhancements in logging, error display, and download command options across documentation and Makefiles.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@vdaas-ci
Copy link
Collaborator

vdaas-ci commented May 2, 2024

[CHATOPS:HELP] ChatOps commands.

  • 🙆‍♀️ /approve - approve
  • 🍱 /format - format codes and add licenses
  • /gen-test - generate test codes
  • 🏷️ /label - add labels
  • 🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

@vdaas-ci
Copy link
Collaborator

vdaas-ci commented May 2, 2024

[WARNING:INTCFG] Changes in interal/config may require you to change Helm charts. Please check.

Copy link

codecov bot commented May 2, 2024

Codecov Report

Attention: Patch coverage is 14.81994% with 615 lines in your changes are missing coverage. Please review.

Project coverage is 18.91%. Comparing base (9e091b1) to head (6dee899).
Report is 7 commits behind head on main.

Files Patch % Lines
apis/grpc/v1/payload/payload.pb.json.go 6.97% 320 Missing ⚠️
pkg/gateway/lb/handler/grpc/aggregation.go 0.00% 71 Missing ⚠️
pkg/gateway/lb/handler/grpc/handler.go 0.00% 57 Missing ⚠️
...is/grpc/v1/rpc/errdetails/error_details.pb.json.go 0.00% 56 Missing ⚠️
internal/net/grpc/option.go 0.00% 41 Missing ⚠️
apis/grpc/v1/payload/payload_vtproto.pb.go 0.00% 40 Missing ⚠️
internal/net/grpc/client.go 0.00% 26 Missing ⚠️
apis/grpc/v1/payload/payload.pb.go 0.00% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2491      +/-   ##
==========================================
- Coverage   18.94%   18.91%   -0.03%     
==========================================
  Files         563      565       +2     
  Lines       64707    65223     +516     
==========================================
+ Hits        12256    12338      +82     
- Misses      51656    52087     +431     
- Partials      795      798       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@kpango kpango force-pushed the feature/gateway-lb/add-efficient-search-for-large-topk-query branch from 884e28b to 2fc3e5e Compare May 2, 2024 12:33
Copy link

cloudflare-workers-and-pages bot commented May 2, 2024

Deploying vald with  Cloudflare Pages  Cloudflare Pages

Latest commit: 6dee899
Status: ✅  Deploy successful!
Preview URL: https://e0d86aef.vald.pages.dev
Branch Preview URL: https://feature-gateway-lb-add-effic.vald.pages.dev

View logs

@kpango kpango force-pushed the feature/gateway-lb/add-efficient-search-for-large-topk-query branch from 2fc3e5e to 391171e Compare May 2, 2024 12:47
internal/net/grpc/client.go Outdated Show resolved Hide resolved
internal/net/grpc/client.go Outdated Show resolved Hide resolved
coderabbitai[bot]
coderabbitai bot previously approved these changes May 2, 2024
@kpango kpango force-pushed the feature/gateway-lb/add-efficient-search-for-large-topk-query branch 2 times, most recently from 7e7960b to 91475b4 Compare May 8, 2024 05:25
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Out of diff range and nitpick comments (14)
docs/tutorial/vald-agent-standalone-on-k8s.md (6)

Line range hint 12-12: Add alternate text to the image for accessibility.

- <img src="../../assets/docs/tutorial/vald-agent-standalone-on-k8s.svg">
+ <img src="../../assets/docs/tutorial/vald-agent-standalone-on-k8s.svg" alt="Architecture of Vald Agent Standalone on Kubernetes">

Line range hint 20-20: Correct the link fragment to match the actual section ID.

- 1. [Check and Satisfy the Requirements](#Requirements)
+ 1. [Check and Satisfy the Requirements](#requirements)

Line range hint 21-21: Correct the link fragment to match the actual section ID.

- 1. [Prepare Kubernetes Cluster](#Prepare-the-Kubernetes-Cluster)
+ 1. [Prepare Kubernetes Cluster](#prepare-the-kubernetes-cluster)

Line range hint 22-22: Correct the link fragment to match the actual section ID.

- 1. [Deploy Vald Agent Standalone on Kubernetes cluster](#Deploy-Vald-Agent-Standalone-on-Kubernetes-Cluster)
+ 1. [Deploy Vald Agent Standalone on Kubernetes cluster](#deploy-vald-agent-standalone-on-kubernetes-cluster)

Line range hint 23-23: Correct the link fragment to match the actual section ID.

- 1. [Run Example Code](#Run-Example-Code)
+ 1. [Run Example Code](#run-example-code)

Line range hint 24-24: Correct the link fragment to match the actual section ID.

- 1. [Cleanup](#Cleanup)
+ 1. [Cleanup](#cleanup)
docs/tutorial/vald-multicluster-on-k8s.md (1)

Line range hint 15-15: Add alternate text to the image for accessibility.

- <img src="../../assets/docs/tutorial/vald-multicluster-on-k8s.png">
+ <img src="../../assets/docs/tutorial/vald-multicluster-on-k8s.png" alt="Architecture of Multiple Vald Clusters using Vald Mirror Gateway">
docs/tutorial/get-started.md (6)

Line range hint 15-15: Add alternate text to the image for accessibility.

- <img src="../../assets/docs/tutorial/getstarted.svg" />
+ <img src="../../assets/docs/tutorial/getstarted.svg" alt="Architecture of Vald Deployment for Getting Started" />

Line range hint 19-19: Correct the link fragment to match the actual section ID.

- 1. [Check and Satisfy the Requirements](#Requirements)
+ 1. [Check and Satisfy the Requirements](#requirements)

Line range hint 20-20: Correct the link fragment to match the actual section ID.

- 1. [Prepare Kubernetes Cluster](#Prepare-the-Kubernetes-Cluster)
+ 1. [Prepare Kubernetes Cluster](#prepare-the-kubernetes-cluster)

Line range hint 21-21: Correct the link fragment to match the actual section ID.

- 1. [Deploy Vald on Kubernetes Cluster](#Deploy-Vald-on-Kubernetes-Cluster)
+ 1. [Deploy Vald on Kubernetes Cluster](#deploy-vald-on-kubernetes-cluster)

Line range hint 22-22: Correct the link fragment to match the actual section ID.

- 1. [Run Example Code](#Run-Example-Code)
+ 1. [Run Example Code](#run-example-code)

Line range hint 23-23: Correct the link fragment to match the actual section ID.

- 1. [Cleanup](#Cleanup)
+ 1. [Cleanup](#cleanup)
pkg/gateway/lb/handler/grpc/handler.go (1)

Line range hint 468-494: Ensure comprehensive unit tests for doSearch.

This function is central to the search functionality and interacts with multiple components. It's crucial to have extensive unit tests covering various scenarios and configurations. Would you like assistance in setting up these tests?

Review Details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits Files that changed from the base of the PR and between 41c002c and 91475b4.
Files ignored due to path filters (25)
  • apis/grpc/v1/agent/core/agent.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/agent/sidecar/sidecar.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/discoverer/discoverer.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/filter/egress/egress_filter.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/filter/ingress/ingress_filter.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/manager/index/index_manager.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/mirror/mirror.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/payload/payload.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/payload/payload.pb.json.go is excluded by !**/*.pb.json.go
  • apis/grpc/v1/payload/payload_vtproto.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/rpc/errdetails/error_details.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/rpc/errdetails/error_details.pb.json.go is excluded by !**/*.pb.json.go
  • apis/grpc/v1/vald/filter.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/vald/flush.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/vald/insert.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/vald/object.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/vald/remove.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/vald/search.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/vald/update.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • apis/grpc/v1/vald/upsert.pb.go is excluded by !**/*.pb.go, !**/*.pb.go
  • example/client/go.mod is excluded by !**/*.mod
  • example/client/go.sum is excluded by !**/*.sum
  • go.mod is excluded by !**/*.mod
  • go.sum is excluded by !**/*.sum
  • rust/Cargo.lock is excluded by !**/*.lock
Files selected for processing (57)
  • .gitfiles (8 hunks)
  • .github/ISSUE_TEMPLATE/bug_report.md (1 hunks)
  • .github/ISSUE_TEMPLATE/security_issue_report.md (1 hunks)
  • .github/PULL_REQUEST_TEMPLATE.md (1 hunks)
  • Makefile (1 hunks)
  • Makefile.d/client.mk (1 hunks)
  • Makefile.d/dependencies.mk (2 hunks)
  • Makefile.d/functions.mk (1 hunks)
  • Makefile.d/helm.mk (3 hunks)
  • Makefile.d/k3d.mk (1 hunks)
  • Makefile.d/k8s.mk (1 hunks)
  • Makefile.d/kind.mk (1 hunks)
  • Makefile.d/minikube.mk (1 hunks)
  • Makefile.d/proto.mk (1 hunks)
  • Makefile.d/tools.mk (3 hunks)
  • apis/docs/v1/docs.md (1 hunks)
  • apis/proto/v1/payload/payload.proto (2 hunks)
  • apis/swagger/v1/vald/filter.swagger.json (1 hunks)
  • apis/swagger/v1/vald/search.swagger.json (1 hunks)
  • buf.gen.yaml (1 hunks)
  • charts/vald-benchmark-operator/values.schema.json (1 hunks)
  • charts/vald-helm-operator/crds/valdrelease.yaml (52 hunks)
  • charts/vald/values.schema.json (195 hunks)
  • charts/vald/values.yaml (6 hunks)
  • dockers/tools/benchmark/job/Dockerfile (2 hunks)
  • docs/tutorial/get-started.md (2 hunks)
  • docs/tutorial/vald-agent-standalone-on-k8s.md (1 hunks)
  • docs/tutorial/vald-multicluster-on-k8s.md (1 hunks)
  • docs/user-guides/deployment.md (1 hunks)
  • example/client/go.mod.default (1 hunks)
  • hack/go.mod.default (2 hunks)
  • internal/config/grpc.go (1 hunks)
  • internal/config/grpc_test.go (1 hunks)
  • internal/net/grpc/client.go (2 hunks)
  • internal/net/grpc/option.go (6 hunks)
  • internal/net/http/metrics/pprof.go (1 hunks)
  • internal/net/http/metrics/pprof_test.go (1 hunks)
  • internal/safety/safety.go (2 hunks)
  • k8s/agent/ngt/configmap.yaml (3 hunks)
  • k8s/discoverer/configmap.yaml (3 hunks)
  • k8s/discoverer/deployment.yaml (1 hunks)
  • k8s/gateway/gateway/lb/configmap.yaml (3 hunks)
  • k8s/gateway/gateway/lb/deployment.yaml (1 hunks)
  • k8s/manager/index/configmap.yaml (3 hunks)
  • k8s/manager/index/deployment.yaml (1 hunks)
  • k8s/operator/helm/crds/valdrelease.yaml (52 hunks)
  • pkg/gateway/lb/handler/grpc/aggregation.go (22 hunks)
  • pkg/gateway/lb/handler/grpc/handler.go (15 hunks)
  • pkg/gateway/lb/handler/grpc/search_benchmark_test.go (2 hunks)
  • rust/libs/proto/src/payload.v1.rs (1 hunks)
  • versions/GOLANGCILINT_VERSION (1 hunks)
  • versions/GO_VERSION (1 hunks)
  • versions/HELM_VERSION (1 hunks)
  • versions/JAEGER_OPERATOR_VERSION (1 hunks)
  • versions/K3S_VERSION (1 hunks)
  • versions/KUBECTL_VERSION (1 hunks)
  • versions/PROMETHEUS_STACK_VERSION (1 hunks)
Files not processed due to max files limit (16)
  • versions/REVIEWDOG_VERSION
  • versions/actions/ACTIONS_CHECKOUT
  • versions/actions/ACTIONS_DOWNLOAD_ARTIFACT
  • versions/actions/ACTIONS_SETUP_GO
  • versions/actions/ACTIONS_UPLOAD_ARTIFACT
  • versions/actions/AZURE_SETUP_HELM
  • versions/actions/CODECOV_CODECOV_ACTION
  • versions/actions/DOCKER_SETUP_BUILDX_ACTION
  • versions/actions/GITHUB_CODEQL_ACTION_ANALYZE
  • versions/actions/GITHUB_CODEQL_ACTION_AUTOBUILD
  • versions/actions/GITHUB_CODEQL_ACTION_INIT
  • versions/actions/GITHUB_CODEQL_ACTION_UPLOAD_SARIF
  • versions/actions/GITHUB_ISSUE_METRICS
  • versions/actions/REVIEWDOG_ACTION_HADOLINT
  • versions/actions/REVIEWDOG_ACTION_LANGUAGETOOL
  • versions/actions/SHOGO82148_ACTIONS_UPLOAD_RELEASE_ASSET
Files not summarized due to errors (1)
  • charts/vald/values.schema.json: Error: Message exceeds token limit
Files not reviewed due to errors (1)
  • pkg/gateway/lb/handler/grpc/aggregation.go (no review received)
Files skipped from review due to trivial changes (1)
  • dockers/tools/benchmark/job/Dockerfile
Files skipped from review as they are similar to previous changes (49)
  • .gitfiles
  • .github/ISSUE_TEMPLATE/bug_report.md
  • .github/ISSUE_TEMPLATE/security_issue_report.md
  • .github/PULL_REQUEST_TEMPLATE.md
  • Makefile
  • Makefile.d/client.mk
  • Makefile.d/dependencies.mk
  • Makefile.d/functions.mk
  • Makefile.d/helm.mk
  • Makefile.d/k3d.mk
  • Makefile.d/k8s.mk
  • Makefile.d/kind.mk
  • Makefile.d/minikube.mk
  • Makefile.d/proto.mk
  • Makefile.d/tools.mk
  • apis/proto/v1/payload/payload.proto
  • apis/swagger/v1/vald/filter.swagger.json
  • apis/swagger/v1/vald/search.swagger.json
  • buf.gen.yaml
  • charts/vald-benchmark-operator/values.schema.json
  • charts/vald-helm-operator/crds/valdrelease.yaml
  • charts/vald/values.yaml
  • docs/user-guides/deployment.md
  • example/client/go.mod.default
  • hack/go.mod.default
  • internal/config/grpc.go
  • internal/config/grpc_test.go
  • internal/net/grpc/client.go
  • internal/net/grpc/option.go
  • internal/net/http/metrics/pprof.go
  • internal/net/http/metrics/pprof_test.go
  • internal/safety/safety.go
  • k8s/agent/ngt/configmap.yaml
  • k8s/discoverer/configmap.yaml
  • k8s/discoverer/deployment.yaml
  • k8s/gateway/gateway/lb/configmap.yaml
  • k8s/gateway/gateway/lb/deployment.yaml
  • k8s/manager/index/configmap.yaml
  • k8s/manager/index/deployment.yaml
  • k8s/operator/helm/crds/valdrelease.yaml
  • pkg/gateway/lb/handler/grpc/search_benchmark_test.go
  • rust/libs/proto/src/payload.v1.rs
  • versions/GOLANGCILINT_VERSION
  • versions/GO_VERSION
  • versions/HELM_VERSION
  • versions/JAEGER_OPERATOR_VERSION
  • versions/K3S_VERSION
  • versions/KUBECTL_VERSION
  • versions/PROMETHEUS_STACK_VERSION
Additional Context Used
Learnings (1)
pkg/gateway/lb/handler/grpc/aggregation.go (4)
User: kpango
PR: vdaas/vald#2491
File: pkg/gateway/lb/handler/grpc/aggregation.go:500-505
Timestamp: 2024-05-08T02:19:58.067Z
Learning: User: kpango
PR: vdaas/vald#2491
File: pkg/gateway/lb/handler/grpc/aggregation.go:500-505
Timestamp: 2024-05-08T02:17:42.078Z
Learning: The `GetNum()` method in the `valdStdAggr` struct should include a nil check to prevent runtime panics when called on a nil instance. This is a defensive programming practice to enhance the robustness of the code.
User: kpango
PR: vdaas/vald#2491
File: pkg/gateway/lb/handler/grpc/aggregation.go:500-505
Timestamp: 2024-05-08T02:17:42.078Z
Learning: The `GetNum()` method in the `valdStdAggr` struct should include a nil check to prevent runtime panics when called on a nil instance. This is a defensive programming practice to enhance the robustness of the code.
User: kpango
PR: vdaas/vald#2491
File: pkg/gateway/lb/handler/grpc/aggregation.go:689-690
Timestamp: 2024-05-07T03:58:42.344Z
Learning: The direct slicing approach (`result = result[:v.fnum]`) is preferred over using `slices.Clip` for its simplicity and performance in the context of limiting the number of results in aggregation methods in `pkg/gateway/lb/handler/grpc/aggregation.go`.
User: kpango
PR: vdaas/vald#2491
File: pkg/gateway/lb/handler/grpc/handler.go:306-316
Timestamp: 2024-05-07T04:33:44.775Z
Learning: User: kpango
PR: vdaas/vald#2491
File: pkg/gateway/lb/handler/grpc/handler.go:306-316
Timestamp: 2024-05-07T04:15:11.886Z
Learning: In this project, vtprotobuf is used, which provides enhanced methods for protobuf objects: CloneVT (object cloning), EqualVT (object comparison), SizeVT (returns object size), MarshalVT (faster version of proto.Marshal), and UnmarshalVT (faster version of proto.Unmarshal). These methods offer better performance than usual protobuf usage.
Additional comments not posted (4)
apis/docs/v1/docs.md (1)

798-809: The addition of the ratio field in the search configuration is well-documented. Ensure that all services using this configuration are updated to handle this new field.

pkg/gateway/lb/handler/grpc/handler.go (3)

453-467: Optimize the calculation logic in calculateNum.


453-467: Optimize the calculation logic in calculateNum.


Line range hint 327-437: Verify comprehensive error handling in SearchByID.

This function includes multiple layers of error handling and status updates. It's crucial to verify through unit tests that all potential error scenarios are covered effectively. Here's a script to check the coverage of error handling paths:

apis/docs/v1/docs.md Show resolved Hide resolved
datelier
datelier previously approved these changes May 8, 2024
Makefile.d/tools.mk Outdated Show resolved Hide resolved
Makefile.d/tools.mk Outdated Show resolved Hide resolved
coderabbitai[bot]
coderabbitai bot previously approved these changes May 8, 2024
@kpango kpango dismissed stale reviews from coderabbitai[bot] and datelier via 9da530e May 8, 2024 05:55
coderabbitai[bot]
coderabbitai bot previously approved these changes May 8, 2024
@kpango kpango requested a review from kmrmt May 8, 2024 05:56
kmrmt
kmrmt previously approved these changes May 8, 2024
Copy link
Contributor

@kmrmt kmrmt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review Details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits Files that changed from the base of the PR and between 91475b4 and 9da530e.
Files selected for processing (2)
  • Makefile.d/k3d.mk (1 hunks)
  • Makefile.d/tools.mk (3 hunks)
Files skipped from review as they are similar to previous changes (2)
  • Makefile.d/k3d.mk
  • Makefile.d/tools.mk

@kpango kpango dismissed stale reviews from kmrmt and coderabbitai[bot] via 5cae30b May 8, 2024 06:16
@kpango kpango force-pushed the feature/gateway-lb/add-efficient-search-for-large-topk-query branch from 9da530e to 5cae30b Compare May 8, 2024 06:16
kmrmt
kmrmt previously approved these changes May 8, 2024
Signed-off-by: kpango <[email protected]>
@kpango kpango force-pushed the feature/gateway-lb/add-efficient-search-for-large-topk-query branch from 5cae30b to 6dee899 Compare May 8, 2024 06:22
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review Details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits Files that changed from the base of the PR and between 91475b4 and 5cae30b.
Files selected for processing (3)
  • Makefile.d/k3d.mk (1 hunks)
  • Makefile.d/kind.mk (1 hunks)
  • Makefile.d/tools.mk (3 hunks)
Files skipped from review as they are similar to previous changes (3)
  • Makefile.d/k3d.mk
  • Makefile.d/kind.mk
  • Makefile.d/tools.mk

@kpango kpango requested review from kmrmt and datelier May 8, 2024 06:43
@kmrmt kmrmt merged commit 6d4b505 into main May 8, 2024
177 of 178 checks passed
@kmrmt kmrmt deleted the feature/gateway-lb/add-efficient-search-for-large-topk-query branch May 8, 2024 06:49
vdaas-ci pushed a commit that referenced this pull request May 8, 2024
* add efficient search logic for large top-k users

Signed-off-by: kpango <[email protected]>

* fix bug

Signed-off-by: kpango <[email protected]>

---------

Signed-off-by: kpango <[email protected]>
hlts2 added a commit that referenced this pull request May 8, 2024
* fix: git add chart directory for release (#2356)

Signed-off-by: hlts2 <[email protected]>

* 🔖 🤖 Release v1.7.11 (#2360) (#2361)

* 🔖 🤖 Release v1.7.11



* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 578746e according to the output
from Gofumpt and Prettier.

Details: #2360

---------

Signed-off-by: Vdaas CI <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* Change docker scan timeout longer (#2363)

* refactor code using golangci-lint (#2362)

Signed-off-by: kpango <[email protected]>

* Create SECURITY.md (#2367)

* Create SECURITY.md

Signed-off-by: Yusuke Kato <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in e8609b0 according to the output
from Gofumpt and Prettier.

Details: #2367

---------

Signed-off-by: Yusuke Kato <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* add commit hash build image (#2359)

Signed-off-by: kpango <[email protected]>

* Make agent export index metrics to Pod k8s resource (#2319)

* Add config

* Add clusterRole and serviceAccount for agent

* Add more metrics

* Format

* Add export index info duration configuration

* Refactor internal/k8s

* Move apply function to internal/k8s

* Add more config

* Remove unused import and variable

* Use vald errors

* Add pod name and pod namespace to hack e2e agent

* Apply tagalign

* Revert unnecesarry specification of PodName

* Enable export index info to Kubernetes in agent templates only when the config is enabled

* Fix agent clusterrole name

* Update key name

* Update comments

* Add index count as export metrics

* Ignore gomnd

* Add index count entry to exportMetricsOnCreateIndex function

* Update values schema

* Fix formatter job

* Remove unused defines

* update docker build target platform selection rules (#2370)

Signed-off-by: kpango <[email protected]>
Co-authored-by: Yusuke Kadowaki <[email protected]>

* improve Backport Pull Request Title & Description Quality (#2373)

Signed-off-by: kpango <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>

* Update workflow to release readreplica chart (#2383)

* improve Backport Pull Request Title & Description Quality (#2373)

Signed-off-by: kpango <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>

* Add readreplica chart release

* update k8s/readreplica

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 70f1dad according to the output
from Gofumpt and Prettier.

Details: #2383

* [bugfix] fix backport github value bypass failure (#2378)

Signed-off-by: kpango <[email protected]>

* Revert backport.yaml

---------

Signed-off-by: kpango <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* 💚 ♻️ Add Con-Bench helm chart to the Vald charts (#2388)

Signed-off-by: vankichi <[email protected]>

* Delete unnecessary code for mirror (#2366)

* fix: delete unnecessary code for mirror

Signed-off-by: hlts2 <[email protected]>

* fix: make deps/install and format

Signed-off-by: hlts2 <[email protected]>

* improve Backport Pull Request Title & Description Quality (#2373)

Signed-off-by: kpango <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>

* fix: naming of variable

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Signed-off-by: kpango <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>

* change JP logo to EN logo (#2369)

Signed-off-by: kpango <[email protected]>
Co-authored-by: taisuou <[email protected]>

* Add rotate-all option to rotator (#2305)

* Add rotate-all option to rotator

* Fix linter warnings

* Make it possible to choose multiple rotation target

* Add tests for parseReplicaID

* Add tests for parseReplicaID

* Hide testify in internal

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in dd84674 according to the output
from Gofumpt and Prettier.

Details: #2305

* Apply format

* Lint

* Lint

---------

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Yusuke Kato <[email protected]>

* Resolve kvs already closed before last saving (#2390)

Signed-off-by: kpango <[email protected]>

* create continous benchmark doc (#2352)

* improve Backport Pull Request Title & Description Quality (#2373)

Signed-off-by: kpango <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>

* 📝 create continous benchmark doc

Signed-off-by: vankichi <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in ace001a according to the output
from Gofumpt and Prettier.

Details: #2352

* ♻️ rafactor default params

Signed-off-by: vankichi <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 9880dfc according to the output
from Gofumpt and Prettier.

Details: #2352

* ♻️ Fix format

Signed-off-by: vankichi <[email protected]>

* Revert "improve Backport Pull Request Title & Description Quality (#2373)"

This reverts commit 4246def.

* ♻️ remove dimension from con-bench

Signed-off-by: vankichi <[email protected]>

* 📝 Add sample benchmark manifest

Signed-off-by: vankichi <[email protected]>

* Update docs/performance/continuos-benchmatk.md

Co-authored-by: Yusuke Kato <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>

* Update docs/performance/continuos-benchmatk.md

Co-authored-by: Hiroto Funakoshi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>

---------

Signed-off-by: kpango <[email protected]>
Signed-off-by: vankichi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* fix: build error of internal kvs test (#2396)

Signed-off-by: hlts2 <[email protected]>

* 🤖 Update license headers / Format Go codes and YAML files (#2397)

Signed-off-by: Vdaas CI <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* fix: disable protobuf dispatch for client (#2401)

Signed-off-by: hlts2 <[email protected]>

* update deps (#2404)

Signed-off-by: kpango <[email protected]>

* 🔖 🤖 Release v1.7.12 (#2408) (#2409)

* 🔖 🤖 Release v1.7.12



* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 5bd8605 according to the output
from Gofumpt and Prettier.

Details: #2408

---------

Signed-off-by: Vdaas CI <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* Add tests for index information export (#2412)

* Add tests for index information export

* Add comments

* 📝 Fix typo of file name (#2413)

Signed-off-by: vankichi <[email protected]>

* Fix agent-faiss build failed (#2418)

* 🐳 Fix agent-faiss build failed

Signed-off-by: vankichi <[email protected]>

* 🐳 Fix COPY binary

Signed-off-by: vankichi <[email protected]>

* 🐳 Change COPY binary (use ld-linux-x86-64)

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>

* Fix the logic to determine docker image (#2410)

* fix: update action to get tag list

Signed-off-by: hlts2 <[email protected]>

* fix: refactor sub command execution

Signed-off-by: hlts2 <[email protected]>

* fix: use tag list in detect ci container workflow

Signed-off-by: hlts2 <[email protected]>

* fix: name for detected result

Signed-off-by: hlts2 <[email protected]>

* fix: invalid variable bug

Signed-off-by: hlts2 <[email protected]>

* fix: execution error and invalid variable

Signed-off-by: hlts2 <[email protected]>

* fix: deleted unnecessary comment

Signed-off-by: hlts2 <[email protected]>

* fix: deleted unnecessary parameters

Signed-off-by: hlts2 <[email protected]>

* fix: use new determine tag logic in the docker build workflow

Signed-off-by: hlts2 <[email protected]>

* fix: bugfix extra tags handling and small refactor

Signed-off-by: hlts2 <[email protected]>

* fix: build tag rule or release branch based on tag list

Signed-off-by: hlts2 <[email protected]>

* fix: typo

Signed-off-by: hlts2 <[email protected]>

* fix: indent

Signed-off-by: hlts2 <[email protected]>

* fix: refactor syntax

Signed-off-by: hlts2 <[email protected]>

* fix: typo and invalid option error

Signed-off-by: hlts2 <[email protected]>

* fix: use helm extra option

Signed-off-by: hlts2 <[email protected]>

* fix: add comment for tag management

Signed-off-by: hlts2 <[email protected]>

* fix: unsetbug of helm extra option

Signed-off-by: hlts2 <[email protected]>

* fix: build target condition

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* Update build rule for nightly image (#2421)

* fix: update build rule for nightly

Signed-off-by: hlts2 <[email protected]>

* fix: indent and deleted unnecessary tag

Signed-off-by: hlts2 <[email protected]>

* fix: update tag name when tag is unknown

Signed-off-by: hlts2 <[email protected]>

* fix: add secondary tags for build performance

Signed-off-by: hlts2 <[email protected]>

* fix: argument order

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* Add `index-operator` template implementation (#2375)

* improve Backport Pull Request Title & Description Quality (#2373)

Signed-off-by: kpango <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>

* [bugfix] fix backport github value bypass failure (#2378)

Signed-off-by: kpango <[email protected]>

* [bugfix] fix backport github value bypass failure (#2381)

Signed-off-by: kpango <[email protected]>

* Add index operator boilerplate

* Add k8s controller

* Add pod and job controllers

* Add annotations in the pod reconciler

* Update auto indexing service configurations

* Update WithOnReconcileFunc signature to include context

* Refactor debugging log

* Refactor reconcile funcs

* add nolint

* remove unused parameters

* lint

* Add dockerfile and build ci

* remove params

* Add index-operator image build command to Makefile

* Add index operator k8s templates

* Update helm schema

* Update readme

* Add internal/k8s as a trigger for images that depends on controller-runtime

* remove comment

* Update pkg/index/operator/service/operator.go

Co-authored-by: Yusuke Kato <[email protected]>
Signed-off-by: Yusuke Kadowaki <[email protected]>

* Update pkg/index/operator/service/operator.go

Co-authored-by: Yusuke Kato <[email protected]>
Signed-off-by: Yusuke Kadowaki <[email protected]>

* Refactor

* Add build commands

* Refactor

Co-authored-by: Yusuke Kato <[email protected]>

---------

Signed-off-by: kpango <[email protected]>
Signed-off-by: Yusuke Kadowaki <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>

* Fix output settings to determine-docker-image-tag action and release branch build tag name (#2423)

* fix: add outut settings for action

Signed-off-by: hlts2 <[email protected]>

* fix: tag name for release branch build

Signed-off-by: hlts2 <[email protected]>

* fix: refactor

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* fix: typo of execution rule (#2426)

Signed-off-by: hlts2 <[email protected]>

* Add flush API for deleting all vectors (#2273)

* add flush api

Signed-off-by: aknishid <[email protected]>

* added flush api

* fix typo

* fix pb.go & fix conflict

Signed-off-by: aknishid <[email protected]>

* add flush api

* add handler Flush

Signed-off-by: aknishid <[email protected]>

* add vald.Client flush

* fix filter gateway

Signed-off-by: aknishid <[email protected]>

* fix handler cnts

Signed-off-by: aknishid <[email protected]>

* delete unused var

Signed-off-by: aknishid <[email protected]>

* add author

Signed-off-by: Yamada Takuya <[email protected]>

* fix error

Signed-off-by: aknishid <[email protected]>

* fix flush.go & ngt.go

Signed-off-by: aknishid <[email protected]>

* add flush to example

Signed-off-by: aknishid <[email protected]>

* fix gateway lb

Signed-off-by: aknishid <[email protected]>

* update flush proto

Signed-off-by: aknishid <[email protected]>

* fix docs api README.md

Signed-off-by: aknishid <[email protected]>

* Fix flush ctx

Signed-off-by: aknishid <[email protected]>

* update docs & comment

Signed-off-by: aknishid <[email protected]>

* after make format

Signed-off-by: aknishid <[email protected]>

* add e2e

Signed-off-by: aknishid <[email protected]>

* update example & tutorial

Signed-off-by: aknishid <[email protected]>

* Update internal/errors/ngt.go

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* fix review

Signed-off-by: aknishid <[email protected]>

* modify status.md

Signed-off-by: aknishid <[email protected]>

* add error handling

Signed-off-by: aknishid <[email protected]>

* update proto

Signed-off-by: aknishid <[email protected]>

* fix doc

Signed-off-by: aknishid <[email protected]>

* ♻️ Fix proto

Signed-off-by: vankichi <[email protected]>

* use atomic bool

Signed-off-by: aknishid <[email protected]>

* fix proto

Signed-off-by: aknishid <[email protected]>

* update proto

Signed-off-by: aknishid <[email protected]>

* fix gateway error

Signed-off-by: aknishid <[email protected]>

* ✅ 🚧 Avoid e2e failing due to unimplemented handler and Add unimplmented handler method for the future

Signed-off-by: vankichi <[email protected]>

* Update apis/proto/v1/payload/payload.proto

Co-authored-by: Kiichiro YUKAWA <[email protected]>
Signed-off-by: aknishid <[email protected]>

---------

Signed-off-by: aknishid <[email protected]>
Signed-off-by: Yamada Takuya <[email protected]>
Signed-off-by: vankichi <[email protected]>
Signed-off-by: aknishid <[email protected]>
Co-authored-by: takuyaymd <[email protected]>
Co-authored-by: takuyyam <[email protected]>
Co-authored-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* docs: add aknishid as a contributor for code, and maintenance (#2431)

* docs: update README.md [skip ci]

* docs: update .all-contributorsrc [skip ci]

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in abc4c52 according to the output
from Gofumpt and Prettier.

Details: #2431

---------

Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* update deps & add validation for Flush API when agent is Read Only (#2433)

Signed-off-by: kpango <[email protected]>

* Update README.md (grammer) (#2440)

Signed-off-by: Hrichik Mazumder <[email protected]>

* docs: add hrichiksite as a contributor for doc (#2441)

* docs: update README.md [skip ci]

* docs: update .all-contributorsrc [skip ci]

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in b8054d9 according to the output
from Gofumpt and Prettier.

Details: #2441

---------

Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* Added snapshot timestamp annotations to read replica agent (#2428)

* Add lastTimeRotateReplicaTimestampAnnotationsKey constant

* Use ptr package

* Update annotations in ngt.go and rotator.go

* Add labels to Pod struct and update annotations handling***

***Introduce vald package for k8s related constants***

***Implement rotateIfNeeded function to check if read replica rotation is needed

* Refactor

* set sample log level debug

* Use k8s timestamp

* Refactor

* Format

* Add MatchingLabels method to ValdK8sClientMock

* Format

* Fix test

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 39b66fa according to the output
from Gofumpt and Prettier.

Details: #2428

* Rename constants

Co-authored-by: Yusuke Kato <[email protected]>

* Refactor rotateIfNeeded function to use label selector for listing deployments

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 6ace761 according to the output
from Gofumpt and Prettier.

Details: #2428

* FIx deepsource

* Revert

---------

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Yusuke Kato <[email protected]>

* fix: bugfix version update for docker build (#2445)

Signed-off-by: hlts2 <[email protected]>

* Fix index job logic to pass DNS A record (#2438)

* fix: choose user given target address in priority

Signed-off-by: hlts2 <[email protected]>

* fix: deleted unnecessary code

Signed-off-by: hlts2 <[email protected]>

* fix: add logic to connect the user given target addresses

Signed-off-by: hlts2 <[email protected]>

* fix: add info log about new connection target

Signed-off-by: hlts2 <[email protected]>

* fix: indent of configuration data

Signed-off-by: hlts2 <[email protected]>

* fix: indent bug

Signed-off-by: hlts2 <[email protected]>

* fix: deepsource warning

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* Fix operator-sdk version (#2447)

* fix: down grade operator version

Signed-off-by: hlts2 <[email protected]>

* fix: upgrade operator-sdk version

Signed-off-by: hlts2 <[email protected]>

* fix: operator-sdk version

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* add file name lint (#2417)

Signed-off-by: kpango <[email protected]>

* fix: add extra option for ci-container build (#2451)

Signed-off-by: hlts2 <[email protected]>

* Add base of benchmark operator dashboard (#2430)

* 📈 Add base of benchmark operator dashboard

Signed-off-by: vankichi <[email protected]>

* 💚 Fix build benchmark images trigger

Signed-off-by: vankichi <[email protected]>

* 💚 Add info tag for info metrics

Signed-off-by: vankichi <[email protected]>

* 📈 showing job image and tags

Signed-off-by: vankichi <[email protected]>

* 📈 add bench job status

Signed-off-by: vankichi <[email protected]>

* ♻️ Update dashboard and rename

Signed-off-by: vankichi <[email protected]>

* ♻️ Fix

Signed-off-by: vankichi <[email protected]>

* Update internal/observability/metrics/tools/benchmark/benchmark.go

Co-authored-by: Hiroto Funakoshi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Implement index operator logic for read replica rotation (#2444)

* Remove job reconciler

* Add leader election to index operator

* Add roles for leader election

* Add operator logic for rotation

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 844cca1 according to the output
from Gofumpt and Prettier.

Details: #2444

* Remove unnecesary change

* Fix build

* nits

* Refactor

* Fix clusterrole

* Move to podv2

* Add job concurrency check

* Refactor

* Refactor

* Fix heavy copying

* Update helm schema

* Fix package doc

* Remove unused code and refactor For() method

* Update target read replica ID in config

* Update internal/k8s/podv2/option.go

Co-authored-by: Hiroto Funakoshi <[email protected]>
Signed-off-by: Yusuke Kadowaki <[email protected]>

* Refactor

* rename podv2 to v2/pod

* Add client DI

* Fix envkey helper

* Set namespace from values.yaml

* Use annotations to specify the id

* Update schema

* Update sample

* Update values

* Update schema

* Fix spelling

* Fix package name

* Fix read replica e2e

* Add job templates for rotation, creation, saving, and correction

* Update config.go to decode k8s object

* Update sample.yaml for index operator

* Use job template from cfg

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 584d1f0 according to the output
from Gofumpt and Prettier.

Details: #2444

* Fix tagalign

* Fix config test

* Fix config test

* Disable read replica e2e for now

---------

Signed-off-by: Yusuke Kadowaki <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* add inner product distance type for ngt (#2454)

Signed-off-by: kpango <[email protected]>

* Fix e2e for read replica and add e2e for index operator (#2455)

* Fix read replica e2e

* Update default image list in action.yaml

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 6cdac9f according to the output
from Gofumpt and Prettier.

Details: #2455

---------

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* Add unit tests for index operator (#2460)

* Add prototype tests

* Refactor

* Fix not to use kubeconfig

* 🤖 Automatically update PULL_REQUEST_TEMPLATE and ISSUE_TEMPLATE (#2457)

Signed-off-by: Vdaas CI <[email protected]>
Co-authored-by: Kiichiro YUKAWA <[email protected]>

* Bugfix recreate benchmark job when operator reboot (#2463)

* 🐛 Fix recreate benchmark job already runnnig when operator rebooted

Signed-off-by: vankichi <[email protected]>

* 🐛 Fix benchmarkJobReconcile status handling

Signed-off-by: vankichi <[email protected]>

* ✅ Fix test

Signed-off-by: vankichi <[email protected]>

* ♻️ Update k8s dir

Signed-off-by: vankichi <[email protected]>

* ♻️ Fix

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>

* Refactor k8s types (#2462)

* Refactor types

* Refactor hdf5_test.go

* Format

* Refactor

* add extra flags for NGT build phase (#2467)

* Fix workflow trigger for backport pr creation (#2471)

* Automatically add backport main label for release-pr (#2473)

* feat: automatically add backport main label for release-pr

Signed-off-by: Hiroto Funakoshi <[email protected]>

* fix: deleted unnecessary quotation

Signed-off-by: Hiroto Funakoshi <[email protected]>

---------

Signed-off-by: Hiroto Funakoshi <[email protected]>

* update deps (#2468)

* update deps

Signed-off-by: kpango <[email protected]>

* fix server_test.go due to the grpc-go updates

Signed-off-by: kpango <[email protected]>

---------

Signed-off-by: kpango <[email protected]>

* Update README.md (#2479)

Signed-off-by: Yusuke Kato <[email protected]>

* Implement client metrics interceptor for continuous benchmark job (#2477)

* ♻️ Mount benchmark-operator-config to vald-benchmark-job Pod

Signed-off-by: vankichi <[email protected]>

* 📈 add framework for send gRPC client metrics

Signed-off-by: vankichi <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in a532b7d according to the output
from Gofumpt and Prettier.

Details: #2477

* 📈 Add client metric interceptor

Signed-off-by: vankichi <[email protected]>

* ♻️ Change vbor chart for apply general job configuration

Signed-off-by: vankichi <[email protected]>

* Update pkg/tools/benchmark/job/usecase/benchmarkd.go

Co-authored-by: Hiroto Funakoshi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>

* 🐛 Fix build error and add comment

Signed-off-by: vankichi <[email protected]>

* ♻️ Fix format

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* implement ngt-rs and its unit test and add GitHub Actions workflows (#2478)

* implement test

* update rust dependencies

* fix

* make format

* add Github Actions workflows

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

---------

Co-authored-by: Yusuke Kato <[email protected]>

* 📈 Add client metrics panels for continuous benchmark job (#2481)

Signed-off-by: vankichi <[email protected]>

* Add rust versions (#2482)

* add rust versions

* fix

* add rust version

* add rust-toolchain.toml

* make format

* Update continuous benchmark docs (#2485)

* 📝 Update continous benchmark docs

Signed-off-by: vankichi <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in e3ab1d2 according to the output
from Gofumpt and Prettier.

Details: #2485

* Update docs/performance/continuous-benchmark.md

Signed-off-by: Kiichiro YUKAWA <[email protected]>

* 📝 Fix

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* update NGT version (#2484)

* fix sed line (#2488)

* 🤖 Automatically update PULL_REQUEST_TEMPLATE and ISSUE_TEMPLATE (#2487)

Signed-off-by: Vdaas CI <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add efficient search logic for large top-k users (#2491)

* add efficient search logic for large top-k users

Signed-off-by: kpango <[email protected]>

* fix bug

Signed-off-by: kpango <[email protected]>

---------

Signed-off-by: kpango <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Signed-off-by: Vdaas CI <[email protected]>
Signed-off-by: kpango <[email protected]>
Signed-off-by: Yusuke Kato <[email protected]>
Signed-off-by: vankichi <[email protected]>
Signed-off-by: Kiichiro YUKAWA <[email protected]>
Signed-off-by: Yusuke Kadowaki <[email protected]>
Signed-off-by: aknishid <[email protected]>
Signed-off-by: Yamada Takuya <[email protected]>
Signed-off-by: aknishid <[email protected]>
Signed-off-by: Hrichik Mazumder <[email protected]>
Signed-off-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: vdaas-ci <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Yusuke Kadowaki <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Kosuke Morimoto <[email protected]>
Co-authored-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: taisho <[email protected]>
Co-authored-by: taisuou <[email protected]>
Co-authored-by: aknishid <[email protected]>
Co-authored-by: takuyaymd <[email protected]>
Co-authored-by: takuyyam <[email protected]>
Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
Co-authored-by: Hrichik Mazumder <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment