-
Notifications
You must be signed in to change notification settings - Fork 442
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add RayJob docs and development docs #404
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| ## KubeRay Development Guidance | ||
|
|
||
| Download this repo locally | ||
|
|
||
| ``` | ||
| mkdir -p $GOPATH/src/github.com/ray-project | ||
| cd $GOPATH/src/github.com/ray-project | ||
| git clone https://github.com/ray-project/kuberay.git | ||
| ``` | ||
|
|
||
| ### Develop proto and OpenAPI | ||
|
|
||
| Generate go clients and swagger file | ||
|
|
||
| ``` | ||
| make generate | ||
| ``` | ||
|
|
||
| ### Develop KubeRay Operator | ||
|
|
||
| ``` | ||
| cd ray-operator | ||
|
|
||
| # Build codes | ||
| make build | ||
|
|
||
| # Run test | ||
| make test | ||
|
|
||
| # Build container image | ||
| make docker-build | ||
| ``` | ||
|
|
||
| ### Develop KubeRay APIServer | ||
|
|
||
| ``` | ||
| cd apiserver | ||
|
|
||
| # Build code | ||
| go build cmd/main.go | ||
| ``` | ||
|
|
||
| ### Develop KubeRay CLI | ||
|
|
||
| ``` | ||
| cd cli | ||
| go build -o kuberay -a main.go | ||
| ./kuberay help | ||
| ``` | ||
|
|
||
| ### Deploy Docs locally | ||
|
|
||
| We don't need to configure `mkdocs` environment, to check static website locally, run command | ||
|
|
||
| ``` | ||
| docker run --rm -it -p 8000:8000 -v ${PWD}:/docs squidfunk/mkdocs-material build | ||
| ``` |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| # Observability | ||
|
|
||
| ### Monitor | ||
|
|
||
| We have added a parameter `--metrics-expose-port=8080` to open the port and expose metrics both for the ray cluster and our control plane. We also leverage the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) to start the whole monitoring system. | ||
|
|
||
| You can quickly deploy one by the following on your own kubernetes cluster by using the scripts in install: | ||
|
|
||
| ```shell | ||
| ./install/prometheus/install.sh | ||
| ``` | ||
| It will set up the prometheus stack and deploy the related service monitor in `config/prometheus` | ||
|
|
||
| Then you can also use the json in `config/grafana` to generate the dashboards. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,127 @@ | ||
| ## Ray Job (alpha) | ||
|
|
||
| > Note: This is the alpha version of Ray Job Support in KubeRay. There will be ongoing improvements for Ray Job in the future releases. | ||
|
|
||
| ### Prerequisite | ||
|
|
||
| * Ray 1.10 and above. | ||
| * KubeRay v0.3.0 or master | ||
|
|
||
| ### What is a RayJob? | ||
|
|
||
| The RayJob is a new custom resource (CR) supported by KubeRay in v0.3.0. | ||
|
|
||
| A RayJob manages 2 things: | ||
| * RayCluster: Manages resources in kubernetes cluster. | ||
| * Job: Manages users' job in ray cluster. | ||
|
|
||
| ### What does the RayJob provide? | ||
|
|
||
| * Kubernetes-native support for Ray cluster and Ray Job. You can use a kubernetes config to define a ray cluster and jobs in ray cluster. Then you can use `kubectl` to create the cluster and its job. The cluster can be deleted automatically after the job is finished. | ||
|
|
||
|
|
||
| ### Deploy the KubeRay | ||
|
|
||
| Make sure KubeRay v0.3.0 version is deployed in your cluster. | ||
| For installation details, please check [guidance](../deploy/installation.md) | ||
|
|
||
| ### Run an example Job | ||
|
|
||
| There is one example config file to deploy RayJob included here: | ||
| [ray_v1alpha1_rayjob.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray_v1alpha1_rayjob.yaml) | ||
|
|
||
| ```shell | ||
| # Create a ray job. | ||
| $ kubectl apply -f config/samples/ray_v1alpha1_rayjob.yaml | ||
| ``` | ||
|
|
||
| ```shell | ||
| # List running RayJobs. | ||
| $ kubectl get rayjob | ||
| NAME AGE | ||
| rayjob-sample 7s | ||
| ``` | ||
|
|
||
| ```shell | ||
| # RayJob sample underneath will create a raycluster | ||
| # raycluster will create few resources including pods, services, you can type commands to have a check | ||
| $ kubectl get rayclusters | ||
| $ kubectl get pod | ||
| ``` | ||
|
|
||
| ### RayJob Configuration | ||
|
|
||
| - `entrypoint` - The shell command to run for this job. job_id. | ||
| - `jobId` - Optional. Job ID to specify for the job. If not provided, one will be generated. | ||
| - `metadata` - Arbitrary user-provided metadata for the job. | ||
| - `runtimeEnv` - base64 string of the runtime json string. | ||
| - `shutdownAfterJobFinishes` - whether to recycle the cluster after job finishes. | ||
| - `ttlSecondsAfterFinished` - TTL to clean up the cluster. This is only working if `shutdownAfterJobFinishes` is set. | ||
|
|
||
| ### RayJob Observability | ||
|
|
||
| You can use `kubectl logs` to check the operator logs or the head/worker nodes logs. | ||
| You can also use `kubectl describe rayjobs rayjob-sample` to check the states and event logs of your RayJob instance. | ||
|
|
||
| ``` | ||
| Status: | ||
| Dashboard URL: rayjob-sample-raycluster-vnl8w-head-svc.ray-system.svc.cluster.local:8265 | ||
| End Time: 2022-07-24T02:04:56Z | ||
| Job Deployment Status: Complete | ||
| Job Id: test-hehe | ||
| Job Status: SUCCEEDED | ||
| Message: Job finished successfully. | ||
| Ray Cluster Name: rayjob-sample-raycluster-vnl8w | ||
| Ray Cluster Status: | ||
| Available Worker Replicas: 1 | ||
| Endpoints: | ||
| Client: 32572 | ||
| Dashboard: 32276 | ||
| Gcs - Server: 30679 | ||
| Last Update Time: 2022-07-24T02:04:43Z | ||
| State: ready | ||
| Start Time: 2022-07-24T02:04:49Z | ||
| Events: | ||
| Type Reason Age From Message | ||
| ---- ------ ---- ---- ------- | ||
| Normal Created 90s rayjob-controller Created cluster rayjob-sample-raycluster-vnl8w | ||
| Normal Submitted 82s rayjob-controller Submit Job test-hehe | ||
| Normal Deleted 15s rayjob-controller Deleted cluster rayjob-sample-raycluster-vnl8w | ||
| ``` | ||
|
|
||
|
|
||
| If the job can not successfully run, you can see from the status as well. | ||
| ``` | ||
| Status: | ||
| Dashboard URL: rayjob-sample-raycluster-nrdm8-head-svc.ray-system.svc.cluster.local:8265 | ||
| End Time: 2022-07-24T02:01:39Z | ||
| Job Deployment Status: Complete | ||
| Job Id: test-hehe | ||
| Job Status: FAILED | ||
| Message: Job failed due to an application error, last available logs: | ||
| python: can't open file '/tmp/code/script.ppy': [Errno 2] No such file or directory | ||
|
|
||
| Ray Cluster Name: rayjob-sample-raycluster-nrdm8 | ||
| Ray Cluster Status: | ||
| Available Worker Replicas: 1 | ||
| Endpoints: | ||
| Client: 31852 | ||
| Dashboard: 32606 | ||
| Gcs - Server: 32436 | ||
| Last Update Time: 2022-07-24T02:01:30Z | ||
| State: ready | ||
| Start Time: 2022-07-24T02:01:38Z | ||
| Events: | ||
| Type Reason Age From Message | ||
| ---- ------ ---- ---- ------- | ||
| Normal Created 2m9s rayjob-controller Created cluster rayjob-sample-raycluster-nrdm8 | ||
| Normal Submitted 2m rayjob-controller Submit Job test-hehe | ||
| Normal Deleted 58s rayjob-controller Deleted cluster rayjob-sample-raycluster-nrdm8 | ||
| ``` | ||
|
|
||
|
|
||
| ### Delete the RayJob instance | ||
|
|
||
| ```shell | ||
| $ kubectl delete -f config/samples/ray_v1alpha1_rayjob.yaml | ||
| ``` | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should explain what the operational difference is between a Ray Service and a Ray Job.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add some details. The cluster can be deleted by RayJob controller once it succeed or failed