[DO NOT MERGE][Experiment] RayService updates status frequently

[kevin85421]

Why are these changes needed?

# Build Docker image
make docker-image

# Install KubeRay operator
helm install kuberay-operator kuberay/kuberay-operator --version 0.5.0 --set image.repository=controller,image.tag=latest

# Create a RayService (path: ray-operator/config/samples)
kubectl apply -f ray_v1alpha1_rayservice.yaml

# Collect operator logs after 10 mins
kubectl logs kuberay-operator-xxxxxx | tee operator-log
cat operator-log | grep "r.Status().Update()" | tee experiment_result

• r.Status().Update() is called 287 times in 10 mins.
  • 41 times: RayCluster is not ready.

    2023-05-02T21:46:18.091Z	INFO	controllers.RayService	r.Status().Update() updateState	{"error": "Put \"http://rayservice-sample-raycluster-7wgsc-dashboard-svc.default.svc.cluster.local:52365/api/serve/deployments/\": dial tcp 10.96.145.214:52365: connect: connection refused"}
    

    • kuberay/ray-operator/controllers/ray/utils/dashboard_httpclient.go

      Lines 209 to 212 in 3571d52

      	req, err := http.NewRequest(http.MethodPut, r.dashboardURL+DeployPath, bytes.NewBuffer(deploymentJson))
      	if err != nil {
      	return err
      	}

  • 7 times

    2023-05-02T21:46:27.278Z	INFO	controllers.RayService	r.Status().Update() isHealthy && !isReady
    

  • 239 times

    2023-05-02T21:46:42.046Z	INFO	controllers.RayService	r.Status().Update() Final status update for any CR modification.
    

Related issue number

Experiments for #1061

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(