Bgd 5296 marge latest kubeflow spark operator #19

sigmarkarl · 2024-06-05T10:46:55Z

Merge spark operator upstream master to keep up to date

Jira Ticket

Checklist:

I have filled relevant self assessment (NodeJS, Frontend, Backend)
I have run ESlint on my changes and fixed all warnings and errors (NodeJS & Frontend Services)
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have validated all the requirements in the Jira task were answered
I have all neccessary approvals for the design/mini design of this task
I have approved the API changes and granular permission patterns (documentation subtask) (For public services only)

Signed-off-by: André Bauer <[email protected]>

Added new RBAC permissions needed by default for leader election for the coordination/v1 API. Required after upgrade to golang:1.19.2. In k8s.io/[email protected]/tools/leaderelection/resourcelock/interface.go:166 `configMapsResourceLock` was removed and should be replaced by `ConfigMapsLeasesResourceLock`.

* Added support for setting extra commonLabels * Added support for podLabels on cleanup and init job * Fixed templating errors * Added documentation

Used to run Analytics Jobs and ETL pipelines along with AI/ML jobs.

* add sidecars for operator * bumping chart version

This notes the controller intended to be used with the operator-managed ingress resources. When setting up my own cluster I also tried https://docs.nginx.com/nginx-ingress-controller/ and https://kubernetes-sigs.github.io/aws-load-balancer-controller/ but the path format generated by the operator won't work with those.

* Add envFrom to operator deployment Useful to when env vars are used for auth when downloading `spark.archives` from S3. * Fix over-indenting

Resolves kubeflow#1344 Spark 3.4 supports IPv6: - apache/spark#36868 So I want to make the operator support IPv6. I can confirm that this can submit the spark-job in IPv6-only environment. Although it is necessary to add the following environment variables to the operator ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: spark-on-k8s-spark-operator spec: template: spec: containers: - name: spark-operator env: - name: _JAVA_OPTIONS value: "-Djava.net.preferIPv6Addresses=true" - name: KUBERNETES_DISABLE_HOSTNAME_VERIFICATION value: "true" ```

…necessary for Kubernetes Pod Security Standards Restricted profile. (kubeflow#1768) https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted * Fixed pre-commit jobs. `build-helm-chart` and `integration-test` were failing with: `Run manusa/[email protected] Error: Unsupported OS, action only works in Ubuntu 18 or 20`

Co-authored-by: Yinan Li <[email protected]>

Co-authored-by: Volodymyr Kot <[email protected]>

…lcano on OCP (kubeflow#1724) * fix: fix issue kubeflow#1723 about spark-operator not working with volcano on OCP Signed-off-by: disaster37 <[email protected]> * Update volcano_scheduler.go --------- Signed-off-by: disaster37 <[email protected]>

https://github.com/kubeflow/community/issues/684

Create OWNERS

kubeflow#1946) Signed-off-by: Dmitriy Karbyshev <[email protected]>

Signed-off-by: fazledyn-or <[email protected]>

…and service name (kubeflow#1778) * Installing operator using kustomize and custom namespace and service name * update quick start guide with suggested changes.

Signed-off-by: Yi Chen <[email protected]>

…ow#2016) Signed-off-by: Yi Chen <[email protected]>

Signed-off-by: Cian Gallagher <[email protected]>

…beflow#2043) * feat: give an option to set the priority class for spark-operator pod Signed-off-by: Praveen Gajulapalli <[email protected]> * feat: bumped up helm chart version Signed-off-by: Praveen Gajulapalli <[email protected]> * fix: fixed issue with position of priorityClassName Signed-off-by: Praveen Gajulapalli <[email protected]>

* feat: add support for setting objectSelector on webhook Signed-off-by: Cian Gallagher <[email protected]> * feat: update objectSelector to match expressions Signed-off-by: Cian Gallagher <[email protected]> * chore: use out of the box label parser Signed-off-by: Cian Gallagher <[email protected]> * chore: update chart version Signed-off-by: Cian Gallagher <[email protected]> * chore: update app version Signed-off-by: Cian Gallagher <[email protected]> * fix: use parseSelector Signed-off-by: Cian Gallagher <[email protected]> * ci: update minikube action to latest release Signed-off-by: Cian Gallagher <[email protected]> * revert: undo ci changes. create seperate pr Signed-off-by: Cian Gallagher <[email protected]> * Trigger CI Signed-off-by: Cian Gallagher <[email protected]> * chore: update chart version & docs following previous merge Signed-off-by: Cian Gallagher <[email protected]> * docs: update docs Signed-off-by: Cian Gallagher <[email protected]> --------- Signed-off-by: Cian Gallagher <[email protected]>

…w#2066) Signed-off-by: Vikas Saxena <[email protected]>

Signed-off-by: Jacob Salway <[email protected]>

* Add CNCF Code of Conduct Signed-off-by: Yi Chen <[email protected]> * Update contributing guide Signed-off-by: Yi Chen <[email protected]> * Redirect links to kubeflow website Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]>

* Update docs Signed-off-by: Yi Chen <[email protected]> * Remove docs and update README Signed-off-by: Yi Chen <[email protected]> * Add link to monthly community meeting Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]>

* Add PodDisruptionBudget to chart Signed-off-by: Carlos Sánchez Páez <[email protected]> Signed-off-by: Carlos Sánchez Páez <[email protected]> Signed-off-by: Carlos Sánchez Páez <[email protected]> * PR comments Signed-off-by: Carlos Sánchez Páez <[email protected]> --------- Signed-off-by: Carlos Sánchez Páez <[email protected]> Signed-off-by: Carlos Sánchez Páez <[email protected]>

Signed-off-by: Carlos Sánchez Páez <[email protected]>

* Add workflow for closing staled issues and PRs Signed-off-by: Yi Chen <[email protected]> * Add job permissions Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]>

…kubeflow#2046) * Update .gitignore Signed-off-by: Yi Chen <[email protected]> * Update .dockerignore Signed-off-by: Yi Chen <[email protected]> * Update Makefile Signed-off-by: Yi Chen <[email protected]> * Update the process to generate api docs Signed-off-by: Yi Chen <[email protected]> * Update the workflow to generate api docs Signed-off-by: Yi Chen <[email protected]> * Use controller-gen to generate CRD and deep copy related methods Signed-off-by: Yi Chen <[email protected]> * Update helm chart CRDs Signed-off-by: Yi Chen <[email protected]> * Update workflow for building spark operator Signed-off-by: Yi Chen <[email protected]> * Update README.md Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]>

…lly (kubeflow#2087) Signed-off-by: Yi Chen <[email protected]>

Signed-off-by: Yi Chen <[email protected]>

Signed-off-by: Andrey Velichkevich <[email protected]>

Signed-off-by: Yi Chen <[email protected]>

* Update README and documentation (kubeflow#2047) * Update docs Signed-off-by: Yi Chen <[email protected]> * Remove docs and update README Signed-off-by: Yi Chen <[email protected]> * Add link to monthly community meeting Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]> Signed-off-by: jbhalodia-slack <[email protected]> * Add PodDisruptionBudget to chart (kubeflow#2078) * Add PodDisruptionBudget to chart Signed-off-by: Carlos Sánchez Páez <[email protected]> Signed-off-by: Carlos Sánchez Páez <[email protected]> Signed-off-by: Carlos Sánchez Páez <[email protected]> * PR comments Signed-off-by: Carlos Sánchez Páez <[email protected]> --------- Signed-off-by: Carlos Sánchez Páez <[email protected]> Signed-off-by: Carlos Sánchez Páez <[email protected]> Signed-off-by: jbhalodia-slack <[email protected]> * Set topologySpreadConstraints Signed-off-by: jbhalodia-slack <[email protected]> * Update README and increase patch version Signed-off-by: jbhalodia-slack <[email protected]> * Revert replicaCount change Signed-off-by: jbhalodia-slack <[email protected]> * Update README after master merger Signed-off-by: jbhalodia-slack <[email protected]> * Update README Signed-off-by: jbhalodia-slack <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]> Signed-off-by: jbhalodia-slack <[email protected]> Signed-off-by: Carlos Sánchez Páez <[email protected]> Signed-off-by: Carlos Sánchez Páez <[email protected]> Co-authored-by: Yi Chen <[email protected]> Co-authored-by: Carlos Sánchez Páez <[email protected]>

* Update .helmignore Signed-off-by: Yi Chen <[email protected]> * Add release docs Signed-off-by: Yi Chen <[email protected]> * Update release workflow Signed-off-by: Yi Chen <[email protected]> * Update integration test workflow Signed-off-by: Yi Chen <[email protected]> * Add workflow for pushing tag when VERSION file changes Signed-off-by: Yi Chen <[email protected]> * Update Signed-off-by: Yi Chen <[email protected]> * Remove the leading 'v' from chart version Signed-off-by: Yi Chen <[email protected]> * Update docker image tags Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]>

* Use controller-runtime to reconstruct spark operator Signed-off-by: Yi Chen <[email protected]> * Update helm charts Signed-off-by: Yi Chen <[email protected]> * Update examples Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]>

Signed-off-by: Yi Chen <[email protected]>

antonipp and others added 30 commits March 17, 2023 08:55

Fix signal handling for non-leader processes (kubeflow#1680)

bc05d55

add lifecycle to executor (kubeflow#1674)

e7361ba

Add Kognita to "Who is using" (kubeflow#1637)

fff3ba6

Add support for ephemeral.volumeClaimTemplate in helm chart CRDs (k…

3dc7765

…ubeflow#1661)

add dependabot (kubeflow#1629)

e5c45e2

Signed-off-by: André Bauer <[email protected]>

fix tolerations block in wrong segment for webhook jobs (kubeflow#1633)

a733749

Extra helm chart labels (kubeflow#1669)

72481c0

* Added support for setting extra commonLabels * Added support for podLabels on cleanup and init job * Fixed templating errors * Added documentation

Molex started using spark K8 operator. (kubeflow#1714)

5f2efd4

Used to run Analytics Jobs and ETL pipelines along with AI/ML jobs.

Roblox who-is (kubeflow#1784)

5b7a6e3

Optional sidecars for operator pod (kubeflow#1754)

d596205

* add sidecars for operator * bumping chart version

Add envFrom to operator deployment (kubeflow#1785)

1b842a8

* Add envFrom to operator deployment Useful to when env vars are used for auth when downloading `spark.archives` from S3. * Fix over-indenting

Link to helm upgrade docs fixed (kubeflow#1783)

223ec8d

Updating webhook docs to also mention eks (kubeflow#1763)

93a9aa8

allowing optional annotation on rbac (kubeflow#1770)

f21a4e5

Add qualytics to who is using (kubeflow#1736)

678d313

Co-authored-by: Yinan Li <[email protected]>

Fix docs for Volcano integration (kubeflow#1719)

e146848

Co-authored-by: Volodymyr Kot <[email protected]>

Handle invalid API resources in discovery (kubeflow#1758)

cd78251

Add Rokt to who-is-using.md (kubeflow#1867)

b49b1b4

Create OWNERS

0a44823

https://github.com/kubeflow/community/issues/684

Merge pull request kubeflow#1927 from kubeflow/zijianjoy-patch-1

5478052

Create OWNERS

Allow to set webhook job resource limits (kubeflow#1429,kubeflow#1300) (

ec1c2e3

kubeflow#1946) Signed-off-by: Dmitriy Karbyshev <[email protected]>

Add PVC permission to Operator role (kubeflow#1889)

717a3ec

remove unmatched quotes from user-guide.md (kubeflow#1584)

0dd08b2

BUGFIX: Added cancel method to fix context leak (kubeflow#1917)

14a39e6

Signed-off-by: fazledyn-or <[email protected]>

README for installing operator using kustomize with custom namespace …

43f489d

…and service name (kubeflow#1778) * Installing operator using kustomize and custom namespace and service name * update quick start guide with suggested changes.

ChenYi015 and others added 3 commits June 5, 2024 02:11

Bump appVersion to v1beta2-1.5.0-3.5.0 (kubeflow#2044)

2219563

Signed-off-by: Yi Chen <[email protected]>

Add ChenYi015 as spark-operator reviewer (kubeflow#2045)

089546e

Signed-off-by: Yi Chen <[email protected]>

merge upstream master

adfcb73

sigmarkarl requested review from a team as code owners June 5, 2024 10:46

ChenYi015 and others added 23 commits June 5, 2024 14:39

Certifictes are generated by operator rather than gencerts.sh (kubefl…

5ce3dba

…ow#2016) Signed-off-by: Yi Chen <[email protected]>

Update minikube version in CI (kubeflow#2059)

8f7e325

Signed-off-by: Cian Gallagher <[email protected]>

Modified README.MD as per changes discussed on kubeflow#2062 (kubeflo…

d37ac38

…w#2066) Signed-off-by: Vikas Saxena <[email protected]>

Remove .gitlab-ci.yml (kubeflow#2069)

012b52a

Signed-off-by: Jacob Salway <[email protected]>

Update helm docs (kubeflow#2081)

eca3fc8

Signed-off-by: Carlos Sánchez Páez <[email protected]>

Add workflow for closing stale issues and PRs (kubeflow#2073)

8894a4f

* Add workflow for closing staled issues and PRs Signed-off-by: Yi Chen <[email protected]> * Add job permissions Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]>

Add CHANGELOG.md file and use python script to generate it automatica…

bc9dcc2

…lly (kubeflow#2087) Signed-off-by: Yi Chen <[email protected]>

Add @ChenYi015 to approvers (kubeflow#2096)

b27717d

Signed-off-by: Yi Chen <[email protected]>

Update Stale bot settings (kubeflow#2095)

461ddc9

Signed-off-by: Andrey Velichkevich <[email protected]>

Add Alibaba Cloud to adopters (kubeflow#2097)

51e4886

Signed-off-by: Yi Chen <[email protected]>

Fix broken integration test CI (kubeflow#2109)

6ff204a

Signed-off-by: Yi Chen <[email protected]>

Fix CI: environment variable BRANCH is missed (kubeflow#2111)

e2693f1

Signed-off-by: Yi Chen <[email protected]>

merge upstream master

fb99d91

libcap2

c42456a

alextarasov-spot closed this Aug 11, 2024

alextarasov-spot deleted the BGD-5296-marge-latest-kubeflow-spark-operator branch September 17, 2024 10:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bgd 5296 marge latest kubeflow spark operator #19

Bgd 5296 marge latest kubeflow spark operator #19

sigmarkarl commented Jun 5, 2024 •

edited by jira bot

Loading

Bgd 5296 marge latest kubeflow spark operator #19

Bgd 5296 marge latest kubeflow spark operator #19

Conversation

sigmarkarl commented Jun 5, 2024 • edited by jira bot Loading

Jira Ticket

Checklist:

sigmarkarl commented Jun 5, 2024 •

edited by jira bot

Loading