Update README and documentation

[ChenYi015]

🛑 Important:

Please open an issue to discuss significant work before you start. We appreciate your contributions and don't want your efforts to go to waste!

For guidelines on how to contribute, please review the CONTRIBUTING.md document.

Purpose of this PR

Provide a clear and concise description of the changes. Explain the motivation behind these changes and link to relevant issues or discussions.

Proposed changes:

• Use mkdocs-material to generate static website
• Add a new github action workflow for mkdocs to deploy website to github pages
• Remove docs/*.md and keep only the docs/api-docs.md file.
• Redirect links in README.md to the kubeflow website
• Rename docs/who-is-using.md to ADOPTERS.md and sort the names in alphabetical order

Change Category

Indicate the type of change by marking the applicable boxes:

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

Checklist

Before submitting your PR, please review the following:

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

[vara-bonthu]

Thanks for setting up this static website. I initially considered using Docusaurus, but managing the documentation with MkDocs seems easier.

[vara-bonthu]

/assign @yuchaoran2011

[andreyvelich]

Thank you for updating this Spark Operator docs @ChenYi015! This is great!

I think, we discussed before that we are planning to migrate Spark Operator docs to the Kubeflow website: #1928.
Similar to other Kubeflow components (Model Registry, Pipelines, Training Operator, etc.)
So users can get information about the components from the single place: https://www.kubeflow.org/docs/components/

Also, Kubeflow docs leads can help you with docs review, etc.

/cc @StefanoFioravanzo @hbelmiro @kubeflow/release-team

[thesuperzapper]

@andreyvelich there is a small precedent for having an external website for some docs, namely the KFP SDK.

The main reason to consider using a separate site would be if there are going to be lots of different versions with significant changes between them (as we don't really have the ability to version each component separately on the Kubeflow website).

But obviously, we should try and make it work on the Kubeflow website, if possible.

[vara-bonthu]

@andreyvelich What is the process for migrating documentation to the Kubeflow docs? Which repository should we raise a PR in? Do we have control over approving future documentation changes?

Since @ChenYi015 has done most of the work, it would be helpful to provide him with pointers so he can convert this into a Kubeflow Docs PR.

[thesuperzapper]

@vara-bonthu you can make a PR on the website repo https://github.com/kubeflow/website

It's pretty much the same as MkDocs (uses markdown) but uses Hugo with the Docsy theme.

It should be very straightforward to just copy what has been proposed here into a new section under "components".

[andreyvelich]

@andreyvelich there is a small precedent for having an external website for some docs, namely the KFP SDK.

@thesuperzapper Yes, I am aware of these docs. This is mostly done because KFP is using automation to generate SDK docs from their SDK client: https://github.com/kubeflow/pipelines/tree/master/docs. It is hard to auto generate SDK docs in kubeflow/website repo since we run SDK generator script in the source repo (e.g. pipelines, training-operator)

The main reason to consider using a separate site would be if there are going to be lots of different versions

Yes, this is valid point that components might need to be versioned separately from the website version. We discussed before with @StefanoFioravanzo that ideally docs for the components should live in their repo, and we should configure our Hugo website to render them. But we haven't got a chance to discuss how we can do it.

The main goal is to make sure our users can easily navigate in Kubeflow ecosystem to get desired value.

What is the process for migrating documentation to the Kubeflow docs? Which repository should we raise a PR in? Do we have control over approving future documentation changes?

You should raise a PR under content/en/docs/components/spark-operator website directory.
Yes, you will have full control to approve your PR, since we will create OWNERs file under this directory with Spark Operator maintainers.
E.g. like in Training Operator: https://github.com/kubeflow/website/blob/master/content/en/docs/components/training/OWNERS

[vara-bonthu]

@ChenYi015 Are you looking to raise a PR to this repo https://github.com/kubeflow/website/tree/master/content/en/docs/components? Let me know if you need a help so that I can move the docs to the new repo.

[ChenYi015]

@ChenYi015 Are you looking to raise a PR to this repo https://github.com/kubeflow/website/tree/master/content/en/docs/components? Let me know if you need a help so that I can move the docs to the new repo.

@vara-bonthu Sorry for the late response, I have raised a PR #3767 to move docs to kubeflow website .

[hbelmiro]

@ChenYi015 is this PR still needed since you opened a new PR to the website repo?

[andreyvelich]

@ChenYi015 already opened this PR: kubeflow/website#3767

[hbelmiro]

@ChenYi015 already opened this PR: kubeflow/website#3767

@andreyvelich
Yeah, that's my point. Can't we close this in favor of kubeflow/website#3767?

[andreyvelich]

That sounds good.
@ChenYi015 Do you want to modify this PR to update the README to redirect users to the Spark Operators docs on Kubeflow website: https://www.kubeflow.org/docs/components/ ?

[vara-bonthu]

That sounds good. @ChenYi015 Do you want to modify this PR to update the README to redirect users to the Spark Operators docs on Kubeflow website: https://www.kubeflow.org/docs/components/ ?

That's a good point. We can do that once the docs are published under Kubeflow docs. Let's wait until this PR#3767 is merged

[ChenYi015]

That sounds good. @ChenYi015 Do you want to modify this PR to update the README to redirect users to the Spark Operators docs on Kubeflow website: https://www.kubeflow.org/docs/components/ ?

That's a good point. We can do that once the docs are published under Kubeflow docs. Let's wait until this PR#3767 is merged

@vara-bonthu I have redirected links in README to the kubeflow spark operator docs.

[andreyvelich]

That sounds good. @ChenYi015 Do you want to modify this PR to update the README to redirect users to the Spark Operators docs on Kubeflow website: https://www.kubeflow.org/docs/components/ ?

That's a good point. We can do that once the docs are published under Kubeflow docs. Let's wait until this PR#3767 is merged

@vara-bonthu I have redirected links in README to the kubeflow spark operator docs.

Thank you @ChenYi015!
Do we want to remove these docs from Spark Operator GitHub repo since we have them on Kubeflow website: https://www.kubeflow.org/docs/components/spark-operator/ ?

[ChenYi015]

That sounds good. @ChenYi015 Do you want to modify this PR to update the README to redirect users to the Spark Operators docs on Kubeflow website: https://www.kubeflow.org/docs/components/ ?

That's a good point. We can do that once the docs are published under Kubeflow docs. Let's wait until this PR#3767 is merged

@vara-bonthu I have redirected links in README to the kubeflow spark operator docs.

Thank you @ChenYi015! Do we want to remove these docs from Spark Operator GitHub repo since we have them on Kubeflow website: https://www.kubeflow.org/docs/components/spark-operator/ ?

@andreyvelich I had removed the docs except for docs/api-docs.md.

[andreyvelich]

Thank you for this huge effort @ChenYi015!
/lgtm
/assign @vara-bonthu @yuchaoran2011

[andreyvelich]

/hold

For @vara-bonthu and @yuchaoran2011 review

[vara-bonthu]

/lgtm

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vara-bonthu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vara-bonthu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[andreyvelich]

@vara-bonthu @ChenYi015 Do we want to add the community meeting notes to the readme as I mentioned here: #2047 (comment) ?

[ChenYi015]

@vara-bonthu @ChenYi015 Do we want to add the community meeting notes to the readme as I mentioned here: #2047 (comment) ?

@andreyvelich I had added a link to the community call in the README.

[andreyvelich]

Thank you for this @ChenYi015!
/hold cancel
/lgtm