Update manifests warning

[juliusvonkohout]

@andreyvelich this PR adds the 1.8.1 release and clarifies the support level of kubeflow/manifests for users.

Follow up of #3724 (comment)
and #3724 (comment)

[juliusvonkohout]

CC @jbottum @terrytangyuan

[juliusvonkohout]

CC @kubeflow/wg-manifests-leads @kimwnasptd @kubeflow/kubeflow-steering-committee

[google-oss-prow]

@juliusvonkohout: GitHub didn't allow me to assign the following users: wg-manifests.

Note that only kubeflow members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign wg-manifests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[juliusvonkohout]

/assign @kubeflow/wg-manifests-leads

[jbottum]

/lgtm

[thesuperzapper]

I am happy to make some changes here, but I think we can improve the wording, please see my comments below.

[juliusvonkohout]

I think i have addressed all comments.
@akgraner @jbottum @andreyvelich @kubeflow/kubeflow-steering-committee please review again.

[rimolive]

/lgtm

[thesuperzapper]

@juliusvonkohout replying to #3748 (comment) here, because comments get removed after each commit.

The currently proposed warning is:

Using the Kubeflow manifests requires basic Kubernetes knowledge.
The Kubeflow community support for Kubeflow manifests is only best-effort and not guaranteed for environment-specific issues or custom configurations.

For context, my suggested warning is this:

Using the Kubeflow Manifests in production requires working knowledge of Kubernetes, Istio, and Kubeflow itself.
When using the Kubeflow manifests, the community is not able to provide support for environment-specific issues or custom configurations. 
If you need support, please consider using a [packaged distribution](#packaged-distributions).

---

@juliusvonkohout to reply to your specific comments on my proposal:

I think Basic Kubernetes knowledge is enough. You do not need to understand the details of Istio to get it running for example. Many users install and use it succesfully without understanding Istio.
"Kubeflow itself" does not make sense to me, because you need to install it first to become familiar with Kubeflow.

The goal of the warning is to give useful context for new users, note that my suggested change is "Using the Kubeflow Manifests in *production*...", that warning does not preclude people who are just testing it out.

Implying that users only need "basic knowledge" of Kubernetes will result in many people becoming frustrated. Kubeflow is not a simple system, and it's critical that we set expectations about it and what knowledge is required to use it successfully.

---

We do provide best effort support on Slack and in the repository/issue tracker, so that is not true.

While people like yourself might provide environment-specific support (and I am sure users are very thankful), there is no official support for any environment-specific issues, and by stating it on the website we are speaking as the project.

It's important that users understand they will be largely supporting themselves if they use the manifests, otherwise they create a burden on the project, we should suggest should reach out to a vendor (like yourself) if they need support.

[juliusvonkohout]

@juliusvonkohout replying to #3748 (comment) here, because comments get removed after each commit.

The currently proposed warning is:

Using the Kubeflow manifests requires basic Kubernetes knowledge.
The Kubeflow community support for Kubeflow manifests is only best-effort and not guaranteed for environment-specific issues or custom configurations.

For context, my suggested warning is this:

Using the Kubeflow Manifests in production requires working knowledge of Kubernetes, Istio, and Kubeflow itself.
When using the Kubeflow manifests, the community is not able to provide support for environment-specific issues or custom configurations. 
If you need support, please consider using a [packaged distribution](#packaged-distributions).

@juliusvonkohout to reply to your specific comments on my proposal:

I think Basic Kubernetes knowledge is enough. You do not need to understand the details of Istio to get it running for example. Many users install and use it succesfully without understanding Istio.
"Kubeflow itself" does not make sense to me, because you need to install it first to become familiar with Kubeflow.

The goal of the warning is to give useful context for new users, note that my suggested change is "Using the Kubeflow Manifests in *production*...", that warning does not preclude people who are just testing it out.

Implying that users only need "basic knowledge" of Kubernetes will result in many people becoming frustrated. Kubeflow is not a simple system, and it's critical that we set expectations about it and what knowledge is required to use it successfully.

We do provide best effort support on Slack and in the repository/issue tracker, so that is not true.

While people like yourself might provide environment-specific support (and I am sure users are very thankful), there is no official support for any environment-specific issues, and by stating it on the website we are speaking as the project.

It's important that users understand they will be largely supporting themselves if they use the manifests, otherwise they create a burden on the project, we should suggest should reach out to a vendor (like yourself) if they need support.

@thesuperzapper I do not agree with all of the reasoning in your text, but i still tried to adjust it a second time into your direction without demoting the open source project itself below third-party distributions/consultants or making it a sales page. Please check out the latest commit.

We also got support (/lgtm) for this change here from KSC members (e.g. @jbottum) as well as other distribution owners such as Redhat (@rimolive). I also talked about the changes with @andreyvelich.

{{% alert title="Warning" color="warning" %}}
Using the Kubeflow manifests to install Kubeflow requires basic Kubernetes knowledge.
The Kubeflow community support for Kubeflow manifests is only best-effort, non-commercial
and not guaranteed for environment-specific issues or custom configurations.
Nevertheless, we welcome contributions and bug reports very much.
For commercial production-level usage and support there are many options.  
You can use a third-party commercial distribution, hire consultants or build up the knowledge yourself.
{{% /alert %}}

[thesuperzapper]

@juliusvonkohout I think we are getting closer.

However, I think it's easier to read if we can split that into two paragraphs: one about why people use the manifests (and the level of support we can provide), and the other about what users might consider for production usage.

Here is my proposal, I think it sets new-user expectations well, while not discounting the effort you put into making the manifests:

{{% alert title="Warning" color="warning" %}}
While the manifests are a quick way to start using the Kubeflow Platform, the Kubeflow community only provides best-effort, non-commercial support for manifests deployments.
We are unable to provide support for environment-specific issues or custom configurations. 
Nevertheless, we welcome contributions and bug reports very much.

Successfully using the manifests in production requires a working knowledge of Kubernetes, Istio, and Kubeflow itself.
If you need support, there are many options available.
For example, you may use a [packaged distribution](#packaged-distributions), hire consultants, or build up the knowledge yourself.
{{% /alert %}}

[andreyvelich]

Thank you for creating this @juliusvonkohout!
My thoughts on this.

Should we try to investigate what other Kubernetes projects are doing ?
For example, kubeadm says:
kubeadm performs the actions necessary to get a minimum viable cluster up and running..
So we can say:
Kubeflow manifests performs the actions necessary to get a minimum viable Kubeflow platform up and running.
But maybe we should explore what others are doing

@thesuperzapper to your point about:

and the other about what users might consider for production usage.

I think, it depends on the user. If they are capable enough to install Kubeflow components using kustomize manifests and configure other things, why they should explore Kubeflow distributions ? Similar to how people deploy Kubernetes on-prem vs Kubernetes distributions.
With Kubernetes distribution you get a lot of great features on top, similar to Kubeflow distributions. And Kubeflow distributions should clearly explain the user value in their docs.

For commercial production-level usage and support there are many options. You can use a third-party commercial distribution, hire consultants or build up the knowledge yourself.

@juliusvonkohout @thesuperzapper Do we really need to say this in Kubeflow manifests section ? Why our users can't understand by themselves if they want to take Kubeflow distribution or Kubeflow manifests for their work ?

I would like to get feedback from distribution owners as well.

/assign @kubeflow/kubeflow-steering-committee @StefanoFioravanzo @hbelmiro @ca-scribner @gkcalat @zijianjoy @Linchin @Tomcli @yhwang @johnugeorge @nagar-ajay @rimolive @thesuperzapper @liuqi @xujinheng @alexeadem

[juliusvonkohout]

Thank you for creating this @juliusvonkohout! My thoughts on this.

Should we try to investigate what other Kubernetes projects are doing ? For example, kubeadm says: kubeadm performs the actions necessary to get a minimum viable cluster up and running.. So we can say: Kubeflow manifests performs the actions necessary to get a minimum viable Kubeflow platform up and running. But maybe we should explore what others are doing

I tried to incorporate it with "Using the Kubeflow manifests to install a basic Kubeflow version requires rudimentary Kubernetes knowledge"

@thesuperzapper to your point about:

and the other about what users might consider for production usage.

I think, it depends on the user. If they are capable enough to install Kubeflow components using kustomize manifests and configure other things, why they should explore Kubeflow distributions ? Similar to how people deploy Kubernetes on-prem vs Kubernetes distributions. With Kubernetes distribution you get a lot of great features on top, similar to Kubeflow distributions. And Kubeflow distributions should clearly explain the user value in their docs.

For commercial production-level usage and support there are many options. You can use a third-party commercial distribution, hire consultants or build up the knowledge yourself.

@juliusvonkohout @thesuperzapper Do we really need to say this in Kubeflow manifests section ? Why our users can't understand by themselves if they want to take Kubeflow distribution or Kubeflow manifests for their work ?

It was added in the second revision catering to Matthew but i am also fine dropping all distribution or support related stuff from the manifests section. So either we reduce it to

""Using the Kubeflow manifests to install a basic Kubeflow version requires rudimentary Kubernetes knowledge.
The Kubeflow community support for Kubeflow manifests is only best-effort, non-commercial and not guaranteed for environment-specific issues or custom configurations. Nevertheless, we welcome contributions and bug reports very much."

or we keep something like

"For commercial production-level usage and support there are many options. You can use a third-party commercial distribution, hire consultants or build up the knowledge yourself to maintain and extend your Kubeflow installation."

and it becomes

"Using the Kubeflow manifests to install a basic Kubeflow version requires rudimentary Kubernetes knowledge.
The Kubeflow community support for Kubeflow manifests is only best-effort, non-commercial and not guaranteed for environment-specific issues or custom configurations. Nevertheless, we welcome contributions and bug reports very much.
For commercial production-level usage and support there are many options. You can use a third-party commercial distribution, hire consultants or build up the knowledge yourself to maintain and extend your Kubeflow installation."

[andreyvelich]

I am fine with that. From my perspective our users should be able to choose Kubeflow Manifests or Kubeflow Distributions on their own. @jbottum Any thoughts from your side ?

[thesuperzapper]

@andreyvelich I strongly dislike words like "basic" and "rudimentary" as they can be seen as offensive or derogatory to users who struggle.

The purpose of this warning is to highlight two things:

1. We don't provide environment-specific support as a community.
   • (Some individuals from the community might sell services related to the manifests, but this is not official support)
2. Using the manifests in production requires a working knowledge of Kubernetes, Istio, and Kubeflow itself.
   • (We don't have to point users towards the distributions in this section but it's critical that users understand the key knowledge requirements)

As a compromise, here is a version which does not talk about anything other than those two points:

{{% alert title="Warning" color="warning" %}}
While the manifests provide a quick way to start using the Kubeflow Platform,
successfully using them in production requires a working knowledge of Kubernetes, Istio, and Kubeflow itself.

The Kubeflow community only provides best-effort support for manifests deployments.
We are unable to provide support for environment-specific issues or custom configurations. 
Nevertheless, we welcome contributions and bug reports very much.
{{% /alert %}}

[andreyvelich]

I strongly dislike words like "basic" and "rudimentary" as they can be seen as offensive or derogatory to users who struggle.

I agree with you @thesuperzapper, and my suggestion is just remove this completely. For instance:
"""
Kubeflow manifests performs the actions necessary to get a minimum viable Kubeflow platform up and running.
"""

We don't provide environment-specific support as a community.

Can you explain what exactly do you mean by this statement ? For example, KServe has implementation of S3 storage provider: https://github.com/kserve/kserve/blob/master/pkg/agent/storage/s3.go#L51. If something breaks with this part, we should fix it. Similar to this Kubeflow Training has implementation of S3 dataset provider: https://github.com/kubeflow/training-operator/blob/master/sdk/python/kubeflow/storage_initializer/s3.py
cc @johnugeorge @deepanker13
Isn't ?

Using the manifests in production requires a working knowledge of Kubernetes, Istio, and Kubeflow itself.

Don't you think it is obvious for the users, since we can say that Kubeflow Manifests will only give you: minimum viable Kubeflow Platform up and running ?

[thesuperzapper]

Don't you think it is obvious for the users, since we can say that Kubeflow Manifests will only give you: minimum viable Kubeflow Platform up and running ?

@andreyvelich I think that's a pretty ambiguous statement, especially for new users who won't know anything about Kubeflow or how it works.

Highlighting that production usage of Kubeflow requires some background knowledge will help avoid the frustration that we see from new users.

---

Can you explain what exactly do you mean by this statement ? For example, KServe has implementation of S3 storage provider:

It's not to say that we don't provide integrations for external tools, it's just that we don't provide one-on-one support for users that need help with things that are not bugs or missing features.

We are all volunteers, and while some people might be willing to help, it's good to set expectations and highlight that we don't do it officially.

That's why originally the wording pointed people towards vendors who can help them (like distributions, or consultants).

[juliusvonkohout]

Alright i updated it to

"The Kubeflow manifests provide a quick way to get a minimum viable Kubeflow platform up and running.
The Kubeflow community support for Kubeflow manifests is only best-effort, non-commercial and not guaranteed for environment-specific issues or custom configurations.
Nevertheless, we welcome contributions and bug reports very much.
For commercial production-level usage and support there are many options. You can use a third-party commercial distribution, hire consultants or build up the knowledge yourself to maintain and extend your Kubeflow installation."

[hbelmiro]

/lgtm

[andreyvelich]

Thanks for the update @juliusvonkohout!
Maybe we should capitilized "Kubeflow Platform", like here: https://www.kubeflow.org/docs/started/introduction/#what-is-kubeflow-platform.
Overall, looks good to me.

[andreyvelich]

Thanks @juliusvonkohout
/lgtm
/assign @james-jwu @johnugeorge @terrytangyuan @jbottum

[terrytangyuan]

/lgtm

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: terrytangyuan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• content/en/docs/started/OWNERS [terrytangyuan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment