[SPARK-28985][PYTHON][ML][FOLLOW-UP] Add _IsotonicRegressionBase

[zero323]

What changes were proposed in this pull request?

Adds

class _IsotonicRegressionBase(HasFeaturesCol, HasLabelCol, HasPredictionCol, HasWeightCol): ...

with related Params and uses it to replace JavaPredictor and HasWeightCol in IsotonicRegression base classes and JavaPredictionModel, in IsotonicRegressionModel base classes.

Why are the changes needed?

Previous work (#25776) on SPARK-28985 replaced JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol in IsotonicRegression and JavaModel in IsotonicRegressionModel with newly added JavaPredictor:

spark/python/pyspark/ml/wrapper.py

Line 377 in e97b55d

class JavaPredictor(JavaEstimator, JavaPredictorParams):

and JavaPredictionModel

spark/python/pyspark/ml/wrapper.py

Line 405 in e97b55d

class JavaPredictionModel(JavaModel, JavaPredictorParams):

respectively.

This however is inconsistent with Scala counterpart where both classes extend private IsotonicRegressionBase

spark/mllib/src/main/scala/org/apache/spark/ml/regression/IsotonicRegression.scala

Lines 42 to 43 in 3cb1b57

	private[regression] trait IsotonicRegressionBase extends Params with HasFeaturesCol
	with HasLabelCol with HasPredictionCol with HasWeightCol with Logging {

This preserves some of the existing inconsistencies (model as defined in the official example), i.e.

from pyspark.ml.regression impor IsotonicRegressionMode
from pyspark.ml.param.shared import HasWeightCol

issubclass(IsotonicRegressionModel, HasWeightCol) 
# False

hasattr(model, "weightCol")                                                       
# True

as well as introduces a bug, by adding unsupported predict method:

import inspect

hasattr(model, "predict")                                                         
# True

inspect.getfullargspec(IsotonicRegressionModel.predict)                           
# FullArgSpec(args=['self', 'value'], varargs=None, varkw=None, defaults=None, kwonlyargs=[], kwonlydefaults=None, annotations={})

IsotonicRegressionModel.predict.__doc__                                                                                                                                                           
# Predict label for the given features.\n\n        .. versionadded:: 3.0.0'

model.predict(dataset.first().features)  

# Py4JError: An error occurred while calling o49.predict. Trace:
# py4j.Py4JException: Method predict([class org.apache.spark.ml.linalg.SparseVector]) does not exist
# ...

Furthermore existing implementation can cause further problems in the future, if Predictor / PredictionModel API changes.

Does this PR introduce any user-facing change?

Yes. It:

• Removes invalid IsotonicRegressionModel.predict method.
• Adds HasWeightColumn to IsotonicRegressionModel.

however the faulty implementation hasn't been released yet, and proposed additions have negligible potential for breaking existing code (and none, compared to changes already made in #25776).

How was this patch tested?

• Existing unit tests.
• Manual testing.

CC @huaxingao, @zhengruifeng

[SparkQA]

Test build #111774 has finished for PR 26023 at commit e5183b8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class _IsotonicRegressionBase(HasFeaturesCol, HasLabelCol, HasPredictionCol, HasWeightCol):
• class IsotonicRegression(JavaEstimator, _IsotonicRegressionBase, HasWeightCol,
• class IsotonicRegressionModel(JavaModel, _IsotonicRegressionBase,

[zhengruifeng]

LGTM

[srowen]

Merged to master

[zero323]

Thanks @srowen, @zhengruifeng!