Sync with changes merged after 6378d4bc06cd1bb1a209bd5fb63d10ef52d75eb4

[zero323]

• SPARK-28969 OneVsRestParams parity between scala and python - Move setClassifier from OneVsRestParams to OneVsRest #226
• SPARK-22797 Bucketizer support multi-column - Add multiple column support to Bucketizer #236
• SPARK-27463 Support Dataframe Cogroup via Pandas UDFs - Add annotations for Cogroup  #242
• SPARK-22796 Add multiple columns support to PySpark QuantileDiscretizer - Add multiple column support to Bucketizer #236
• SPARK-28985 Add common classes (JavaPredictor / JavaClassificationModel / JavaProbabilisticClassifier) in PYTHON - SPARK-28985 #262
• SPARK-27659 Allow PySpark to prefetch during toLocalIterator - Add prefetchPartitions to {RDD,DataFrame}.toLocalIterator #240
• SPARK-29142 Pyspark clustering models support column setters/getters/predict - SPARK-29142 Pyspark clustering models support column setters/getters/predict #259
• SPARK-29258 parity between ml.evaluator and mllib.metrics - SPARK-29258 parity between ml.evaluator and mllib.metrics #243
• SPARK-29240 (no changes needed as we accept Any).
• SPARK-29143 Pyspark feature models support column setters/getters - SPARK-29143 #278
• SPARK-29360 PySpark FPGrowthModel supports getter/setter - SPARK-29360 #279
• SPARK-29269 Pyspark ALSModel support getters/setter - Update ALS-related annotations #282
• SPARK-29116 Refactor py classes related to DecisionTree - SPARK-29116 #285
• SPARK-29377 Parity between Scala ML tuning and Python ML tuning - SPARK-29377 #287
• SPARK-29381 / [FOLLOWUP] ([SPARK-29381][FOLLOWUP][PYTHON][ML] Add 'private' _XXXParams classes for classification & regression apache/spark#26142) Add _ before the XXXParams classes - SPARK-29381 #292
• SPARK-23578 Binarizer support multi-column - SPARK-23578 Add HasThresholds and Has(Input,Output)Cols to Binarizer #294
• SPARK-29464 PySpark ML should expose Params.clear() to unset a user supplied Param - SPARK-29464 Add Params.clear #295
• SPARK-29235 Support avgMetrics in read/write of CrossValidatorModel (No changes required)
• SPARK-29499 Add mapPartitionsWithIndex for RDDBarrier - SPARK-29499 Add RDDBarrier.mapPartitionsWithIndex #296
• SPARK-29444 add doc and python parameter for ignoreNullFields in json generating - SPARK-29444 Add ignoreNullFields to DataFrameWriter.json #297
• SPARK-29093 Remove automatically generated param setters in _shared_params_code_gen.py - SPARK-29093, part 1 #298, SPARK-29093, part 2 #301
• SPARK-29566 Imputer should support single-column input/output - SPARK-29566 Add inputCol, outputCol to Imputer #302
• SPARK-29565 OneHotEncoder should support single-column input/output - SPARK-29565 Add inputCol, outputCol to OneHotEncoder #303
• SPARK-29627 Allow array_contains to take column instances (no changes needed as we accept Any).
• SPARK-29645 ML add param RelativeError - SPARK-29645 Mix-in HasRelativeError into ml.feature classes #304
• SPARK-29664 Column.getItem behavior is not consistent with Scala (No changes required)
• SPARK-29656 ML algs expose aggregationDepth - SPARK-29656 Add aggregationDepth to GeneralizedLinearRegression and GaussianMixture #305
• SPARK-29808 StopWordsRemover should support multi-cols - SPARK-29808 Add inputCols / outputCols to StopWordsRemover #306
• SPARK-16872 Impl Gaussian Naive Bayes Classifier SPARK-16872 Add NaiveBayesModel.sigma #307
• SPARK-29959 Summarizer support more metrics - SPARK-29959 Add Summarizer sum and std #308
• SPARK-27990 Add recursiveFileLookup option to Python DataFrameReader - SPARK-27990 Add recursiveFileLookup to DataFrameReader and DataStreamReader methods #309
• SPARK-30091 Document mergeSchema option directly in the PySpark Parquet APIs - SPARK-30091 Add mergeSchema to DataStreamReader.parquet #310
• SPARK-30113 Expose mergeSchema option in PySpark's ORC APIs - SPARK-30113 Expose mergeSchema option in PySpark's ORC APIs #311
• SPARK-30146 Add setWeightCol to GBTs in PySpark - SPARK-30146 Add setWeightCol to GBTs #312
• SPARK-29967 KMeans support instance weighting - SPARK-29967 Add weightCol to KMeans #313
• SPARK-30231 Support explain mode in PySpark df.explain - SPARK-30231 Add mode and change sig of extended to DataFrame.explain #314
• SPARK-29224 Implement Factorization Machines as a ml-pipeline - SPARK-29224 Add annotations for FMRegressor / FMRegressionModel and FMClassifier / FMClassificationModel #316
• SPARK-30247 GaussianMixtureModel in py side should expose gaussian - SPARK-30247 Add MultivariateGaussian and GaussianMixtureModel.gaussians #318
• SPARK-30178 RobustScaler support large numFeatures (No changes required)
• SPARK-30102 GMM supports instance weighting - SPARK-30102 Add weightCol to GaussianMixture #319
• SPARK-30378 Add getter/setter in Python FM - SPARK-29224 Add annotations for FMRegressor / FMRegressionModel and FMClassifier / FMClassificationModel #316
• SPARK-30358 ML expose predictRaw and predictProbability on Python side - SPARK-30358 Add JavaClassificationModel.predictRaw and JavaProbabilisticClassific… #320
• SPARK-30144 Make MultilayerPerceptronClassificationModel extend MultilayerPerceptronParams SPARK-30144 Add _MultilayerPerceptronParams to MultilayerPerceptronClassification… #321
• SPARK-30154 PySpark UDF to convert MLlib vectors to dense arrays - Add ml.functions stub with vector_to_array #317
• SPARK-30434 Move pandas related functionalities into 'pandas' sub-package - SPARK-30434 Move pandas related functionalities into 'pandas' sub-package #327
• SPARK-30452 Add predict and numFeatures in Python IsotonicRegressionModel - SPARK-30452 Add predict and numFeatures to IsotonicRegressionModel #322
• SPARK-30493 Remove OneVsRestModel setClassifier, setLabelCol and setWeightCol methods - SPARK-29093, part 1 #298

[zero323]

All done!