[SPARK-47849][PYTHON][CONNECT] Change release script to release pyspa…

…rk-connect

### What changes were proposed in this pull request?

This PR proposes to change release script to publish `pyspark-connect`.

### Why are the changes needed?

See the SPIP https://issues.apache.org/jira/browse/SPARK-47540.

### Does this PR introduce _any_ user-facing change?

Yes, it releases `pyspark-connect` package into PyPI.

### How was this patch tested?

Did the basic test for individual commands. It will be tested during actual release process (and I will cowork together with 4.0 release manager).

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#46049 from HyukjinKwon/SPARK-47849.

Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>

dev/create-release/release-build.sh

@@ -123,6 +123,12 @@ if [[ "$1" == "finalize" ]]; then
     --repository-url https://upload.pypi.org/legacy/ \
     "pyspark-$RELEASE_VERSION.tar.gz" \
     "pyspark-$RELEASE_VERSION.tar.gz.asc"
+  svn update "pyspark-connect-$RELEASE_VERSION.tar.gz"
+  svn update "pyspark-connect-$RELEASE_VERSION.tar.gz.asc"
+  TWINE_USERNAME=spark-upload TWINE_PASSWORD="$PYPI_PASSWORD" twine upload \
+    --repository-url https://upload.pypi.org/legacy/ \
+    "pyspark-connect-$RELEASE_VERSION.tar.gz" \
+    "pyspark-connect-$RELEASE_VERSION.tar.gz.asc"
   cd ..
   rm -rf svn-spark
   echo "PySpark uploaded"
@@ -302,6 +308,14 @@ if [[ "$1" == "package" ]]; then
         --output $PYTHON_DIST_NAME.asc \
         --detach-sig $PYTHON_DIST_NAME
       shasum -a 512 $PYTHON_DIST_NAME > $PYTHON_DIST_NAME.sha512
+
+      PYTHON_CONNECT_DIST_NAME=pyspark-connect-$PYSPARK_VERSION.tar.gz
+      cp spark-$SPARK_VERSION-bin-$NAME/python/dist/$PYTHON_CONNECT_DIST_NAME .
+
+      echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --armour \
+        --output $PYTHON_CONNECT_DIST_NAME.asc \
+        --detach-sig $PYTHON_CONNECT_DIST_NAME
+      shasum -a 512 $PYTHON_CONNECT_DIST_NAME > $PYTHON_CONNECT_DIST_NAME.sha512
     fi
 
     echo "Copying and signing regular binary distribution"

dev/make-distribution.sh

@@ -249,6 +249,7 @@ if [ "$MAKE_PIP" == "true" ]; then
   # Delete the egg info file if it exists, this can cache older setup files.
   rm -rf pyspark.egg-info || echo "No existing egg info file, skipping deletion"
   python3 packaging/classic/setup.py sdist
+  python3 packaging/connect/setup.py sdist
   popd > /dev/null
 else
   echo "Skipping building python distribution package"