GH-44980: [CI] Remove retrieval of Arrow version from Java on Spark integration and update test structure for PySpark

[raulcd]

Rationale for this change

The job is currently failing.

What changes are included in this PR?

Remove unnecessary check on Java code and refactor pyspark test modules to follow new test structure: apache/spark#49104

Are these changes tested?

Via archery

Are there any user-facing changes?

No

• GitHub Issue: [CI][Java][Python] Integration spark jobs are failing due to missing java folder #44980

[github-actions]

⚠️ GitHub issue #44980 has been automatically assigned in GitHub to PR creator.

[raulcd]

@github-actions crossbow submit test--spark-

[github-actions]

Revision: c410de9

Submitted crossbow builds: ursacomputing/crossbow @ actions-5eb2ac471e

Task	Status
test-conda-python-3.11-spark-master

[raulcd]

Tests seem to be failing but this failure is not related to the original issue:

 Starting test(python): pyspark.sql.tests.test_arrow (temp output: /spark/python/target/71d33db2-ca30-4e89-8edf-6aed57761990/python__pyspark.sql.tests.test_arrow__kzeeud_w.log)
/opt/conda/envs/arrow/bin/python: No module named pyspark.sql.tests.test_arrow

Had test failures in pyspark.sql.tests.test_arrow with python; see logs.

[raulcd]

@github-actions crossbow submit test--spark-

[github-actions]

Revision: 3deeecd

Submitted crossbow builds: ursacomputing/crossbow @ actions-ab6d626f46

Task	Status
test-conda-python-3.11-spark-master

[raulcd]

@github-actions crossbow submit test-conda-python-3.11-spark-master

[github-actions]

Revision: 084399a

Submitted crossbow builds: ursacomputing/crossbow @ actions-f1ce0ab770

Task	Status
test-conda-python-3.11-spark-master

[raulcd]

@github-actions crossbow submit test-conda-python-3.11-spark-master

[github-actions]

Revision: e661dea

Submitted crossbow builds: ursacomputing/crossbow @ actions-7f74565d52

Task	Status
test-conda-python-3.11-spark-master

[kou]

+1

[kou]

Are they known issues? If so, could you add an issue number?

[raulcd]

I've added a comment with the issue: #44986
As shared on the new issue those tests were never run in the past but might be worth adding them as they are arrow related.

[raulcd]

👍
Merging this to fix the spark integration failures,.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 2 benchmarking runs that have been run so far on merge-commit 5db2843.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.