Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-1267]Adding a pip installer setup file for PySpark. #464

Closed
wants to merge 1 commit into from

Conversation

prabinb
Copy link

@prabinb prabinb commented Apr 21, 2014

Following changes are made in this pull request,

  1. A pip installer python/setup.py file is added.
  2. A new file is added, pyspark/pyspark_version.py for maintaining PySpark version. This needs to be updated, whenever a new version of PySpark is released.
  3. Changed pyspark/init.py to cross validate SPARK_HOME variable and pyspark & spark version mismatch.

This PySpark build distribution has to be registered and uploaded to PyPi (python package index) for all the releases of PySpark. (somebody needs to maintain it)
python setup.py register
python setup.py sdist upload
More details on registering and uploading a package can be found here, https://docs.python.org/2/distutils/packageindex.html

Once the package is uploaded, users will be able to install PySpark by running,
pip install pyspark

Following validations are added to the import, (all suggestions to improve it are welcome)

  1. For using this package, the user should set SPARK_HOME environment variable to it's spark installation directory, else the import fails.
  2. The spark version in SPARK_HOME/pom.xml file (...spark version...) should match the pyspark version (python/pyspark/pyspark_version.py), else the import fails.

… a new file pyspark/pyspark_version.py for maintaining PySpark version. Changed pyspark/__init__.py to cross validate SPARK_HOME variable and pyspark & spark version mismatch
@AmplabJenkins
Copy link

Can one of the admins verify this patch?

1 similar comment
@SparkQA
Copy link

SparkQA commented Sep 5, 2014

Can one of the admins verify this patch?

@JoshRosen
Copy link
Contributor

Hi @prabinb,

Davies commented on this on JIRA: https://issues.apache.org/jira/browse/SPARK-1267?focusedCommentId=14192279&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14192279

I agree with his comments, so do you mind closing this PR for now?

@asfgit asfgit closed this in f73b56f Nov 10, 2014
j-esse pushed a commit to j-esse/spark that referenced this pull request Jan 24, 2019
bzhaoopenstack pushed a commit to bzhaoopenstack/spark that referenced this pull request Sep 11, 2019
* Change periodic time to "0 4,8,16 * * *"

* Delete 1
RolatZhang pushed a commit to RolatZhang/spark that referenced this pull request Aug 15, 2022
RolatZhang pushed a commit to RolatZhang/spark that referenced this pull request Aug 15, 2022
* compatible for spark 3.1 (apache#464)

* code for adapt 3.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants