[SPARK-12268][PYSPARK] Make pyspark shell pythonstartup work under python3 #10255
Conversation
This replaces the execfile used for running custom python shell scripts with explicit open, compile and exec. The reason is to make the pythonstartup option compatible with python3.
|
Could you please file a JIRA for this? See https://issues.apache.org/jira/browse/SPARK and https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark |
tyro89
changed the title
|
Jenkins, this is ok to test. |
|
Test build #47542 has finished for PR 10255 at commit
|
|
@JoshRosen Is there anything else missing on this PR or can this get merged? |
|
Jenkins, retest this please. |
|
Sorry for letting this slip through the cracks. Let me manually test this out now and if it looks good I'll merge it into master and branch-1.6. |
| @@ -76,4 +76,6 @@ | |||
| # which allows us to execute the user's PYTHONSTARTUP file: | |||
| _pythonstartup = os.environ.get('OLD_PYTHONSTARTUP') | |||
| if _pythonstartup and os.path.isfile(_pythonstartup): | |||
| execfile(_pythonstartup) | |||
| with open(_pythonstartup) as f: | |||
There was a problem hiding this comment.
One minor consequence of this change is that the variable f is now defined in in the REPL, e.g. f will now point to a closed file. We could try to add some fancy logic to prevent this, but it would be equally brittle in the opposite direction. Therefore I'm fine with this change; it's extremely unlikely that user code in the REPL will take conditional actions based on whether f is defined.
|
I tested this manually on Python 2.6.9, 2.7.10, 3.4.2, and PyPy 2.4.0 and it worked as expected, so I'm going to merge this change into |
|
Thank you! |
…thon3 This replaces the `execfile` used for running custom python shell scripts with explicit open, compile and exec (as recommended by 2to3). The reason for this change is to make the pythonstartup option compatible with python3. Author: Erik Selin <[email protected]> Closes #10255 from tyro89/pythonstartup-python3. (cherry picked from commit e4e0b3f) Signed-off-by: Josh Rosen <[email protected]>
|
Test build #49328 has finished for PR 10255 at commit
|
This replaces the
execfileused for running custom python shell scriptswith explicit open, compile and exec (as recommended by 2to3). The reason
for this change is to make the pythonstartup option compatible with python3.