How do I get bed2psl.py to run, in a conda (--devel) installation of CGAT? #392
Comments
|
As a sanity check, would you be able to run The issue isn't clear to me at the moment and I have no issues running bed2psl, but if its related to your installation process @sebastian-luna-valero is the best person to answer this one. |
|
Thanks for the suggestions. Here's what I get on my system, after running the following commands: [the (cgat-s) environment appears] ~/src/cgat/conda-install/envs/cgat-s/bin/python Python 3.6.3 [whoops! nothing at all shows up!] So, it looks as if I may need to be supplying $PYTHONPATH to my shell. What would a good value for it be? |
|
Ah your PYTHONPATH is required for modules to be available within python. I wonder if this is the issue. if you try Then try if that works then you may want to automate this in your .basrc |
|
Thanks, both. Apart from A minor amendment to @Acribbs comments above. The steps to setup your Please let us know how you get on. Best regards, |
|
Thanks @sebastian-luna-valero, just to clarify, would that mean you would get $HOME/src/cgat/cgat as your PYTHONPATH? |
|
Thanks Adam, for pointing that out. Sorry, it should be PYTHONPATH=$HOME/src/cgat/cgat-scripts |
|
Updates: In my comment above: should actually be: But more importantly, once you If possible, please report: |
|
I've followed the consensus of the advice above. Long story short -- Here are the details of what I've just run: [Once more unto the breach, dear friends, once more!] [Whoops, failure:] Traceback (most recent call last): [But notice that this, at least, does superficially work;] cgat.py - Computational Genomics Analysis Tools [etc. etc.] [OK, let's take a look at this dubious variable, then.] /opt/software/cudatoolkit/6.0/computeprof/bin:/opt/software/cudatoolkit/6.0/lib64:/opt/software/cudatoolkit/6.0/lib:/opt/software/R/2.15.1--GCC-4.4.5/lib64/R/lib:/opt/software/boost/1.47.0--GCC-4.4.5/lib:/opt/software/ClusterStudio/2013sp1/composer_xe_2013_sp1.0.080/mkl/lib/intel64:/opt/software/MATLAB/R2014a/sys/opengl/lib/glnxa64:/opt/software/Python/2.7.2--GCC-4.4.5/lib/python2.7/lib-dynload:/opt/software/Python/2.7.2--GCC-4.4.5/lib:/opt/software/ClusterStudio/2011.0/composerxe-2011.5.220/tbb/lib/intel64/cc4.1.0_libc2.4_kernel2.6.16.21/irml:/opt/software/ClusterStudio/2011.0/composerxe-2011.5.220/tbb/lib/intel64/cc4.1.0_libc2.4_kernel2.6.16.21:/opt/software/OpenMPI/1.4.3--GCC-4.4.5/lib:/mnt/home/emsch/lib:/mnt/home/emsch/lib64 None of these directories overlap with anything inside the CGAT installation. So, now what should I do? |
|
Thanks! Could you unset your Does this make any difference? Best regards, |
|
I've tried that too. It makes no difference. I have to wonder: should I be explicitly setting Here are the details of what I just did: Traceback (most recent call last): /mnt/home/emsch/src/cgat/cgat-scripts [still nothing!] |
|
Setting the $LD_LIBRARY_PATH is usually only required for shared libraries. I assume that @sebastian-luna-valero was asking to unset your path because it contained shared libraries for python2.7 I will try to do some more digging into this, unfortunately sebastian is now on holiday for a week. Just as another sanity check, are you able to run these two scripts: |
|
Hi, I think the issue that some multiple python versions got muddled. One possible quick-fix could be:
|
|
Briefly: I reinstalled conda (as part of Anaconda2, which is Python2-based, not Python3-based), then reinstalled CGAT, then followed the advice above to the best of my understanding. Things still did not work. Here, I'll give the full line-by-line story of what I've done. In the next post, I'll ask a few questions about what things I might be able to do to fix the problems I'm encountering. What I did: Log out, log back in, do the Anaconda2 installation from scratch, then do the manual editing of .bashrc to fit what I'm learning I should be doing. 5b1b5784cae93cf696e11e66983d8756 Anaconda2-5.1.0-Linux-x86_64.sh Accept default installation site: /mnt/home/emsch/anaconda2 ~/anaconda2/bin/conda """ [etc. etc.] Log out, log back in, check again: /usr/bin/which: no conda in [...] Now we get '(base)' in prompt. ~/anaconda2/bin/conda Do basic setup to get bioconda functioning. Log out, log back in, activate conda, then within activated conda try doing the CGAT installation again. ~/anaconda2/bin/python (No lines.) /opt/software/cudatoolkit/6.0/computeprof/bin /opt/software/Python/2.7.2--GCC-4.4.5/lib/python2.7/lib-dynload -rwxrwxr-x 1 johnj helpdesk 9752110 Jan 11 2012 libpython2.7.a lrwxrwxrwx 1 emsch beacon 19 Apr 15 15:04 libpython2.7.so -> libpython2.7.so.1.0 62 60 Oh wait ... CGAT is based on Python 3.6.3! export LD_LIBRARY_PATH=/opt/software/cudatoolkit/6.0/computeprof/bin: [This effectively deletes the following from LD_LIBRARY_PATH -- """
Installed /mnt/home/emsch/src/cgat/cgat-scripts The code was successfully installed! To activate the CGAT environment type: To deactivate the environment, use: Zero lines. """ Traceback (most recent call last): Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): Traceback (most recent call last): Try deleting *.so and then rebuilding: rm /mnt/home/emsch/src/cgat/cgat-scripts/CGAT/GeneModelAnalysis.cpython-36m-x86_64-linux-gnu.so ; """ Installed /mnt/home/emsch/src/cgat/cgat-scripts """ Traceback (most recent call last): Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): Traceback (most recent call last): ... so this is completely failing. One thing I might have done would have been to prepend the following to LIB or to PYTHONPATH: Or maybe, these? -rwxr-xr-x 1 emsch beacon 22628068 Apr 15 16:53 libpython3.6m.a 65 Go ahead and try the latter possibility myself. /opt/software/cudatoolkit/6.0/computeprof/bin export LD_LIBRARY_PATH=/mnt/home/emsch/src/cgat/conda-install/lib/python3.6/lib-dynload:/mnt/home/emsch/src/cgat/conda-install/lib:/opt/software/cudatoolkit/6.0/computeprof/bin:/opt/software/cudatoolkit/6.0/lib64:/opt/software/cudatoolkit/6.0/lib:/opt/software/R/2.15.1--GCC-4.4.5/lib64/R/lib:/opt/software/boost/1.47.0--GCC-4.4.5/lib:/opt/software/ClusterStudio/2013sp1/composer_xe_2013_sp1.0.080/mkl/lib/intel64:/opt/software/MATLAB/R2014a/sys/opengl/lib/glnxa64:/opt/software/ClusterStudio/2011.0/composerxe-2011.5.220/tbb/lib/intel64/cc4.1.0_libc2.4_kernel2.6.16.21/irml:/opt/software/ClusterStudio/2011.0/composerxe-2011.5.220/tbb/lib/intel64/cc4.1.0_libc2.4_kernel2.6.16.21:/opt/software/OpenMPI/1.4.3--GCC-4.4.5/lib:/mnt/home/emsch/lib:/mnt/home/emsch/lib64 ; [standard help message] Traceback (most recent call last): Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): Traceback (most recent call last): Try one more time to delete *.so files and rebuild them: rm /mnt/home/emsch/src/cgat/cgat-scripts/CGAT/GeneModelAnalysis.cpython-36m-x86_64-linux-gnu.so ; """ Installed /mnt/home/emsch/src/cgat/cgat-scripts """ Traceback (most recent call last): Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): Traceback (most recent call last): Short of doing a complete reinstall of Anaconda3 and trying to create a conda environment based on Python3 rather than Python2, I'm not sure what I'm supposed to be doing here. |
|
@Acribbs wrote: "are you able to run these two scripts: cgat bam2bam --help and cgat bed2bed --help?" I tried both of these. They have both persistently failed. You can see the gory long details in my immediately preceding post... |
|
@AndreasHeger wrote: "I think the issue that some multiple python versions got muddled. One possible quick-fix could be: (best with an LD_LIBRARY_PATH that does not include any python's and within the cgat conda environment)..." I tried this, and it did not succeed. However, it is very far from obvious to me that I have either the correct Python version in Anaconda, or the correct directories within cgat pre-pended to LD_LIBRARY_PATH. So, my questions:
|
|
Hi, r.e. running 3 different scripts: Some of those scripts use cython code some do not so wanted to eliminate this as the underlying issue. My take on your questions, but @AndreasHeger may have other ideas.
|
|
Just as another note, because you said you were new to python. Python 2 will no longer be supported from 2020, so if you can I would move completely to python3. https://pythonclock.org |
|
Thank you, @Acribbs, for the advice so far! I have attempted to follow it completely. For whatever reason, this has not led any of my scripts to work properly -- they're still exploding in Pythonic chaos when I try to run them. Here's a detailed account of what I did: Log out and log back in. Then: Get a whole bunch of text while installation proceeds: So, let's see if this works! Verify that conda isn't already in our $PATH; activate CGAT; see if conda has appeared in our $PATH (which implies that everything else has been properly made available); fix $PYTHONPATH; try running CGAT scripts. /usr/bin/which: no conda in (/mnt/home/emsch/perl5/bin: [etc. etc.] ~/src/cgat/conda-install/envs/cgat-s/bin/conda [reasonable help message] [...] Traceback (most recent call last): Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): Traceback (most recent call last): ~/src/cgat/conda-install/envs/cgat-s/bin/python Python 3.6.3 |
|
Also, I tried rebuilding the *.so files with the Python3 provided by CGAT; that, also, failed to help. Here are details: rm /mnt/home/emsch/src/cgat/cgat-scripts/CGAT/Timeseries/cmetrics.cpython-36m-x86_64-linux-gnu.so ; ~/src/cgat/conda-install/envs/cgat-s/bin/python [long set of messages, ending in...] Installed /mnt/home/emsch/src/cgat/cgat-scripts Traceback (most recent call last): Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): Traceback (most recent call last): |
|
Hi @SchwarzEM Sorry for all the hassle. I am surprised you did not have the same error messages when you raised a previous issue #388 . I assume you are now installing the code in a different runtime environment. Can I ask you to try the following steps instead, please? If that works, then you should always unset both Could you please let me know how it goes? Best regards, |
|
Unfortunately, following your instructions hasn't fixed any of the problems. Here's what I got when I tried: [Lots of text messages:] Warning: 2 possible package resolutions (only showing differing packages):
dbus post-link :: .. using /proc/sys/kernel/random/boot_id [...] Installed /mnt/home/emsch/src/cgat/cgat-scripts The code was successfully installed! To activate the CGAT environment type: To deactivate the environment, use: cgat.py - Computational Genomics Analysis Tools Python GFF Annotation Traceback (most recent call last): Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): Traceback (most recent call last): So, failure once again. |
|
Here's a question: although I believe CGAT is supposed to import most of what it needs in order to compile itself (starting with miniconda), I assume that it does not try to import GCC compilers (gcc and g++). What version(s) of gcc and g++ are optimal for installing and running CGAT? Is one reason that I keep getting messages like this: that CGAT really wants gcc 6+, but my system default is 4.8.5? |
|
I went ahead and re-installed CGAT after deleting not only the pre-existing installation, but my preexisting anaconda2 installation and its hidden directories -- to get rid of any files that might be confusing the installation. Result: again, failure of the modules to run. Details: /mnt/home/emsch/src/cgat [lots of messages] [standard, OK-looking messages] Traceback (most recent call last): Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): Traceback (most recent call last): |
|
So, 22 days ago, I had a reasonable solution proposed by @sebastian-luna-valero in #388 -- namely, do a standard installation of CGAT with the "--production" option rather than the "--devel" option, then run these commands to patch bed2psl.py into CGAT: But it seemed more elegant to do a "--devel" installation, so I tried that instead. As my posts on this github ticket indicate, misery ensued. For three weeks, that somehow kept me from remembering the original, untried suggestion. Tonight I finally snapped out of my mental fog and tried it. Surprise! It actually seems to work! Moreover, other scripts seem to work too! Here's what I did: [long messages] [looks OK] [works! or, at least, gives clean help message...] [! works! or, at least, gives clean help message...] [!! works! or, at least, gives clean help message...] Traceback (most recent call last): [Which is the error message I should see, given that bed2psl.py isn't in the --production version of CGAT. So, try patching in the script.] [Voila! A clean-looking help message, rather than an explosion of Pythonic failure!] From this, I conclude that the problem wasn't that I'm utterly incompetent at getting CGAT installed. When I used "--production" to install it, I get scripts whose help messages (at least) run just fine. The problem isn't my system either (although it may have been prudent to unset LD_LIBRARY_PATH before installing and running CGAT). The problem, at least in my hands, is that "--devel" installations of CGAT do not work. |
|
Thanks for your feedback @SchwarzEM and sorry for all the hassle. It is sad to see that after all the effort we put into making the installation of the code portable, it still does not work and waste other people's time out there. The differences between the installation of There must be a conflict in the Please note that with the I will close this issue now but feel free to reopen if you want to follow up. |
I've followed the advice in #388 in order to install a version of CGAT that contains
bed2psl.py.[...]
[... long installation process that eventually reaches this message ...]
[...]
The code was successfully installed!
After that, I activate CGAT, as instructed during the installation process:
The CGAT installation appears to have
bed2pslas one of its components:[... many tools ...]
bed2psl
[... etc. ..]
But then, when I try to run
bed2pslin various ways, I get the following messages:Traceback (most recent call last):
File "/mnt/home/emsch/src/cgat/conda-install/envs/cgat-s/bin/cgat", line 11, in
load_entry_point('CGAT', 'console_scripts', 'cgat')()
File "/mnt/home/emsch/src/cgat/cgat-scripts/CGAT/cgat.py", line 129, in main
module = imp.load_module(command, file, pathname, description)
File "/mnt/home/emsch/src/cgat/conda-install/envs/cgat-s/lib/python3.6/imp.py", line 235, in load_module
return load_source(name, filename, file)
File "/mnt/home/emsch/src/cgat/conda-install/envs/cgat-s/lib/python3.6/imp.py", line 172, in load_source
module = _load(spec)
File "", line 684, in _load
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/mnt/home/emsch/src/cgat/cgat-scripts/CGAT/scripts/bed2psl.py", line 27, in
import CGAT.Blat as Blat
File "/mnt/home/emsch/src/cgat/cgat-scripts/CGAT/Blat.py", line 25, in
from CGAT import Components as Components
ImportError: /mnt/home/emsch/src/cgat/cgat-scripts/CGAT/Components.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _Py_ZeroStruct
Traceback (most recent call last):
File "/mnt/home/emsch/src/cgat/conda-install/envs/cgat-s/bin/cgat", line 11, in
load_entry_point('CGAT', 'console_scripts', 'cgat')()
File "/mnt/home/emsch/src/cgat/cgat-scripts/CGAT/cgat.py", line 129, in main
module = imp.load_module(command, file, pathname, description)
File "/mnt/home/emsch/src/cgat/conda-install/envs/cgat-s/lib/python3.6/imp.py", line 235, in load_module
return load_source(name, filename, file)
File "/mnt/home/emsch/src/cgat/conda-install/envs/cgat-s/lib/python3.6/imp.py", line 172, in load_source
module = _load(spec)
File "", line 684, in _load
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/mnt/home/emsch/src/cgat/cgat-scripts/CGAT/scripts/bed2psl.py", line 27, in
import CGAT.Blat as Blat
File "/mnt/home/emsch/src/cgat/cgat-scripts/CGAT/Blat.py", line 25, in
from CGAT import Components as Components
ImportError: /mnt/home/emsch/src/cgat/cgat-scripts/CGAT/Components.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _Py_ZeroStruct
-bash: bed2psl: command not found
-bash: bed2psl: command not found
So, is there something I'm supposed to be doing, and which I'm not doing, that would make it possible to run
bed2psl? If so, what is it?Conversely, if there's something I've done wrong in my installation that would have the effect of crippling
bed2psl, how can I redo my CGAT installation better?The text was updated successfully, but these errors were encountered: