Test installation of jNeuroML toolchain from source on Bash for Windows

[pgleeson]

See https://msdn.microsoft.com/en-gb/commandline/wsl/about

@Simon-at-Ely has been testing this out

[Simon-at-Ely]

The Windows Subsystem for Linux that can be activated after the Anniversary Update of Windows 10 has been installed allows a full version of Ubuntu 14.04 (16.04 will be featured after the Spring update) to be added and run from a command prompt. X windows is not supported from the Bash prompt, however installing Cygwin/x (xorg-server, xinit, xhost, kde-workspace, xfonts-100dpi, xfonts-75dpi) natively on windows allows communication with the Bash terminal if the following command is entered into the Cygwin xterminal:

$ xinit startkde -- -listen tcp

The -listen tcp is important as it allows bash on windows to talk to it.

In the Bash on Windows session the following command completes the link:

$ export DISPLAY=:0.0

Now running myLEMSfile_nrn.py with nrngui works perfectly:

sudo apt-get install works as normal to install most applications.

For the moment java only works if using the "-xint" option (this can be done from the .bashrc file with an export message). This bug has been fixed for insider builds and the bug fix will come to regular users in the Spring 2017 update.

So now, I have pyNeuron, nrngui, nrnivmodl and pyNeuroML all working in Bash for Windows. But I am still working on installing jNeuroML the sticking point is that it requires a Maven install and the process runs until mvn is invoked and maven fails to start. I will report back with further details Maven issue soon.

[pgleeson]

Thanks for the update on this @Simon-at-Ely

[Simon-at-Ely]

Your welcome @pgleeson here is the 2nd installment of the problem I having doing the Maven install of jNeuroML.

To start with here is the stuff I have added to my .bashrc file:

added by Simon

export NEURON_HOME=/home/Simon/neuron/nrn/x86_64
export PATH="/home/Simon/neuron/nrn/bin:$PATH"
export PATH="/home/Simon/pyNeuroML/pyneuroml/lib:$PATH"
export JAVA_HOME="/usr/lib/jvm/java-7-openjdk-amd64/jre"
export PATH="/usr/share/maven/bin:$PATH"
export _JAVA_OPTIONS="-Xint"

added by Anaconda2 4.1.1 installer

export PATH="/home/Simon/anaconda2/bin:$PATH"
set KMP_AFFINITY=disabled
export KMP_AFFINITY

Next checking that I have succeeded in installing Maven:

Simon@DESKTOP-GMI3HL5:~$ mvn -v
Picked up _JAVA_OPTIONS: -Xint
Apache Maven 3.0.5
Maven home: /usr/share/maven
Java version: 1.7.0_111, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre
Default locale: en_GB, platform encoding: UTF-8
OS name: "linux", version: "3.4.0+", arch: "amd64", family: "unix"

Now what happens when I try to install jNeuroML:

Simon@DESKTOP-GMI3HL5:$ cd neuroml_dev/jNeuroML
Simon@DESKTOP-GMI3HL5:/neuroml_dev/jNeuroML$ python getNeuroML.py

------ Updating: LEMS/jLEMS -------

Executing: (git branch) in dir: ../jLEMS

• master
  Executing: (git pull) in dir: ../jLEMS

------ Updating: LEMS/LEMS -------

Executing: (git branch) in dir: ../LEMS

• master
  Executing: (git pull) in dir: ../LEMS

------ Updating: LEMS/pylems -------

Executing: (git branch) in dir: ../pylems

• master
  Executing: (git pull) in dir: ../pylems

------ Updating: NeuroML/NeuroML2 -------

Executing: (git branch) in dir: ../NeuroML2

• master
  Executing: (git pull) in dir: ../NeuroML2

------ Updating: NeuralEnsemble/libNeuroML -------

Executing: (git branch) in dir: ../libNeuroML

• master
  Executing: (git pull) in dir: ../libNeuroML

------ Updating: NeuroML/org.neuroml.model.injectingplugin -------

Executing: (git branch) in dir: ../org.neuroml.model.injectingplugin

• master
  Executing: (git pull) in dir: ../org.neuroml.model.injectingplugin
  It's a Java repository, so installing using Maven...
  Executing: (mvn install) in dir: ../org.neuroml.model.injectingplugin
  Picked up _JAVA_OPTIONS: -Xint
  Successful installation using : mvn install!

------ Updating: NeuroML/org.neuroml.model -------
Creating a new directory: ../org.neuroml.model by cloning from GitHub

Executing: (git clone https://github.com/NeuroML/org.neuroml.model) in dir: ..
Cloning into 'org.neuroml.model'...
remote: Counting objects: 2614, done.
remote: Total 2614 (delta 0), reused 0 (delta 0), pack-reused 2614
Receiving objects: 100% (2614/2614), 575.92 KiB | 556.00 KiB/s, done.
Resolving deltas: 100% (1092/1092), done.
Checking connectivity... done.
Executing: (git branch) in dir: ../org.neuroml.model

• master
  Executing: (git pull) in dir: ../org.neuroml.model
  It's a Java repository, so installing using Maven...
  Executing: (mvn install) in dir: ../org.neuroml.model
  Picked up _JAVA_OPTIONS: -Xint
  Error: 1
  Simon@DESKTOP-GMI3HL5:~/neuroml_dev/jNeuroML$

I found a thread for Maven not working on Microsoft's BashOnWindows Github Issues here:

microsoft/WSL#253

I currently have Build 14393. Build 14901 that is reported has Maven working in the link above is an insider build, so it looks like I will have to wait until the Spring 2017 update to have this working. Unless they decide to issue some bug fixes to the wider community before the full update.

[Simon-at-Ely]

Yesterday I manually updated to Windows 10 Build 15063.13 (the Creator Edition)

This was after being tipped off by ArsTechnica that an update tool was now available here: https://www.microsoft.com/en-gb/software-download/windows10?tduid=(9266948c4684727c0ed5fed807fb2bd0)(266696)(1503186)(100796X1556580Xc73af4688b531cee7dd5de8235f0665c)().

I then was able to update from Ubuntu 14.04 to Ubuntu 16.04 using "sudo do-release-upgrade".
I then edited my .bashrc file to remove the line: export _JAVA_OPTIONS="-Xint" and moved java-7-openjdk-amd64 up to version 8 as this had been updated when I moved from Ubuntu 14.04 to 16.04. Trying to compile a Java HelloWorld script revealed that javac is not contained in java-8-openjdk-amd64, but this was quickly fixed with "sudo apt-get install openjdk-8-jdk" which installs java-8-openjdk-headless. Now that I had confirmed that the Java HelloWorld script worked I switched to looking at a:

Maven install of jNeuroML.

As before running python getNeuroML.py runs through the git pull steps, then crashes with error 1 when mvn is invoked.

So next I tried invoking a maven install manually with "mvn clean install" this started OK with:

[INFO] Building jNeuroML 0.7.7

and proceeded to make a series of downloads for example:

Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.4/maven-jar-plugin-2.4.pom (6 KB at 101.7 KB/sec)

There were about 20 of these then it started trying to download some NeuroML specific items which failed and aborted maven:

Downloading: https://repo.maven.apache.org/maven2/org/neuroml1/model/org.neuroml1.model/1.4.6/org.neuroml1.model-1.4.6.pom
[WARNING] The POM for org.neuroml1.model:org.neuroml1.model:jar:1.4.6 is missing, no dependency information available
Downloading: https://repo.maven.apache.org/maven2/org/neuroml/import/org.neuroml.import/1.4.6/org.neuroml.import-1.4.6.pom
[WARNING] The POM for org.neuroml.import:org.neuroml.import:jar:1.4.6 is missing, no dependency information available
Downloading: https://repo.maven.apache.org/maven2/org/neuroml1/model/org.neuroml1.model/1.4.6/org.neuroml1.model-1.4.6.jar
Downloading: https://repo.maven.apache.org/maven2/org/neuroml/import/org.neuroml.import/1.4.6/org.neuroml.import-1.4.6.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.507 s
[INFO] Finished at: 2017-04-06T21:50:33+00:00
[INFO] Final Memory: 11M/131M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project jNeuroML: Could not resolve dependencies for project org.neuroml:jNeuroML:jar:0.7.7: The following artifacts could not be resolved: org.neuroml1.model:org.neuroml1.model:jar:1.4.6, org.neuroml.import:org.neuroml.import:jar:1.4.6: Could not find artifact org.neuroml1.model:org.neuroml1.model:jar:1.4.6 in central (https://repo.maven.apache.org/maven2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

I am not sure why it is looking for the NeuroML specific code on the apache maven repository? Is this code normally held there? I did not have this problem when I recently updated a Fedora install of jNeuroML using the same wifi router so the problem seems to be specific to the BASH on Windows installation. Could it be something to do with Windows Defender blocking something? Any ideas @pgleeson ?

[Simon-at-Ely]

Installation of neuroConstruct

Installation of neuroConstruct went smoothly. I cloned from the GitHub repository with:

git clone https://github.com/NeuralEnsemble/neuroConstruct.git

I then needed to install subversion with:

sudo apt-get install subversion

to run ./updatenC.sh (cd neuroConstruct to run the update)

then 'ant' resulted in a successful build.

I then cloned my AOB mitral cell model:
git clone https://github.com/Simon-at-Ely/AccessoryOlfactoryBulbMitralCell.git
Fired up Cygwin and entered 'xinit startkde -- -listen tcp' in the terminal and 'export DISPLAY=:0.0' into the BASH terminal. Followed with ./nC.sh to fire up neuroConstruct. Here is a screen shot of the mitral cell that I loaded displayed on KDE on Cygwin on Windows talking to Ubuntu 16.04 BASH on Windows:

Next step was to generate Neuron code for the model within neuroConstruct but this produced an error that was fairly simple to sort out:

It was fixed by setting Neuron Home directory in neuroConstructs pull down settings menu to:
/home/Simon/neuron/nrn/x86_64 where I had installed neuron
Trying to launch the resulting Neuron code resulted in another error message:

I need to figured out what I need to enter into neuroConstruct settings to work in the Cygwin display. I have to update the Command line executable used to run external programs. It is currently set to 'konsole' which should be right for the kde desktop that I am using in Cygwin (I also tried Run/run). On the KDE display menu I could not find 'konsole' in a search of application so maybe I need to install kde konsole on Cygwin? Will repost when I figure this out.

[Simon-at-Ely]

Figured out how to get a command line executable to launch when the run simulation button is clicked for the generated code on neuroConstruct. I needed to install the kubuntu desktop on top of Ubuntu 16.04. This is a large installation that takes some time and there is lots of stuff that I do not need like Libre Office and Amrok. Maybe there is a more basic setup that it is possible to install. But this is how I managed the installation:

sudo add-apt-repository ppa:kubuntu-ppa/backports
sudo apt-get update
sudo apt-get install kubuntu-desktop

I also edited my .bashrc file to include:

export DISPLAY=:0.0

I then fired up neuroConstruct changed settings > General Settings and Project Default Settings > Command line: back to the original 'konsole' so now when I generated the Neuron code for my model it worked as expected and displayed in the Cygwin window:

So now I just need to figure out how to get Maven to see the NeuroML repositories and I will have the full set of tools working on BASH on Windows.

[pgleeson]

@Simon-at-Ely Glad to see you got nC running this way. Installing the kubuntu desktop was an option, but it was probably the nuclear option... An easier solution (which is what the warning GUI recommended for Ubuntu desktops) was to change to gnome-terminal in nC settings which is the default terminal on Ubuntu...

Regarding the install of jNeuroML, the org.neuroml.import-1.4.6.jar etc. can't be downloaded from online maven repositories (live most mavenised packages), they need to be built from the latest development sources.

In jNeuroML try:

git checkout development
python getNeuroML.py clean
python getNeuroML.py development

If that fails, you're best going to each of the nml2 repos in turn and typing 'mvn clean install' until you see what the problem is:

../jLEMS
../org.neuroml.model.injectingplugin
../org.neuroml.model
../org.neuroml1.model
../org.neuroml.export
../org.neuroml.import
../jNeuroML

[Simon-at-Ely]

@pgleeson Thanks for your thoughts... I did try setting gnome-terminal in nC along with quite a few other things, but it seems that within BASH on Windows there is no desktop environment installed as a default. This is because it is strictly command line as x-windows is not implemented. I was able to bypass this by installing Cygwin x on Windows and getting it to listen to what I was doing in the BASH terminal. It may have been simpler to find a way to spawn a new BASH terminal to handle running Neuron but I did not manage to find a way to do this. So I opted for the more cumbersome kubuntu desktop in preference to gnome-unity as I have been using KDE for years; first in SUSE then going for the KDE spin of Fedora as I did not feel at home with the Gnome default. Neither was installed so it was my choice which to install on the BASH on Windows version of Ubuntu.

I will give your suggestions a go for jNeuroML over the next day or two and report back with how I get on.

[Simon-at-Ely]

@pgleeson the first command 'git checkout development' executed successfully and 2nd 'python getNeuroML.py clean' partially completed but then generated an error (see below). The 3rd python getNeuroML.py development gave plenty of warnings about unused attributes but then reported a successful build in the end. I will try to validate some channels with it to see if it works and report back.

Simon@DESKTOP-GMI3HL5:/neuroml_dev/jNeuroML$ git checkout development
Branch development set up to track remote branch development from origin.
Switched to a new branch 'development'
Simon@DESKTOP-GMI3HL5:/neuroml_dev/jNeuroML$ python getNeuroML.py clean
Cleaning repos
------ Cleaning: LEMS/jLEMS -------
It's a Java repository, so cleaning using Maven...

Executing: (mvn clean) in dir: ../jLEMS
------ Cleaning: LEMS/LEMS -------
------ Cleaning: LEMS/pylems -------
------ Cleaning: NeuroML/NeuroML2 -------
------ Cleaning: NeuralEnsemble/libNeuroML -------
------ Cleaning: NeuroML/org.neuroml.model.injectingplugin -------
It's a Java repository, so cleaning using Maven...
Executing: (mvn clean) in dir: ../org.neuroml.model.injectingplugin
------ Cleaning: NeuroML/org.neuroml.model -------
It's a Java repository, so cleaning using Maven...
Executing: (mvn clean) in dir: ../org.neuroml.model
------ Cleaning: NeuroML/org.neuroml1.model -------
It's a Java repository, so cleaning using Maven...
Executing: (mvn clean) in dir: ../org.neuroml1.model
Traceback (most recent call last):
File "getNeuroML.py", line 175, in
main()
File "getNeuroML.py", line 79, in main
info = execute_command_in_dir(command, local_dir)
File "getNeuroML.py", line 152, in execute_command_in_dir
p = subprocess.Popen(command, cwd=directory, shell=True, stdout=subprocess.PIPE)
File "/home/Simon/anaconda2/lib/python2.7/subprocess.py", line 711, in init
errread, errwrite)
File "/home/Simon/anaconda2/lib/python2.7/subprocess.py", line 1343, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory: '../org.neuroml1.model'

[Simon-at-Ely]

Well I wasn't expecting that! I had also tried installing jNeuroML natively on Windows as well as on BASH on windows and now in the latest build of Windows 10 the two sides can communicate with each other. So when I tried validating a file with jnml this was the result:

Simon@DESKTOP-GMI3HL5:/NML2_Test/AOB_mitral_cell$ jnml -validate *.nml
/mnt/c/neuroml_dev/jNeuroML/jnml: line 2: $'\r': command not found
/mnt/c/neuroml_dev/jNeuroML/jnml: line 7: $'\r': command not found
/mnt/c/neuroml_dev/jNeuroML/jnml: line 9: $'\r': command not found
/mnt/c/neuroml_dev/jNeuroML/jnml: line 10: $'\r': command not found
/mnt/c/neuroml_dev/jNeuroML/jnml: line 12: $'\r': command not found
/mnt/c/neuroml_dev/jNeuroML/jnml: line 18: syntax error: unexpected end of file
Simon@DESKTOP-GMI3HL5:/NML2_Test/AOB_mitral_cell$ whereis jnml
jnml: /mnt/c/neuroml_dev/jNeuroML/jnml /mnt/c/neuroml_dev/jNeuroML/jnml.bat

I guess I need to put a path in the .bashrc file so that it finds the right version of jnml.

[Simon-at-Ely]

Put a path for JNML_HOME in .bashrc but it still finds the one on the windows side even though the one on the BASH side is now showing up:
Simon@DESKTOP-GMI3HL5:~/NML2_Test/AOB_mitral_cell$ whereis jnml
jnml: /mnt/c/neuroml_dev/jNeuroML/jnml /mnt/c/neuroml_dev/jNeuroML/jnml.bat /home/Simon/neuroml_dev/jNeuroML/jnml /home/Simon/neuroml_dev/jNeuroML/jnml.bat

However putting an explicit path in the command finds the right one and jnml successfully carried out the validation test on the files:

Simon@DESKTOP-GMI3HL5:~/NML2_Test/AOB_mitral_cell$ /home/Simon/neuroml_dev/jNeuroML/jnml -validate *.nml
jNeuroML v0.8.0
Validating: /home/Simon/NML2_Test/AOB_mitral_cell/BK_IAMC_ChannelML.channel.nml
Valid against schema and all tests
No warnings
.../...
Validating: /home/Simon/NML2_Test/AOB_mitral_cell/NML2_SingleCompHHCell.nml
File: /home/Simon/NML2_Test/AOB_mitral_cell/NML2_SingleCompHHCell.nml is not valid against the schema: /Schemas/NeuroML2/NeuroML_v2beta5.xsd!!
cvc-complex-type.4: Attribute 'ion' must appear on element 'channelDensity'.

Validating: /home/Simon/NML2_Test/AOB_mitral_cell/Vm_iMC1_cell_1_origin.sim.nml
Valid against schema and all tests
No warnings
Validating: /home/Simon/NML2_Test/AOB_mitral_cell/Vm_Test.sim.nml
Valid against schema and all tests
No warnings

Validated 13 files: 12 passed, 1 failed

So it looks like I am there, it is possible to install and run all the NeuroML tools on BASH on Windows. Probably some minor adjustments needed to smooth some hiccups out. But at least I have shown that it is now possible.

[pgleeson]

Thanks again for testing this @Simon-at-Ely. Glad to see it can work if required, but as mentioned my advice would always be to either get the toolchain working natively on Windows or (preferably) install a dual boot with Ubuntu...

[Simon-at-Ely]

I did try for sometime to get the toolchain to work natively on Windows and then switched to BASH on Windows as I could get more of it working. From memory one of the sticking points was the python version of Neuron that is available for Windows is really cut down and lacks all the gui elements. But generally getting all the dependencies installed is much easier on Ubuntu than Windows.

I ran my laptop as dual boot for about 5 years but continuously rebooting is a pain especially if files need to be saved to a flash drive to transfer from one side to the other. Admittedly things have got easier on this score with Dropbox and Github. However in presentations it is nice to show off neuroConstruct especially when talking to people that haven't seen it. But then to swap backwards and forwards with a PowerPoint presentation is difficult in Ubuntu as in my experience Wine does not work that well. I also use Excel quite a bit in conjunction with python. Excel is a real pain when used with Wine. If you open two Excel workbooks at once it becomes really unstable and even with just one workbook open simple things like editing the scale on a graph causes a crash. The latter is not so much of a problem as I can do all my plotting in python. But it would be so much easier if I could accomplish all my work without resorting to dual boot.

I am now at the point where I am ready to buy a new laptop so deciding what OS to go with is the first decision that needs to be made. I was experimenting with my current windows laptop and linux desktop try to clarify the suitability of the various options. If Windows were to implement x-windows in their Windows Subsystem for Linux it would make the choice simple. That is not the case at the moment and I cannot be sure that it ever will be the case. An even more unlikely scenario would be that they produce a fully Linux compatible release of Windows so that I could achieve everything I want on Ubuntu (I have tried LibreOffice and its compatibility and capabilities do not meet my requirements). I guess I could look a bit further at the native install of the tool chain on Windows but I did get the impression that it wasn't currently an option when I looked at it in 2016. The final option was that I try with Apple OSX, but I seem to remember that I was advised there was problems with that route also and I would have to learn a new OS from scratch as I have never used it. Apple users say that it is intuitive, but with 30 years of ms-dos/windows and 10 years of Linux experience it may not be quite so intuitive.