Segfault while transferring a model from NEURON to CoreNEURON #1839
Comments
|
@Helveg Pramod and I looked into this together and concluded that the problem is multiple output gids on the same cell that were created with the |
|
threshdetect.mod |
|
@Helveg #1850 is ready for use. (I'm just going to tweak the test to get full coverage of the change before marking as ready for review.) Note that the administrative coding problem of associating gids to using NetCon reference to voltage is more difficul than the strategy of using a NetCon reference to a POINT_PROCESS because there can be any number of POINT_PROCESS instances in a single segment/compartment each associated with their own gid, but only one distinct NetCon reference to source voltage in a given compartment and thus only one gid for the voltage in that compartment. In fact, I realize I need a to augment my test that so that it includes a test that reduces nseg to 1 after creating the net of axons with nseg=5 with each segment voltage having a gid. |
|
@nrnhines : I haven't looked at the all changes and above comments in detail but I am simply tried to run @Helveg's model with new branches Device preparation on node 0 took 0.05 seconds
Device creation on node 0 took 0.0 seconds
Simulator preparation took 33.99 seconds
Simulating...
num_mpi=1
num_omp_thread=1
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2020
Version : 9.0.0 3522a8b (2022-06-19 06:06:57 -0400)
Additional mechanisms from files
exp2syn.mod expsyn.mod glia__dbbs_mod_collection__AMPA__0.mod glia__dbbs_mod_collection__AMPA__granule.mod glia__dbbs_mod_collection__CaL13__0.mod glia__dbbs_mod_collection__Ca__granule_cell.mod glia__dbbs_mod_collection__Cav2_1__0.mod glia__dbbs_mod_collection__Cav2_2__0.mod glia__dbbs_mod_collection__Cav2_3__0.mod glia__dbbs_mod_collection__Cav3_1__0.mod glia__dbbs_mod_collection__Cav3_2__0.mod glia__dbbs_mod_collection__Cav3_3__0.mod glia__dbbs_mod_collection__GABA__0.mod glia__dbbs_mod_collection__GABA__biexp.mod glia__dbbs_mod_collection__GABA__granule.mod glia__dbbs_mod_collection__HCN1__0.mod glia__dbbs_mod_collection__HCN1__golgi.mod glia__dbbs_mod_collection__HCN2__0.mod glia__dbbs_mod_collection__Kca1_1__0.mod glia__dbbs_mod_collection__Kca2_2__0.mod glia__dbbs_mod_collection__Kca3_1__0.mod glia__dbbs_mod_collection__Kir2_3__0.mod glia__dbbs_mod_collection__Km__granule_cell.mod glia__dbbs_mod_collection__Kv1_1__0.mod glia__dbbs_mod_collection__Kv1_5__0.mod glia__dbbs_mod_collection__Kv2_2__0.mod glia__dbbs_mod_collection__Kv3_3__0.mod glia__dbbs_mod_collection__Kv3_4__0.mod glia__dbbs_mod_collection__Kv4_3__0.mod glia__dbbs_mod_collection__Kv7__0.mod glia__dbbs_mod_collection__Leak__0.mod glia__dbbs_mod_collection__Leak__GABA.mod glia__dbbs_mod_collection__NMDA__granule.mod glia__dbbs_mod_collection__NMDA__stellate.mod glia__dbbs_mod_collection__Na__granule_cell.mod glia__dbbs_mod_collection__Na__granule_cell_FHF.mod glia__dbbs_mod_collection__Nav1_1__0.mod glia__dbbs_mod_collection__Nav1_6__0.mod glia__dbbs_mod_collection__cdp5__0.mod glia__dbbs_mod_collection__cdp5__CAM.mod glia__dbbs_mod_collection__cdp5__CAM_GoC.mod glia__dbbs_mod_collection__cdp5__CR.mod glia__dbbs_mod_collection__gap_junction__0.mod glia__dbbs_mod_collection__gap_junction__parallel.mod hh.mod netstim.mod passive.mod pattern.mod stim.mod svclmp.mod
Memory (MBs) : After mk_mech : Max 942.2188, Min 942.2188, Avg 942.2188
Memory (MBs) : After MPI_Init : Max 942.2188, Min 942.2188, Avg 942.2188
Memory (MBs) : Before nrn_setup : Max 942.2188, Min 942.2188, Avg 942.2188
python: /gpfs/bbp.cscs.ch/project/proj16/kumbhar/pramod_scratch/robin_corenrn_model_paper/nrn/external/coreneuron/coreneuron/io/phase2.cpp:1254: void coreneuron::Phase2::populate(coreneuron::NrnThread&, const coreneuron::UserParams&): Assertion `ps->gid_ > -1' failed.
MPT ERROR: Rank 0(g:0) received signal SIGABRT/SIGIOT(6).Does this help? |
|
@nrnhines : Here are the instructions to reproduce the issue:
mkdir -p robin_model && cd robin_model // this will be our working directory
unzip input.zip
/Users/kumbhar/workarena/repos/bbp/nrn/build/install/
export PATH=/Users/kumbhar/workarena/repos/bbp/nrn/build/install/bin/:$PATH
export PYTHONPATH=/Users/kumbhar/workarena/repos/bbp/nrn/build/install/lib/python:$PYTHONPATH
set -ex
git clone [email protected]:dbbs-lab/arborize || echo "OK"
cd arborize && git checkout 0228676a16792421b0212de647526877d2fda451 && cd ..
git clone [email protected]:dbbs-lab/bsb || echo "OK"
cd bsb && git checkout cc03ab7a57024980003de87427388a035832005c && cd ..
git clone [email protected]:dbbs-lab/dbbs-mod-collection || echo "OK"
cd dbbs-mod-collection && git checkout bdd7a50d9c067b143f9a412c6ed03655b3de0898 && cd ..
git clone [email protected]:dbbs-lab/glia || echo "OK"
cd glia && git checkout ae825e090046c479dfd515ce26aaa6be3d7d7a00 && cd ..
git clone [email protected]:dbbs-lab/models || echo "OK"
cd models && git checkout fc8b00ceb2d6133817497ec52fbb92783dbd822c && cd ..
git clone [email protected]:Helveg/patch || echo "OK"
cd patch && git checkout 6b7bcf8f0bb0623ec00cce0f43e59733ab895bdc && cd ..
cp input/*.mod dbbs-mod-collection/dbbs_mod_collection/mod/
rm dbbs-mod-collection/dbbs_mod_collection/mod/glia__dbbs_mod_collection__gap_junction__parallel.mod
python3 -m venv corenrnenv
source corenrnenv/bin/activate
pip install mpi4py
pip install -e bsb
pip install -e models
pip install -e dbbs-mod-collection
pip install -e arborize
pip install -e glia
pip install -e patch
export PATH=/Users/kumbhar/workarena/repos/bbp/nrn/build/install/bin/:$PATH
export PYTHONPATH=/Users/kumbhar/workarena/repos/bbp/nrn/build/install/lib/python:$PYTHONPATH
set -ex
nrnivmodl -coreneuron dbbs-mod-collection/dbbs_mod_collection/mod
. corenrnenv/bin/activate
export GLIA_NOCOMPILE=TRUE
export GLIA_NOLOAD=TRUE
bsb -v 4 simulate cnrn_small --hdf5=`pwd`/input/cnrn_small.hdf5And this produces: + bsb -v 4 simulate cnrn_small --hdf5=/Users/kumbhar/workarena/repos/bbp/robin-paper/input/cnrn_small.hdf5
numprocs=1
NEURON initialised MPI
Alive
import Alive
PreHDF5
PostInstH5
PreSim
insim
preprep
Preparing simulation
ENABLING CORENEURON !!! GPU: False
Load balancing on node 0 took 0.0 seconds
Placing 97 glomerulus
Placing 4 mossy_fibers
Placing 1083 granule_cell
Should we skip? TRUE
Not actually compiling
Placing 2 golgi_cell
Placing 2 purkinje_cell
Placing 6 basket_cell
Placing 12 stellate_cell
Node 0 created 1206 cells
Cell creation on node 0 took 18.71 seconds
Node 0 created 2524 transmitters
Transmitter creation on node 0 took 3.71 seconds
Indexing relays.
Adding 3887 glomerulus_to_granule connections as terminal.
Adding 46 glomerulus_to_golgi connections as terminal.
Adding 97 mossy_to_glomerulus connections as intermediate.
Relays indexed, resolving intermediates.
Intermediate resolved to 685 targets
Intermediate resolved to 1066 targets
Intermediate resolved to 1071 targets
Intermediate resolved to 1111 targets
Relays resolved.
Node 0 needs to relay 101 relays.
Receiver creation on node 0 took 4.09 seconds
Device preparation on node 0 took 0.01 seconds
Device creation on node 0 took 0.0 seconds
Simulator preparation took 26.87 seconds
Simulating...
num_mpi=1
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2020
Version : 9.0.0 3522a8bd (2022-06-19 06:06:57 -0400)
Additional mechanisms from files
exp2syn.mod expsyn.mod glia__dbbs_mod_collection__AMPA__0.mod glia__dbbs_mod_collection__AMPA__granule.mod glia__dbbs_mod_collection__CaL13__0.mod glia__dbbs_mod_collection__Ca__granule_cell.mod glia__dbbs_mod_collection__Cav2_1__0.mod glia__dbbs_mod_collection__Cav2_2__0.mod glia__dbbs_mod_collection__Cav2_3__0.mod glia__dbbs_mod_collection__Cav3_1__0.mod glia__dbbs_mod_collection__Cav3_2__0.mod glia__dbbs_mod_collection__Cav3_3__0.mod glia__dbbs_mod_collection__GABA__0.mod glia__dbbs_mod_collection__GABA__biexp.mod glia__dbbs_mod_collection__GABA__granule.mod glia__dbbs_mod_collection__HCN1__0.mod glia__dbbs_mod_collection__HCN1__golgi.mod glia__dbbs_mod_collection__HCN2__0.mod glia__dbbs_mod_collection__Kca1_1__0.mod glia__dbbs_mod_collection__Kca2_2__0.mod glia__dbbs_mod_collection__Kca3_1__0.mod glia__dbbs_mod_collection__Kir2_3__0.mod glia__dbbs_mod_collection__Km__granule_cell.mod glia__dbbs_mod_collection__Kv1_1__0.mod glia__dbbs_mod_collection__Kv1_5__0.mod glia__dbbs_mod_collection__Kv2_2__0.mod glia__dbbs_mod_collection__Kv3_3__0.mod glia__dbbs_mod_collection__Kv3_4__0.mod glia__dbbs_mod_collection__Kv4_3__0.mod glia__dbbs_mod_collection__Kv7__0.mod glia__dbbs_mod_collection__Leak__0.mod glia__dbbs_mod_collection__Leak__GABA.mod glia__dbbs_mod_collection__NMDA__granule.mod glia__dbbs_mod_collection__NMDA__stellate.mod glia__dbbs_mod_collection__Na__granule_cell.mod glia__dbbs_mod_collection__Na__granule_cell_FHF.mod glia__dbbs_mod_collection__Nav1_1__0.mod glia__dbbs_mod_collection__Nav1_6__0.mod glia__dbbs_mod_collection__cdp5__0.mod glia__dbbs_mod_collection__cdp5__CAM.mod glia__dbbs_mod_collection__cdp5__CAM_GoC.mod glia__dbbs_mod_collection__cdp5__CR.mod glia__dbbs_mod_collection__gap_junction__0.mod hh.mod netstim.mod passive.mod pattern.mod stim.mod svclmp.mod
Memory (MBs) : After mk_mech : Max 763.1367, Min 763.1367, Avg 763.1367
Memory (MBs) : After MPI_Init : Max 763.1445, Min 763.1445, Avg 763.1445
Memory (MBs) : Before nrn_setup : Max 763.1602, Min 763.1602, Avg 763.1602
Assertion failed: (ps->gid_ > -1), function populate, file /Users/kumbhar/workarena/repos/bbp/nrn/external/coreneuron/coreneuron/io/phase2.cpp, line 1254.
[bb-c02dw1tfml85:72943] *** Process received signal ***
[bb-c02dw1tfml85:72943] Signal: Abort trap: 6 (6)I hope this will help to reproduce the issue on your machine as well. I will send you the email with the |
|
In NEURON I see that the number of real cells is 1106 and the number of PreSyn that point to a voltage so that they can check for threshold crossing is 3627 (which is the same as the total number of PreSyn) (So I guess you are using the new multiple-gid-per-cell feature). But the number of PreSyn that have been explicitly associated with a gid (>=0) is 2524 (hence,over in CoreNEURON the assert is triggered). I printed some info about the PreSyn with no gid but don't see much except that they appear to be from the first 1105 created by NetCon with distinct _ref_v.. e.g. |
Context
Overview of the issue
I am running a test example provided by @Helveg and I see segfault why model is being transferred from NEURON to CoreNEURON.
Expected result/behavior
Model should be transferred correctly without segfault and CoreNEURON should be able to run simulation without any error.
NEURON setup
Minimal working example - MWE
I wouldn't say I have miniaml reproducer but "most" of the instructions can be found:
/gpfs/bbp.cscs.ch/project/proj16/kumbhar/pramod_scratch/robin_corenrn_model_paperLogs
On Bb5 I did:
and backtrace from core file shows:
Issue seems to be coming from
nrniv/nrncore_write/data/cell_group.cppinCellGroup::datumindex_fill():nrn/src/nrniv/nrncore_write/data/cell_group.cpp
Line 215 in fe11246
Note that the
printfshowing channel name is:glia__dbbs_mod_collection__Kca2_2__0 dparam=0x1f972a60 data=0x24ddd1c0 j=0 etype=26 ca_ion@nrnhines : might be easy to discuss/debug this via Skype.
The text was updated successfully, but these errors were encountered: