runtime issue in latest version

[amoradinejad]

Hi Joe,

We are developing an extension of CosmoSIS for galaxy clustering analysis, in which we had previously used older version of CosmoSIS. Recently, I tried to update our pipeline with the latest version of CosmoSIS, but am facing various problems with the modifications.

At the moment, I am facing problems with running the pipeline when using various samplers. When using "test" sampler, the pipeline runs ok and I get a computed value of the likelihood. But when I use "emcee", I get the following error,

Traceback (most recent call last):
  File "/cosmosis/env/bin/cosmosis", line 4, in <module>
    status = cosmosis.main.main()
             ^^^^^^^^^^^^^^^^^^^^
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/main.py", line 560, in main
    return run_cosmosis(ini=args.inifile, pool=None, override=args.params, profile_mem=args.mem, profile_cpu=args.profile, variables=args.variables, only=args.only)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/main.py", line 410, in run_cosmosis
    sampler.config()
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/samplers/emcee/emcee_sampler.py", line 78, in config
    p0_norm = sample_ball(center_norm, sigma_norm, size=self.nwalkers)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/samplers/sampler.py", line 333, in sample_ball
    return sample_ellipsoid(p0, covmat, size)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/samplers/sampler.py", line 321, in sample_ellipsoid
    return np.random.multivariate_normal(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "numpy/random/mtrand.pyx", line 4233, in numpy.random.mtrand.RandomState.multivariate_normal
ValueError: cannot reshape array of size 0 into shape (0) 

If using a different sampler, for instance Polychord, I get a different error after the first evaluation of the likelihood:

Python(32552,0x1f92ccf80) malloc: Heap corruption detected, free list is damaged at 0x600002539640
*** Incorrect guard value: 13813798514139068622
Python(32552,0x1f92ccf80) malloc: *** set a breakpoint in malloc_error_break to debug
Fatal Python error: Aborted

Current thread 0x00000001f92ccf80 (most recent call first):
  File "/cosmosis/cosmosis-standard-library/boltzmann/class/class_interface.py", line 349 in execute
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/runtime/module.py", line 246 in execute
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/runtime/pipeline.py", line 649 in run
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/runtime/pipeline.py", line 1142 in run_parameters
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/runtime/pipeline.py", line 1370 in likelihood
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/runtime/pipeline.py", line 1238 in run_results
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/samplers/polychord/polychord_sampler.py", line 181 in likelihood
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/samplers/polychord/polychord_sampler.py", line 267 in sample
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/samplers/polychord/polychord_sampler.py", line 212 in execute
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/main.py", line 72 in sampler_main_loop
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/main.py", line 425 in run_cosmosis
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/main.py", line 560 in main
  File "/cosmosis/env/bin/cosmosis", line 4 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, cython.cimports.libc.math, scipy._lib._ccallback_c, scipy.special._ufuncs_cxx, scipy.special._ufuncs, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy.special._specfun, scipy.special._comb, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.special._ellip_harm_2, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._direct, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.interpolate._bspl, yaml._yaml, fitsio._fitsio_wrap, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, classy (total: 84)
zsh: abort      cosmosis  ini_files/desi/Gps_rsd_Abacus_fnl30_z0p95_1kf_kmaxp25.in

Is there any known issue with latest cosmosis version, and for instance python 3.12?

Thank you in advance.

[joezuntz]

Hi @amoradinejad

How many parameters are you varying here? The first error suggests that maybe they are all fixed, which would work with the test sampler but might break the others (there should definitely be a better error message - sorry about that!)

The print-out of all the parameter ranges should show it.

Cheers,
Joe

[amoradinejad]

Hi Joe, thanks for the quick response. Indeed in the run I sent the error message for, the parmaeters were fixed. But now varying two parmaeters h and linear galaxy bias, and keeping the rest fixed,

Parameter Priors
----------------
cosmological_parameters--h0       ~ U(0.55, 0.85)
cosmological_parameters--ombh2    ~ delta(0.02237)
cosmological_parameters--omch2    ~ delta(0.12)
cosmological_parameters--a_s      ~ delta(2.083e-09)
cosmological_parameters--n_s      ~ delta(0.9649)
cosmological_parameters--nnu      ~ delta(3.046)
cosmological_parameters--fnl      ~ delta(30.0)
cosmological_parameters--ps0      ~ delta(0.0)
cosmological_parameters--ps02     ~ delta(0.0)
cosmological_parameters--ps2      ~ delta(0.0)
cosmological_parameters--ps4      ~ delta(0.0)
cosmological_parameters--pc0      ~ delta(0.0)
cosmological_parameters--pc2      ~ delta(0.0)
cosmological_parameters--pc4      ~ delta(0.0)
cosmological_parameters--pc6      ~ delta(0.0)
galaxy_bias--b1                   ~ U(0.9, 3.5)
galaxy_bias--b2                   ~ delta(-0.5)
galaxy_bias--bg2                  ~ delta(-0.5)
galaxy_bias--btd                  ~ delta(0.5)

now, I get similar errors from emcee and polychord after one or a few evaluations of the likelihood,

Python(33160,0x1f92ccf80) malloc: Heap corruption detected, free list is damaged at 0x600003838180
*** Incorrect guard value: 13817658873700274002
Python(33160,0x1f92ccf80) malloc: *** set a breakpoint in malloc_error_break to debug
Fatal Python error: Aborted

Current thread 0x00000001f92ccf80 (most recent call first):
  File "/cosmosis/env/lib/python3.12/site-packages/cosmosis/runtime/module.py", line 246 in execute
  File "/cosmosi/env/lib/python3.12/site-packages/cosmosis/runtime/pipeline.py", line 649 in run
  File "/cosmosi/env/lib/python3.12/site-packages/cosmosis/runtime/pipeline.py", line 1142 in run_parameters
  File "/cosmosi/env/lib/python3.12/site-packages/cosmosis/runtime/pipeline.py", line 1370 in likelihood
  File "/cosmosi/env/lib/python3.12/site-packages/cosmosis/runtime/pipeline.py", line 1238 in run_results
  File "/cosmosi/env/lib/python3.12/site-packages/cosmosis/samplers/emcee/emcee_sampler.py", line 8 in log_probability_function
  File "/cosmosi/env/lib/python3.12/site-packages/emcee/ensemble.py", line 640 in __call__
  File "/cosmosi/env/lib/python3.12/site-packages/emcee/ensemble.py", line 496 in compute_log_prob
  File "/cosmosis/env/lib/python3.12/site-packages/emcee/ensemble.py", line 351 in sample
  File "/cosmosi/env/lib/python3.12/site-packages/cosmosis/samplers/emcee/emcee_sampler.py", line 154 in execute
  File "/cosmosi/env/lib/python3.12/site-packages/cosmosis/main.py", line 72 in sampler_main_loop
  File "/cosmosi/env/lib/python3.12/site-packages/cosmosis/main.py", line 425 in run_cosmosis
  File "/cosmosi/env/lib/python3.12/site-packages/cosmosis/main.py", line 560 in main
  File "/cosmosi/env/bin/cosmosis", line 4 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, cython.cimports.libc.math, scipy._lib._ccallback_c, scipy.special._ufuncs_cxx, scipy.special._ufuncs, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy.special._specfun, scipy.special._comb, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.special._ellip_harm_2, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._direct, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.interpolate._bspl, yaml._yaml, fitsio._fitsio_wrap, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, classy, scipy.special.cython_special, scipy.stats._stats, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._biasedurn, scipy.stats._stats_pythran, scipy.stats._levy_stable.levyst, scipy.stats._ansari_swilk_statistics, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.ndimage._nd_image, scipy.ndimage._rank_filter_1d, _ni_label, scipy.ndimage._ni_label (total: 98)
zsh: abort      cosmosis euclid_runs/ini_files/desi/Gps_rsd_Abacus_fnl30_z0p95_1kf_kmaxp25.in

[joezuntz]

The first line of the traceback tells us it's happening inside a module, so we'll need to figure out which one. You can do these two to help narrow it down:

• set verbosity = debug in the [runtime] section
• run with the --segfaults flag