Causalml fails to install xgboost even with xgboost present in my conda env

[kgolyaev]

Installing causalml through pip fails in my environment with the following error:

Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/4f/ck1srdbj21s8j7nbkbdn6zdm0000gn/T/pip-install-n76pirmd/xgboost/setup.py", line 42, in <module>
        LIB_PATH = libpath['find_lib_path']()
      File "/private/var/folders/4f/ck1srdbj21s8j7nbkbdn6zdm0000gn/T/pip-install-n76pirmd/xgboost/xgboost/libpath.py", line 48, in find_lib_path
        'List of candidates:\n' + ('\n'.join(dll_path)))
    XGBoostLibraryNotFound: Cannot find XGBoost Library in the candidate path, did you install compilers and run build.sh in root path?

I thought I'd try to install xgboost myself, but this did not solve the problem. Here is my system spec:

Darwin PCNAME 18.5.0 Darwin Kernel Version 18.5.0: Mon Mar 11 20:40:32 PDT 2019; root:xnu-4903.251.3~3/RELEASE_X86_64 x86_64

And here is the conda environment inside which I am trying to install causalml:

# platform: osx-64
_py-xgboost-mutex=2.0=cpu_0
absl-py=0.8.1=pypi_0
appnope=0.1.0=py37_0
astor=0.8.0=pypi_0
attrs=19.3.0=py_0
backcall=0.1.0=py37_0
blas=1.0=mkl
bleach=3.1.0=py37_0
ca-certificates=2019.11.27=0
certifi=2019.11.28=py37_0
cycler=0.10.0=py37_0
cython=0.29.14=py37h0a44026_0
dbus=1.13.12=h90a0687_0
decorator=4.4.1=py_0
defusedxml=0.6.0=py_0
econml=0.6=pypi_0
entrypoints=0.3=py37_0
expat=2.2.6=h0a44026_0
freetype=2.9.1=hb4e5f40_0
gast=0.2.2=pypi_0
gettext=0.19.8.1=h15daf44_3
glib=2.63.1=hd977a24_0
google-pasta=0.1.8=pypi_0
grpcio=1.25.0=pypi_0
h5py=2.10.0=pypi_0
icu=58.2=h4b95b61_1
importlib_metadata=1.1.0=py37_0
intel-openmp=2019.4=233
ipykernel=5.1.3=py37h39e3cac_0
ipython=7.10.1=py37h39e3cac_0
ipython_genutils=0.2.0=py37_0
ipywidgets=7.5.1=py_0
jedi=0.15.1=py37_0
jinja2=2.10.3=py_0
joblib=0.14.0=py_0
jpeg=9b=he5867d9_2
jsonschema=3.2.0=py37_0
jupyter=1.0.0=py37_7
jupyter_client=5.3.4=py37_0
jupyter_console=5.2.0=py37_1
jupyter_core=4.6.1=py37_0
keras=2.3.1=pypi_0
keras-applications=1.0.8=pypi_0
keras-preprocessing=1.1.0=pypi_0
kiwisolver=1.1.0=py37h0a44026_0
libcxx=9.0.0=h89e68fa_1
libedit=3.1.20181209=hb402a30_0
libffi=3.2.1=h475c297_4
libgfortran=3.0.1=h93005f0_2
libiconv=1.15=hdd342a3_7
libpng=1.6.37=ha441bb4_0
libsodium=1.0.16=h3efe00b_0
libxgboost=0.90=h4a8c4bd_4
llvm-openmp=9.0.0=h40edb58_0
llvmlite=0.30.0=pypi_0
markdown=3.1.1=pypi_0
markupsafe=1.1.1=py37h1de35cc_0
matplotlib=3.0.3=pypi_0
mistune=0.8.4=py37h1de35cc_0
mkl=2019.4=233
mkl-service=2.3.0=py37hfbe908c_0
mkl_fft=1.0.15=py37h5e564d8_0
mkl_random=1.1.0=py37ha771720_0
more-itertools=7.2.0=py37_0
nbconvert=5.6.1=py37_0
nbformat=4.4.0=py37_0
ncurses=6.1=h0a44026_1
notebook=6.0.2=py37_0
numba=0.46.0=pypi_0
numpy=1.17.4=py37h890c691_0
numpy-base=1.17.4=py37h6575580_0
openssl=1.1.1d=h1de35cc_3
opt-einsum=3.1.0=pypi_0
pandas=0.25.3=py37h0a44026_0
pandoc=2.2.3.2=0
pandocfilters=1.4.2=py37_1
parso=0.5.1=py_0
patsy=0.5.1=py37_0
pcre=8.43=h0a44026_0
pexpect=4.7.0=py37_0
pickleshare=0.7.5=py37_0
pip=19.3.1=py37_0
prometheus_client=0.7.1=py_0
prompt_toolkit=3.0.2=py_0
protobuf=3.11.1=pypi_0
ptyprocess=0.6.0=py37_0
py-xgboost=0.90=py37_4
pygments=2.5.2=py_0
pyparsing=2.4.5=py_0
pyqt=5.9.2=py37h655552a_2
pyrsistent=0.15.6=py37h1de35cc_0
python=3.7.5=h359304d_0
python-dateutil=2.8.1=py_0
python-graphviz=0.13.2=pypi_0
pytz=2019.3=py_0
pyyaml=5.2=pypi_0
pyzmq=18.1.0=py37h0a44026_0
qt=5.9.7=h468cd18_1
qtconsole=4.6.0=py_0
readline=7.0=h1de35cc_5
scikit-learn=0.21.3=py37h27c97d8_0
scipy=1.3.1=py37h1410ff5_0
send2trash=1.5.0=py37_0
setuptools=42.0.2=py37_0
sip=4.19.8=py37h0a44026_0
six=1.13.0=py37_0
sparse=0.8.0=pypi_0
sqlite=3.30.1=ha441bb4_0
statsmodels=0.10.1=py37h1d22016_0
tensorboard=1.15.0=pypi_0
tensorflow=1.15.0=pypi_0
tensorflow-estimator=1.15.1=pypi_0
termcolor=1.1.0=pypi_0
terminado=0.8.3=py37_0
testpath=0.4.4=py_0
tk=8.6.8=ha441bb4_0
tornado=6.0.3=py37h1de35cc_0
traitlets=4.3.3=py37_0
wcwidth=0.1.7=py37_0
webencodings=0.5.1=py37_1
werkzeug=0.16.0=pypi_0
wheel=0.33.6=py37_0
widgetsnbextension=3.5.1=py37_0
wrapt=1.11.2=pypi_0
xgboost=0.90=py37h4a8c4bd_4
xlrd=1.2.0=py37_0
xz=5.2.4=h1de35cc_4
zeromq=4.3.1=h0a44026_3
zipp=0.6.0=py_0
zlib=1.2.11=h1de35cc_3

[peterfoley]

That error message sounds like a common xgboost installation issue rather than something specific to causalml. The threads for this issue and this one list some solutions that might work for you.

To confirm that your xgboost install didn't package libxgboost.dylib properly, run

$ python -c 'import xgboost'

and you'll get the same Cannot find XGBoost Library in the candidate path... as before.

[kgolyaev]

Thank you for the pointers. Unfortunately the problem persists even when xgboost is already present in the conda environment into which I attempt to install causalml. The command you suggested, python -c 'import xgboost' runs successfully with no issues, yet the above error message persists.

I realize the problem is not in the causalml installation per se and rather in its dependency on xgboost. At the same time, I don't understand why using pip install causalml into an environment that already has xgboost installed would fail at the xgboost installation stage.

[jeongyoonlee]

Good to hear from you, Konstantin!

Did you install XGBoost from source? I noticed that when there's an existing XGBoost installation done without pip, pip install xgboost raises an error. However, if your python -c 'import xgboost' runs fine, you should be able to run causalml. The issue here seems like an issue with XGBoost installation itself. I'd recommend removing and reinstalling XGBoost, and try again.

Happy new year!

[peterfoley]

Has anyone been able to reproduce this from a totally fresh environment without conda caching or package reuse involved? The code @fritzo mentioned in #134 doesn't break when I test it in Docker, which makes me think that issues may be related to the conda package cache or something like that.