Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need RAVEN dependency for DISPATCHES #76

Closed
jghouse88 opened this issue Dec 15, 2021 · 40 comments · Fixed by #157
Closed

Need RAVEN dependency for DISPATCHES #76

jghouse88 opened this issue Dec 15, 2021 · 40 comments · Fixed by #157
Assignees
Labels
Priority:High High Priority Issue or PR

Comments

@jghouse88
Copy link

RAVEN is currently not part of the required list of packages for DISPATCHES. Can we add this?

@jghouse88 jghouse88 added the Priority:High High Priority Issue or PR label Dec 15, 2021
@lbianchi-lbl lbianchi-lbl self-assigned this Dec 15, 2021
@lbianchi-lbl
Copy link
Contributor

@jghouse88 I'd be happy to look into this but a quick web search (e.g. https://github.com/idaholab/raven/wiki/installationLinux) seems to suggest that this could be more complicated that I would have guessed. Does anyone have more insight on which components we need and in what way are we planning to manage this as a dependency, e.g. for CI, DISPATCHES users, etc?

@jghouse88
Copy link
Author

@klfrick2 @joshua-cogliati-inl We can use this issue to track the dependency. Please get back here which portions of RAVEN are required for ARMA and if pip will work. Same for TEAL.

@joshua-cogliati-inl
Copy link

joshua-cogliati-inl commented Jan 31, 2022

RAVEN work on creating a wheel: idaholab/raven#1748
(Goal is to finish this by ~ April)

@joshua-cogliati-inl
Copy link

Sample pip package in:
https://test.pypi.org/project/raven-framework/

@radhakrishnatg
Copy link
Contributor

@joshua-cogliati-inl I tried installing Raven with pip and I got this error message:

ERROR: Cannot unpack file C:\Users\Radhakrishna\AppData\Local\Temp\pip-unpack-6gq69zms\simple (downloaded from C:\Users\Radhakrishna\AppData\Local\Temp\pip-req-build-vjcdsati, content-type: text/html; charset=UTF-8); cannot detect archive format
ERROR: Cannot determine archive format of C:\Users\Radhakrishna\AppData\Local\Temp\pip-req-build-vjcdsati

@joshua-cogliati-inl
Copy link

@radhakrishnatg Rats. Just to check, what command did you use to get the package?

@radhakrishnatg
Copy link
Contributor

@joshua-cogliati-inl I tried multiple commands:

pip install -i https://test.pypi.org/simple/ raven-framework
pip install https://test.pypi.org/simple/ raven-framework
pip install raven-framework
pip install https://test.pypi.org/simple/raven-framework

and, none of them worked. All the above commands gave a similar error message.

@joshua-cogliati-inl
Copy link

joshua-cogliati-inl commented Jun 22, 2022

Which python version are you using? (At the moment, I have only built binary wheels for python 3.7)

@radhakrishnatg
Copy link
Contributor

Ah. okay. That could be the issue. I was using python 3.10. I will switch to python 3.7 and try installing it.

@radhakrishnatg
Copy link
Contributor

I see the same error message with python 3.7 as well

(raven-test) C:\Users\Radhakrishna>pip install https://test.pypi.org/project/raven-framework
Collecting https://test.pypi.org/project/raven-framework
  Using cached https://test.pypi.org/project/raven-framework (34 kB)
  ERROR: Cannot unpack file C:\Users\RADHAK~1\AppData\Local\Temp\pip-unpack-gklci2x1\raven-framework (downloaded from C:\Users\RADHAK~1\AppData\Local\Temp\pip-req-build-4ztqac6s, content-type: text/html; charset=UTF-8); cannot detect archive format
ERROR: Cannot determine archive format of C:\Users\RADHAK~1\AppData\Local\Temp\pip-req-build-4ztqac6s

@joshua-cogliati-inl
Copy link

Okay, thank you for letting me know. I will try and figure out the problem, tho' I don't have time in the near future.

@joshua-cogliati-inl
Copy link

python3.7 -m pip install --extra-index-url https://test.pypi.org/simple/ raven-framework works on Linux, I just need to build other platforms.

@joshua-cogliati-inl
Copy link

joshua-cogliati-inl commented Jul 28, 2022

After making a new raven_framework package, I was able to install raven in windows (using the miniconda python) with:

conda create --name test
conda activate test
conda install python=3.7
python -m pip install --extra-index-url https://test.pypi.org/simple/ raven-framework
pip install ray==1.9.*

@radhakrishnatg
Copy link
Contributor

Thank you @joshua-cogliati-inl ! I could install raven in windows without any issues. But, I'm not sure how to use it. I tried to import raven in the following way, and it did not work:

>>> import raven
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'raven'

>>> import raven_framework
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'raven_framework'

Could you please tell me how I can run raven?

@joshua-cogliati-inl
Copy link

joshua-cogliati-inl commented Aug 1, 2022

import ravenframework should work, but the typical way we run raven is running the command line with an input file:
raven_framework input.xml

An example input file is: https://github.com/idaholab/raven/blob/devel/doc/workshop/forwardSampling/exercises/1_sample_and_plot.xml

@PaulTalbot-INL
Copy link

@radhakrishnatg we've got some people working separately on how to use the synthetic histories via the external ROM loader as well as using TEAL for constructing pyomo objectives; as Joshua notes, RAVEN wasn't built as a library but as a code, so directly loading it as a library has limited application.

@radhakrishnatg
Copy link
Contributor

Thank you @joshua-cogliati-inl and @PaulTalbot-INL ! I tried to run the way you suggested, and I got the following error:

(raven-test) C:\Users\Radhakrishna\Desktop\code_repositories>raven_framework price_sample_cluster.xml

Copyright 2017 Battelle Energy Alliance, LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

      .---.        .------######       #####     ###   ###  ########  ###    ###
     /     \  __  /    --###  ###    ###  ###   ###   ###  ###       #####  ###
    / /     \(  )/    --###  ###    ###   ###  ###   ###  ######    ### ######
   //////   ' \/ `   --#######     #########  ###   ###  ###       ###  #####
  //// / // :    :   -###   ###   ###   ###    ######   ####      ###   ####
 // /   /  /`    '---###    ###  ###   ###      ###    ########  ###    ###
//          //..\
===========UU====UU=============================================================
           '//||\`
             ''``

PluginFactory: No installed plugins detected.
Traceback (most recent call last):
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\Scripts\raven_framework.exe\__main__.py", line 7, in <module>
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Driver.py", line 34, in wheelMain
    main(False)
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Driver.py", line 44, in main
    from .Simulation import Simulation
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Simulation.py", line 28, in <module>
    from . import Steps
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Steps\__init__.py", line 22, in <module>
    from .Factory import factory
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Steps\Factory.py", line 24, in <module>
    from .SingleRun import SingleRun
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Steps\SingleRun.py", line 30, in <module>
    from .. import Models
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Models\__init__.py", line 25, in <module>
    from .Dummy import Dummy
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Models\Dummy.py", line 31, in <module>
    from ..Decorators.Parallelization import Parallel
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Decorators\Parallelization.py", line 29, in <module>
    from ray import remote as _remote
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ray\__init__.py", line 91, in <module>
    import ray._raylet  # noqa: E402
  File "python\ray\_raylet.pyx", line 104, in init ray._raylet
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ray\exceptions.py", line 5, in <module>
    from ray.core.generated.common_pb2 import RayException, Language, PYTHON
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ray\core\generated\common_pb2.py", line 37, in <module>
    create_key=_descriptor._internal_create_key),
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\google\protobuf\descriptor.py", line 755, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

As the per the suggestion, I downgraded the protobuf package to 3.20.*

This time it ran with a few deprecation warnings:

PluginFactory: No installed plugins detected.
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\matplotlib\__init__.py:202: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(module.__version__) < minver:
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\setuptools\_distutils\version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\pandas\compat\numpy\__init__.py:10: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  _nlv = LooseVersion(_np_version)
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\pandas\compat\numpy\__init__.py:11: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  _np_version_under1p16 = _nlv < LooseVersion("1.16")
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\pandas\compat\numpy\__init__.py:12: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  _np_version_under1p17 = _nlv < LooseVersion("1.17")
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\pandas\compat\numpy\__init__.py:13: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  _np_version_under1p18 = _nlv < LooseVersion("1.18")
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\pandas\compat\numpy\__init__.py:14: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  _np_version_under1p19 = _nlv < LooseVersion("1.19")
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\pandas\compat\numpy\__init__.py:15: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  _np_version_under1p20 = _nlv < LooseVersion("1.20")
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\setuptools\_distutils\version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\pandas\compat\numpy\function.py:125: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(_np_version) >= LooseVersion("1.17.0"):
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\matplotlib\backend_bases.py:60: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(PILLOW_VERSION) >= "3.4":
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\setuptools\_distutils\version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\xarray\core\dask_array_compat.py:61: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(dask_version) > LooseVersion("2.9.0"):
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\xarray\core\pdcompat.py:45: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(pd.__version__) < "0.25.0":
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\setuptools\_distutils\version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\utils\utils.py:525: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

But it ultimately terminated with an error message:

Traceback (most recent call last):
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\Scripts\raven_framework.exe\__main__.py", line 7, in <module>
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Driver.py", line 34, in wheelMain
    main(False)
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Driver.py", line 203, in main
    raven()
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Driver.py", line 156, in raven
    simulation.run()
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Simulation.py", line 823, in run
    self.executeStep(stepInputDict, stepInstance)
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Simulation.py", line 775, in executeStep
    stepInstance.takeAstep(stepInputDict)
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Steps\Step.py", line 326, in takeAstep
    self._localTakeAstepRun(inDictionary)
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Steps\IOStep.py", line 239, in _localTakeAstepRun
    unpickledObj = pickle.load(open(fileobj.getAbsFile(),'rb+'))
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\shiboken2\files.dir\shibokensupport\__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Models\__init__.py", line 24, in <module>
    from .Model import Model
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\shiboken2\files.dir\shibokensupport\__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\ravenframework\Models\Model.py", line 26, in <module>
    from ..BaseClasses import BaseEntity, Assembler, InputDataUser
  File "C:\Users\Radhakrishna\.conda\envs\raven-test\lib\site-packages\shiboken2\files.dir\shibokensupport\__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
ValueError: attempted relative import beyond top-level package

@joshua-cogliati-inl
Copy link

Hm, that is a strange error, (and one that I don't get from my raven pip install.). It is an error I got when I was first working on the pip package, but it should not be possible now (unless somehow an old version of Driver.py was running with a new version of Model.py)

@joshua-cogliati-inl
Copy link

Oh, this was during unpickling. The pickle file must be from before the imports were fixed for the raven package (~ Mar 2022).

@radhakrishnatg
Copy link
Contributor

@joshua-cogliati-inl Yes, I received the price_sample_cluster.xml and arma.pk files from Konor back in November 2021. Could you please share a file that is compatible with the new raven package? I can run the tests with the new file and see if it works.

@joshua-cogliati-inl
Copy link

The way I have been testing is checking out raven with git, and then going to an example file (like doc/workshop/forwardSampling/exercises/1_sample_and_plot.xml ) and running raven_framework to check it. If you want a new version of something like price_sample_cluster.xml maybe @PaulTalbot-INL can point you at it.

@radhakrishnatg
Copy link
Contributor

Thank you @joshua-cogliati-inl ! I tested with the example you mentioned, and it ran without any errors (I still see the above deprecation warnings though).

@PaulTalbot-INL
Copy link

@radhakrishnatg did Konor also provide the data files on which the price_sample_cluster.xml trains the ARMA model?

We keep compatibility scripts to convert old inputs to new ones, but I don't think the ARMA input structure has changed in a while, though the inner workings have. Mostly, we've made improvements to the internal random number engine for robustness and portability.

@radhakrishnatg
Copy link
Contributor

@PaulTalbot-INL No, I did not receive the data files used for training the ARMA models. If I remember correctly, Konor used the 2019/2020 NYISO data. I don't know if this data is easily accessible or not.

@PaulTalbot-INL
Copy link

Okay. All that data should be available in https://github.com/idaholab/FORCE/tree/main/use_cases/2020_12, we can talk more about which set of data and what should be used for the use case. For now I'll focus on just the mechanics of integration and we'll work on the specifics later.

@joshua-cogliati-inl
Copy link

Hm, the biggest problem with updating to newer than Python 3.7 packages is that cbcpy in pip only goes to 3.7 and that is a dependency for one of RAVEN's plugins.

@joshua-cogliati-inl
Copy link

I am working on supporting Python 3.8 in: idaholab/raven#1933

@joshua-cogliati-inl
Copy link

Python 3.8 support has been added, I'll generate python 3.8 packages next time I create a new package.

@joshua-cogliati-inl
Copy link

I generated new raven_framework-2.2rc3 packages in python 3.7 and 3.8

@radhakrishnatg
Copy link
Contributor

Thank you, @joshua-cogliati-inl I will test it either today or on Monday morning. Does this package also include the TEAL plugin? If not, could you please tell me how I can install it?

@joshua-cogliati-inl
Copy link

The TEAL plugin is separate and can be installed with:
pip install teal-ravenframework --extra-index-url https://test.pypi.org/simple/

@radhakrishnatg
Copy link
Contributor

Thank you, @joshua-cogliati-inl ! I'm seeing similar issues as before. First, it is not able to install all the required packages by itself. I had to install them manually. Second, because of the version check, I had to downgrade the version of some of the packages (numpy, scipy, pandas). I'm not sure if DISPATCHES and IDAES are compatible with the older versions.
Even after installing all the missing packages, the installation failed.
I used pip install -i https://test.pypi.org/simple/ raven-framework==2.2rc3 and it gave me the following error message:

INFO: pip is looking at multiple versions of pandas to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of numdifftools to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of netcdf4 to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of matplotlib to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of imageio to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cloudpickle to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of raven-framework to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install ray[default]==1.12.0 and ray[default]==1.12.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    ray[default] 1.12.1 depends on virtualenv
    ray[default] 1.12.0 depends on virtualenv

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

@joshua-cogliati-inl
Copy link

Hm, sometimes I have had problems with using pip install -i https://test.pypi.org/simple/ raven-framework==2.2rc3 instead of --extra-index-url https://test.pypi.org/simple (The later one allows pip to also use packages at regular pypi.org in addition to test.pypi)
I do agree that our library versions can cause problems.

@radhakrishnatg
Copy link
Contributor

Thank you @joshua-cogliati-inl Yeah, --extra-index-url https://test.pypi.org/simple worked. I did not get the version conflict error this time.

I also tried pip install teal-ravenframework --extra-index-url https://test.pypi.org/simple/ and it worked without any errors. Gabriel gave me notebook that uses TEAL to generate cashflow expressions. The notebook ran without any errors, so I'm assuming that TEAL has been installed correctly. I also ran raven_framework raven-main\doc\workshop\forwardSampling\exercises\1_sample_and_plot.xml, and I didn't get any error message (I did see some deprecation warnings though), so I assume raven is also correctly installed.

@joshua-cogliati-inl
Copy link

joshua-cogliati-inl commented Sep 21, 2022

In python 3.8 it is now in pypi so: python -m pip install teal-ravenframework can be used.

@lbianchi-lbl
Copy link
Contributor

lbianchi-lbl commented Oct 24, 2022

  • @radhakrishnatg will point out a basic "smoke test" piece of code that can be run to see if the RAVEN installation is working
  • @lbianchi-lbl will open a draft PR adding a CI workflow testing the RAVEN integration

@radhakrishnatg
Copy link
Contributor

@lbianchi-lbl After installing Raven, I run raven_framework doc/workshop/forwardSampling/exercises/1_sample_and_plot.xml to see if the installation is successful or not. Here is the link to the Raven repo containing the test file https://github.com/idaholab/raven

Repository owner moved this from In Progress to Done in Dec 2022 Release Dec 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority:High High Priority Issue or PR
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

8 participants