Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move structure.calc setting into the initializer #1068

Merged
merged 11 commits into from
Jun 16, 2023

Conversation

liamhuber
Copy link
Member

@liamhuber liamhuber commented May 29, 2023

Over in pyiron_contrib #694, @sinazadeh was running into trouble that the NEB protocol was breaking with Gpaw jobs. Looking into it, this is because the protocols explicitly call:

self._job = pr.load(self._job_name)
self._job.interactive_open()
self._job.interactive_initialize_interface()

The current implementation of AseJob.interactive_initialize_interface calls for self._structure.calc.set_label(self.working_directory + "/"), but self._structure.calc is only set in AseJob.run_if_interactive! Thus the protocols run into trouble by explicitly initializing the interface instead of just blindly jumping into a run.

run_if_interactive calls interactive_initialize_interface if it hasn't been called already, some my hope is that we can simply move the setting of self._structure.calc into the initializer with zero side-effects. Certainly it makes more sense to me to do this in initialization rather than running, all else being equal.

As discussed in #1066, I don't have Gpaw on my local machine right now, so I can't verify that this works beforehand. Looking at the .ci_support/environment.yml it also looks like we don't install Gpaw for the CI unit tests. And what's more, looking at the test suite itself, I don't see any tests for the interactive behaviour. Is there someone with GPaw access who can manually see if this change introduces any bugs?

EDIT: Jan released an OSX-compliant version of gpaw on conda forge, so I'll just check it myself.

EDIT: Turns out you need it in both places, so that if you're running interactively and update to a new structure object you still guarantee a calculator.

@liamhuber liamhuber added the help wanted Extra attention is needed label May 29, 2023
@liamhuber liamhuber marked this pull request as ready for review May 29, 2023 19:56
@coveralls
Copy link

coveralls commented May 29, 2023

Pull Request Test Coverage Report for Build 5292881233

  • 1 of 6 (16.67%) changed or added relevant lines in 1 file are covered.
  • 1 unchanged line in 1 file lost coverage.
  • Overall coverage decreased (-0.01%) to 67.665%

Changes Missing Coverage Covered Lines Changed/Added Lines %
pyiron_atomistics/gpaw/pyiron_ase.py 1 6 16.67%
Files with Coverage Reduction New Missed Lines %
pyiron_atomistics/gpaw/pyiron_ase.py 1 40.18%
Totals Coverage Status
Change from base Build 5244326909: -0.01%
Covered Lines: 11162
Relevant Lines: 16496

💛 - Coveralls

@liamhuber
Copy link
Member Author

Unittests / build (macos-latest, 3.11, osx-64-py-3-11 failed with

======================================================================
ERROR: test_convergence_goal (testing.test_serialMaster_non_modal.TestSerialMaster.test_convergence_goal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/pyiron_atomistics/pyiron_atomistics/tests/testing/test_serialMaster_non_modal.py", line 90, in test_convergence_goal
    self.project.wait_for_job(job_ser, interval_in_s=5, max_iterations=50)
  File "/Users/runner/miniconda3/envs/my-env/lib/python3.11/site-packages/pyiron_base/project/generic.py", line 1455, in wait_for_job
    wait_for_job(
  File "/Users/runner/miniconda3/envs/my-env/lib/python3.11/site-packages/pyiron_base/jobs/job/extension/server/queuestatus.py", line 223, in wait_for_job
    raise ValueError(
ValueError: Maximum iterations reached, but the job was not finished.
----------------------------------------------------------------------

I think this was probably just a hiccup on the CI side, sp I'm just going to re-run for now.

@@ -50,8 +50,6 @@ def run_static(self):
self.server.run_mode = pre_run_mode

def run_if_interactive(self):
if self.structure.calc is None:
self.set_calculator()
super(AseJob, self).run_if_interactive()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we need this here. For example when running and MD calculation we would update the structure internally so while the first structure might have a calculator the next structure might not.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't gotten to testing it yet, but what you say sounds reasonable. I'll add a test when I get back to this PR and we'll find out. If so, it should be here and in the new location.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you were spot on that this was necessary. The following breaks when I only set the calculator during interactive open, but works fine now that I make sure it's set on open and at run time:

from pyiron_atomistics import Project

pr = Project("tmp")
pr.remove_jobs(silently=True, recursive=True)

job = pr.create.job.Gpaw("gpaw", delete_existing_job=True)
job.input["encut"] = 100
job.input["kpoints"] = 3*[1]

s1 = pr.atomistics.structure.bulk("Al", cubic=True)
s2 = pr.atomistics.structure.bulk("Al", cubic=True)
s2.positions[0, 0] += 0.2

job.structure = s1
with job.interactive_open() as ijob:
    ijob.run()
    ijob.structure = s2
    ijob.run()

The Gpaw package is not part of the regular environment, but is there for notebooks, so I included this in the integration tests.

In the process I discovered that the AseJob class is actually abstract, only gets used for Gpaw, and is completely undocumented and untested. This displeases me, but is outside the scope of this PR.

In case you interactively opened and have since supplied a new structure.
Only in the integration tests, since we don't have the package in the main tests.
@liamhuber liamhuber added integration Start the notebook integration tests for this PR and removed help wanted Extra attention is needed labels Jun 15, 2023
@liamhuber
Copy link
Member Author

The integration tests are shot straight to hell, but not the part from this PR.

======================================================================
[52](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:53)
ERROR: test_Fe_ferro_C (tests_sphinx_sphinx_check_all.TestSphinx)
[53](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:54)
----------------------------------------------------------------------
[54](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:55)
Traceback (most recent call last):
[55](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:56)
  File "/home/runner/work/pyiron_atomistics/pyiron_atomistics/test_integration/tests_sphinx_sphinx_check_all.py", line 73, in test_Fe_ferro_C
[56](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:57)
    job.structure.set_initial_magnetic_moments([2, 2])
[57](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:58)
  File "/home/runner/work/pyiron_atomistics/pyiron_atomistics/pyiron_atomistics/atomistics/structure/atoms.py", line 2631, in set_initial_magnetic_moments
[58](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:59)
    raise ValueError("magmoms can be collinear or non-collinear.")
[59](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:60)
ValueError: magmoms can be collinear or non-collinear.
[60](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:61)

[61](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:62)
======================================================================
[62](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:63)
ERROR: test_Fe_ferro_constraint (tests_sphinx_sphinx_check_all.TestSphinx)
[63](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:64)
----------------------------------------------------------------------
[64](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:65)
Traceback (most recent call last):
[65](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:66)
  File "/home/runner/work/pyiron_atomistics/pyiron_atomistics/test_integration/tests_sphinx_sphinx_check_all.py", line 125, in test_Fe_ferro_constraint
[66](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:67)
    job.structure.set_initial_magnetic_moments([2, 2])
[67](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:68)
  File "/home/runner/work/pyiron_atomistics/pyiron_atomistics/pyiron_atomistics/atomistics/structure/atoms.py", line 2631, in set_initial_magnetic_moments
[68](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:69)
    raise ValueError("magmoms can be collinear or non-collinear.")
[69](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:70)
ValueError: magmoms can be collinear or non-collinear.
[70](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:71)

[71](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:72)
======================================================================
[72](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:73)
ERROR: test_Fe_nonmag (tests_sphinx_sphinx_check_all.TestSphinx)
[73](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:74)
----------------------------------------------------------------------
[74](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:75)
Traceback (most recent call last):
[75](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:76)
  File "/home/runner/work/pyiron_atomistics/pyiron_atomistics/test_integration/tests_sphinx_sphinx_check_all.py", line 61, in test_Fe_nonmag
[76](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:77)
    job.structure.set_initial_magnetic_moments([2, 2])
[77](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:78)
  File "/home/runner/work/pyiron_atomistics/pyiron_atomistics/pyiron_atomistics/atomistics/structure/atoms.py", line 2631, in set_initial_magnetic_moments
[78](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:79)
    raise ValueError("magmoms can be collinear or non-collinear.")
[79](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:80)
ValueError: magmoms can be collinear or non-collinear.
[80](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:81)

[81](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:82)
======================================================================
[82](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:83)
ERROR: test_sxextopt_Fe (tests_sphinx_sphinx_check_all.TestSphinx)
[83](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:84)
----------------------------------------------------------------------
[84](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:85)
Traceback (most recent call last):
[85](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:86)
  File "/home/runner/work/pyiron_atomistics/pyiron_atomistics/test_integration/tests_sphinx_sphinx_check_all.py", line 166, in test_sxextopt_Fe
[86](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:87)
    spx.structure.set_initial_magnetic_moments([2, 2])
[87](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:88)
  File "/home/runner/work/pyiron_atomistics/pyiron_atomistics/pyiron_atomistics/atomistics/structure/atoms.py", line 2631, in set_initial_magnetic_moments
[88](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:89)
    raise ValueError("magmoms can be collinear or non-collinear.")
[89](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:90)
ValueError: magmoms can be collinear or non-collinear.
[90](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:91)

[91](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:92)
======================================================================
[92](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:93)
FAIL: test_Al_minimize (tests_sphinx_sphinx_check_all.TestSphinx)
[93](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:94)
----------------------------------------------------------------------
[94](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:95)
Traceback (most recent call last):
[95](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:96)
  File "/home/runner/work/pyiron_atomistics/pyiron_atomistics/test_integration/tests_sphinx_sphinx_check_all.py", line 106, in test_Al_minimize
[96](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:97)
    self.assertGreater(E[0], E[1], 'Energy not decreased')
[97](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:98)
AssertionError: -57.19326599140298 not greater than -57.19326592068059 : Energy not decreased
[98](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:99)

[99](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:100)
----------------------------------------------------------------------
[100](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:101)
Ran 12 tests in 29.004s
[101](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:102)

[102](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:103)
FAILED (failures=1, errors=4, skipped=1)
[103](https://github.com/pyiron/pyiron_atomistics/actions/runs/5283708743/jobs/9560297100?pr=1068#step:6:104)
Error: Process completed with exit code 1.

@liamhuber liamhuber removed the integration Start the notebook integration tests for this PR label Jun 15, 2023
@@ -26,7 +25,7 @@
__date__ = "Sep 1, 2018"


class AseJob(GenericInteractive):
class AseJob(GenericInteractive, ABC):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the AseJob now an abstract class? I thought the use case of this class is that ASE atoms which have a corresponding calculator object attached can be executed directly inside pyiron using this class.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made it an abstract class because it was only used as a parent to GpawJob and it had a method which, unless a user makes special modifications to a particular attribute using extra-pyiron tools, gets called and raises a NotImplementedError. That smells so much to me like an abstract class with an abstract method that I made it one.

You can't even pr.create.job this class.

To appease you I will remove the abstractness, but this class is terrible software.

@jan-janssen
Copy link
Member

self._job = pr.load(self._job_name)
self._job.interactive_open()
self._job.interactive_initialize_interface()

I am not even sure if we want to support this syntax. the recommended syntax would be:

self._job = pr.create.job.Gpaw(self._job_name)
self._job.server.run_mode = "interactive"
self._job.run()

@liamhuber
Copy link
Member Author

So to be fair, my memory is not very good, but I am nonetheless 90% certain that one of the original objectives of interactive_open was to avoid this business of setting attributes to strings and have things accessible in a tab completeable way. So I think we probably should support it.

Anyhow, that is (a) in an entirely different repository, and (b) not what is being fixed here. The fundamental problem I have with HEAD is that for gpaw you need to call run before you can call interactive_initialize_interface. Needing to call run before initialize is just nonsense.

@jan-janssen
Copy link
Member

You can use self._job.server.run_mode.interactive = True to my knowledge interactive_open() is an internal function and should not be called by the user. It is just not marked with _ because the IDEs complain.

@liamhuber
Copy link
Member Author

You can use self._job.server.run_mode.interactive = True to my knowledge interactive_open() is an internal function and should not be called by the user. It is just not marked with _ because the IDEs complain.

And a user is not using it, a developer is inside a job class over on contrib.

Again though, the point is that one should not have to call run before calling initialize. If you have a problem with the fine details of how the protocol jobs are implemented, please go raise an issue on contrib. I don't see any need to discuss the best way to set the interactive mode here.

@liamhuber
Copy link
Member Author

Actually, I just realized now there are tests so this usage is in this PR. Sorry, it is appropriate to discuss it here. And I still feel interactive open is the right way to do it.

@liamhuber liamhuber added the integration Start the notebook integration tests for this PR label Jun 16, 2023
@liamhuber
Copy link
Member Author

I am not even sure if we want to support this syntax. the recommended syntax would be:

self._job = pr.create.job.Gpaw(self._job_name)
self._job.server.run_mode = "interactive"
self._job.run()

Here is the source:

    def interactive_open(self):
        """
        Set the run mode to interactive.

        This is the same as setting :attr:`.server.run_mode.interactive`.

        Must be called before :meth:`.run()` is called.
        """
        self.server.run_mode.interactive = True
        return _WithInteractiveOpen(self)

Literally all job.interactive_open() does is act as an alias for setting the server run mode and giving the option to use with syntax. Where are you getting the idea that this is not meant for users? I am doubling-down and saying that job.interactive_open() (even better, using with) is exactly the syntax we want to encourage for this behaviour. It's clear and easy to read, accomplishes exactly the desired functionality, and is tab-completable.

@liamhuber
Copy link
Member Author

Integration test failures are unrelated, see #1085.

That means the tests are passing. IMO this is ready to be merged.

@pmrv
Copy link
Contributor

pmrv commented Jun 16, 2023

I'm not sure anymore whether we discussed it anywhere in writing but interface_open returns a context manager precisely so that users can do with job.interactive_open() as int: ..., so that definitely is a user facing function. That's not the case for interactive_initialize_interface, but we are taking about library code there, so I don't see a big issue.

@liamhuber liamhuber added integration Start the notebook integration tests for this PR and removed integration Start the notebook integration tests for this PR labels Jun 16, 2023
@liamhuber liamhuber merged commit bec9730 into main Jun 16, 2023
@delete-merged-branch delete-merged-branch bot deleted the asejob_set_calc_with_initialize branch June 16, 2023 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
integration Start the notebook integration tests for this PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants