Test notebooks with test-eagle
#2447
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
This allows us to limit `least_busy` to only select backends on the open plan.
frankharkins
force-pushed
the
FH/test-functions
branch
from
75a68aa to
70e9aa4
Compare
| "tags": [ | ||
| "raises-exception" | ||
| ] |
There was a problem hiding this comment.
Nice feature of most Jupyter execution tools. The exception in this cell is intentional.
| @@ -261,7 +275,8 @@ | |||
| "maxcut_job = solver.run(\n", | |||
| " problem=problem_as_str,\n", | |||
| " problem_type=\"maxcut\",\n", | |||
| " instance=hub + \"/\" + group + \"/\" + project,\n", | |||
| " instance=instance, # E.g. \"ibm-q/open/main\"\n", | |||
| " backend_name=backend_name, # E.g. \"ibm_kyiv\"\n", | |||
There was a problem hiding this comment.
This is the main change to most notebooks: We load the backend_name and instance variables in a hidden cell, then use them in the function call with a comment giving an example of the value.
There was a problem hiding this comment.
I've added a new patch. This is because we've added the test_eagle_us-east device to the instance we use for testing. The problem is that this device is always the least busy, so it'll be selected when we're doing our hardware runs. We can't just switch to the open instance for this test because I'm regularly out of credits and the priority is too low to run all the jobs in our time window.
This new patch picks the least busy backend that's also available on the open plan, regardless of which instance you're using.
| @@ -74,7 +74,7 @@ def get_notebook_jobs(args: argparse.Namespace) -> Iterator[NotebookJob]: | |||
|
|
|||
| patch = config.get_patch_for_group(group) | |||
|
|
|||
| if patch: | |||
| if patch and not "# nb-tester: allow-write" in patch: | |||
There was a problem hiding this comment.
New minor feature: We can allow writing notebooks to disk with specific patches, see the new README entry for more information.
|
Closing because I'm going to split this into smaller PRs |
Changes
New test strategy
This PR adds a new "extended" test that runs the notebooks on IBM Quantum's
test-eagledevice. This device accepts jobs and returns random bit strings in a very short time period, which is great for testing notebooks that submit jobs too large for a local simulator. This also allows us to test functions notebooks, which take a backend name (string) of an IBM Quantum device, rather than the backend object itself.I originally hoped we could test notebooks with
test-eaglein CI, but the jobs can be flaky and take just a little too long to complete. Running these jobs on every commit is too expensive and could block writers.Instead, I've added a new "extended" test that we can trigger manually on branches when we need it. Examples could be dependabot PRs, or PRs to functions notebooks; when the PR is ready, we'll trigger the workflow on the PR branch to check the notebooks run correctly. To trigger the workflow, go to Actions > Test notebook (extended) > Run workflow, then select the PR branch and click "Run workflow".
Re-categorise notebooks
With the new "extended" test strategy available, I've recategorised the notebooks so we can test them. I've also modified some notebooks slightly so they'll run from start to finish.
New patch type
This PR adds a new type of patch:
qiskit-ibm-runtime-open, which patchesleast_busyto only return backends in the open plan. This is to stopleast_busyfrom always returningtest_eagle_us-east, which is always the least busy backed in our testing instance.Allow writing some patches to disk
Finally, this PR adds a feature to allow writing some patched notebooks to disk. We haven't allowed this in the past, but it makes sense with the new
qiskit-ibm-runtime-openpatch.After writing this all out, I realise I should split this into smaller PRs. Stay tuned.