Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test functions notebooks in fortnightly cron job #2239

Closed
3 tasks
frankharkins opened this issue Nov 4, 2024 · 0 comments · Fixed by #2463
Closed
3 tasks

Test functions notebooks in fortnightly cron job #2239

frankharkins opened this issue Nov 4, 2024 · 0 comments · Fixed by #2463
Assignees
@frankharkins
Labels
infra 🏗️

Comments

@frankharkins
Copy link
Member

Investigate if we can test any of the functions notebooks in the fortnightly cron job.

Todo:

  • Check which notebooks we can already test
  • (Optional) Add feature to find/replace standardized strings like <YOUR_API_TOKEN> and <YOUR_IQP_GROUP> when testing
  • Add testable notebooks to cron job

Large notebook changes are out of scope of this issue.
@frankharkins frankharkins mentioned this issue Nov 4, 2024
Closed
@frankharkins frankharkins self-assigned this Nov 4, 2024
@frankharkins frankharkins linked a pull request Dec 9, 2024 that will close this issue
Test notebooks with test-eagle #2447
Closed
@frankharkins frankharkins mentioned this issue Dec 11, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Dec 16, 2024
@frankharkins
Test notebooks with test-eagle (#2463)
6ba4574 
There are three changes in this PR

1. (2d2fb9c) **Make notebooks runnable
with nb-tester**

This commit modifies some of the notebooks to make them executable from
start to finish. Most common change is defining `instance` and
`backend_name` in a hidden cell and giving example values through a
comment, rather than having `backend_name = '<your-backend>'`.

2. (945b8d7) **Add extended strategy
and reclassify notebooks**

Adds a new "extended" test that runs the notebooks on IBM Quantum's
`test-eagle` device.

This device accepts jobs and returns random bit strings in a very short
time period, which is great for testing notebooks that submit jobs too
large for a local simulator. This also allows us to test functions
notebooks, which take a backend name (string) of an IBM Quantum device,
rather than the backend object itself.

3. (9f3df9e) **Add extended notebook
tests action**

I originally hoped we could test notebooks with test-eagle in CI, but
the jobs can be flaky and take just a little too long to complete.
Running these jobs on every commit is too expensive and could block
writers.

Instead, I've added a new "extended" test that we can trigger manually
on branches when we need it. Examples could be dependabot PRs, or PRs to
functions notebooks; when the PR is ready, we'll trigger the workflow on
the PR branch to check the notebooks run correctly. To trigger the
workflow, go to Actions > Test notebook (extended) > Run workflow, then
select the PR branch and click "Run workflow".


***

Fixes #2240, fixes #2239, and fixes #2198.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@frankharkins frankharkins

Labels
infra 🏗️
Projects
Docs Planning
Archived in project
Milestone
No milestone
2 participants
@javabster @frankharkins