Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Test performance updates #466

Merged
merged 31 commits into from
Dec 31, 2024
Merged

test: Test performance updates #466

merged 31 commits into from
Dec 31, 2024

Conversation

c-ryan-k
Copy link
Member

@c-ryan-k c-ryan-k commented Dec 19, 2024

Major changes:

  • Workflow cleanup and refactoring for performance
  • New workflow solely for containerized e2e tests
  • pytest markers rpsaas and e2e to further define test groupings for pytest
  • new tox envs for various integration test configs using pytest markers
  • Quicker integration tests due to parallelization with pytest-xdist

High-level summary of workflow file changes:

  • int_test.yml
    • instance delete / redeploy is now only done in workflow, not inside init integration tests
    • removed unused inputs and env vars (AIO_SP_* and CA_*)
    • removed OIDC token exchange step (no longer necessary on github workflows)
    • updated tox env to run with parallelization params
  • container_int_test.yml added
    • This is mainly for quickly running e2e tests, no init matrix, default init and create, and runs the containerized python-e2e-int tox env
  • create-k3s-cluster/action.yml action added - can be used in other workflows to quickly start a k3s cluster and setup kubeconfig
  • publish_test_container_image.yml modified to use container_int_test.yml instead of full int_test.yml
  • Dockerfile updated to run python-e2e-int as well as add parallelization to run tests more quickly.

Code changes:

  • tox.ini
    • consolidated integration testing environments into one pytest runner to avoid duplicate configuration - [testenv:python-{init,e2e,rpsaas,edge,all}-int]
    • under setenv you can see the pytest marker inputs for each scenario:
      • init: SCENARIO="init_scenario_test"
      • e2e: SCENARIO="e2e"
      • rpsaas: SCENARIO="rpsaas"
      • edge: SCENARIO="not rpsaas and not init_scenario_test"
      • all: SCENARIO="not init_scenario_test"
    • added pytest-randomly dependency to integration tests to fix test distribution / randomization issue with pytest-xdist
  • azext_edge/tests/generators.py
    • Changed generate_names to use insecure random strings (no need for cryptographically "strong" resource names), allowing pytest-randomly to assist in seeding randomization per test run (allowing us to parallelize pytest with pytest-xdist)
  • Marked check and support int tests with pytest.mark.e2e to break them out to a separate tox env to run in our containerized test environment.
  • Marked asset lifecycle and schema registry lifecycle integration tests with pytest.mark.rpsaas to break them out to a separate tox env

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Thank you for contributing to Azure IoT Operations tooling!

This checklist is used to make sure that common guidelines for a pull request are followed.

General Guidelines

Intent for Production

  • It is expected that pull requests made to default or core branches such as dev or main are of production grade. Corollary to this, any merged contributions to these branches may be deployed in a public release at any given time. By checking this box, you agree and commit to the expected production quality of code.

Basic expectations

  • If introducing new functionality or modified behavior, are they backed by unit and/or integration tests?
  • In the same context as above are command names and their parameter definitions accurate? Do help docs have sufficient content?
  • Have all the relevant unit and integration tests pass? i.e. pytest <project root> -vv. Please provide evidence in the form of a screenshot showing a succesful run of tests locally OR a link to a test pipeline that has been run against the change-set.
  • Have linter checks passed using the .pylintrc and .flake8 rules? Look at the CI scripts for example usage.
  • Have extraneous print or debug statements, commented out code-blocks or code-statements (if any) been removed from the surface area of changes?
  • Have you made an entry in HISTORY.rst which concisely explains your user-facing feature or change?

Azure IoT Operations CLI maintainers reserve the right to enforce any of the outlined expectations.

A PR is considered ready for review when all basic expectations have been met (or do not apply).

Sorry, something went wrong.

c-ryan-k and others added 28 commits December 6, 2024 11:56
consolidate configs in tox.ini

update test markers and tox selection

docker/workflow int test update

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
moved arc connection check into action, and updated int test workflows to use the new action
@c-ryan-k c-ryan-k marked this pull request as ready for review December 20, 2024 00:06
@c-ryan-k c-ryan-k requested a review from digimaun as a code owner December 20, 2024 00:06
Copy link
Contributor

@vilit1 vilit1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update https://github.com/Azure/azure-iot-ops-cli-extension/blob/dev/docs/integration-tests.md if needed and also post some example runs

otherwise looks good enough to me

vilit1 and others added 3 commits December 30, 2024 11:06

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
@c-ryan-k c-ryan-k merged commit b219188 into Azure:dev Dec 31, 2024
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants