Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rj/refactor unit tests #109

Open
wants to merge 63 commits into
base: main
Choose a base branch
from
Open

Rj/refactor unit tests #109

wants to merge 63 commits into from

Conversation

neontty
Copy link
Contributor

@neontty neontty commented Aug 19, 2024

major changes:

  • tests are parallelizable now with xdist
  • tests inherit from a base class that makes a copy of a repo for each one for isolated testing. cleans up tests on case exit
  • tests all use a new repo that is outside of the current repo (publicly available HelloProphecy in prophecy-samples org)
  • better coverage parity on most tests with the usage of pytest parameterization to check both scala/python

@neontty neontty requested review from prongs and jainabhinav August 19, 2024 21:36
@neontty neontty marked this pull request as draft August 20, 2024 22:43
@neontty
Copy link
Contributor Author

neontty commented Aug 20, 2024

pytests are failing on github worker because spark sessions are crashing (likely due to OOM or something). runs fine on local machine.

need to try bottlenecking the tasks going to spark submit or just disabling parallel tasks

@neontty neontty self-assigned this Oct 7, 2024
@neontty
Copy link
Contributor Author

neontty commented Oct 10, 2024

changes besides unittests:

  1. 383 files changed but majority of that is because I deleted the entire resources directory which had the test projects. We now rely on external HelloProphecy project for testing
  2. copy-paste refactor in pipeline.py that moved code for finding python dependencies into its own function
  3. changes to regexes in prophecy_build_tool.py were to squash warnings for invalid escape characters. did not change regex functionality.
  4. change to airflow.py is to catch a legitimate error found by unit tests
  5. adding code to maven commands to allow for concurrent m2 cache access

@pytest.mark.parametrize("language", ["python", "scala"])
@pytest.mark.parametrize("command", [build, build_v2])
def test_build_path_pipeline_filter(self, language, command):
project_path = self.python_project_path if language == "python" else self.scala_project_path
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: maybe we can make this a method in IsolatedRepoTestCase

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea, will do

@neontty
Copy link
Contributor Author

neontty commented Oct 18, 2024

fixes #90 #96 #106

neontty added a commit that referenced this pull request Oct 22, 2024
neontty added a commit that referenced this pull request Oct 22, 2024
neontty added a commit that referenced this pull request Oct 22, 2024
neontty added a commit that referenced this pull request Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants