Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HUDI-6902] Containerize the Azure CI #10512

Merged
merged 2 commits into from
Feb 10, 2024
Merged

[HUDI-6902] Containerize the Azure CI #10512

merged 2 commits into from
Feb 10, 2024
+154 −123

Conversation

linliu-code
Copy link
Contributor

@linliu-code linliu-code commented Jan 16, 2024 •
edited
Loading

Change Logs

The reason for this change is that 4th module of Azure CI could fail non-deterministically, and our best guess is that this module is somehow correlated with other modules, e.g., folders. Before we have bandwidth to root cause it, we decide to containerize Azure CI to rule out the possibility of module correlation.

In details,

  1. We created a base docker image named "apachehudi_azure_ci_base" and push it to apachehudi docker hub repo.
  2. For each Azure CI module, we build a image by copying the hudi repo with the PR changes into the base image.
  3. For each Azure CI module, we run the build-and-test command in both UT and FT tests. The idea steps should be build-testUT-testFT; however, it is hard to make the container keeps the file changes in the build step after switch to the following test tasks in Azure.
  4. We modified the pom file to use the maven-thrift-plugin for Azure CI tests but keep using the existing logic for thrift compilation for GH CI tests.

Impact

Isolate Azure CI test modules. Potentially we can do it for all modules.

Risk level (write none, low medium or high below)

None.

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch 2 times, most recently from 4d759f3 to a70247f Compare January 17, 2024 00:42
@linliu-code linliu-code changed the title [HUDI-6902] Run Azure tests on different agents [HUDI-6902] Run Azure tests on containers Jan 17, 2024
@linliu-code
Copy link
Contributor Author

@hudi-bot run azure

@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch from a70247f to a5529ad Compare January 17, 2024 18:30
@linliu-code
Copy link
Contributor Author

@hudi-bot run azure

@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch from a5529ad to 40592ef Compare January 18, 2024 00:09
@linliu-code
Copy link
Contributor Author

@hudi-bot run azure

@linliu-code linliu-code changed the title [HUDI-6902] Run Azure tests on containers [HUDI-6902] Ensure "UT FT other modules" run on different machine Jan 18, 2024
@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch from 40592ef to b13310f Compare January 18, 2024 01:41
@linliu-code linliu-code changed the title [HUDI-6902] Ensure "UT FT other modules" run on different machine [HUDI-6902] Containerize the Azure tests Jan 18, 2024
@linliu-code
Copy link
Contributor Author

@hudi-bot run azure

@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch from b13310f to dc93423 Compare January 18, 2024 02:07
@linliu-code
Copy link
Contributor Author

@hudi-bot run azure

@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch 2 times, most recently from d47977a to 1139a8d Compare January 19, 2024 22:39
@linliu-code
Copy link
Contributor Author

@hudi-bot run azure

@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch from 1139a8d to 05791c0 Compare January 22, 2024 18:32
@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch 6 times, most recently from 7e5a620 to 4bb5eb0 Compare January 30, 2024 19:28
@linliu-code
Copy link
Contributor Author

@hudi-bot run azure

@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch from 4bb5eb0 to 3222501 Compare January 31, 2024 08:18
@linliu-code
Copy link
Contributor Author

@hudi-bot run azure

@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch 3 times, most recently from 56fad09 to 7b46d61 Compare January 31, 2024 16:36
@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch 3 times, most recently from a35d70e to 06c2064 Compare February 5, 2024 23:23
@linliu-code linliu-code changed the title [HUDI-6902] Containerize the Azure tests [HUDI-6902] Containerize the Azure CI 4th module Feb 6, 2024
@linliu-code linliu-code requested a review from codope February 6, 2024 06:32
vinothchandar
vinothchandar requested changes Feb 6, 2024
View reviewed changes
Copy link
Member

@vinothchandar vinothchandar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this on. Left some comments.

Dockerfile Show resolved Hide resolved
azure-pipelines-20230430.yml Show resolved Hide resolved
azure-pipelines-20230430.yml Outdated
- task: Maven@4
displayName: maven install
- task: Docker@2
displayName: "build docker image"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the image is not built each CI run, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We copy the hudi repo into the container, which is a build step, which took < 1 minutes.

azure-pipelines-20230430.yml Outdated
jdkVersionOption: '1.8'
- task: Maven@4
displayName: UT other modules
containerRegistry: 'onehouse-docker-hub'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets keep onehouse out of this PR. Can we just use the apachehudi docker hub

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it is just a name for the connection. I can replace it.

hudi-platform-service/hudi-metaserver/hudi-metaserver-server/pom.xml Show resolved Hide resolved
hudi-platform-service/hudi-metaserver/hudi-metaserver-server/pom.xml Show resolved Hide resolved
@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch from 06c2064 to 9d1384b Compare February 7, 2024 22:43
@linliu-code linliu-code changed the title [HUDI-6902] Containerize the Azure CI 4th module [HUDI-6902] Containerize the Azure CI Feb 7, 2024
@linliu-code
Copy link
Contributor Author

@vinothchandar will add the azure-ci folder in a separate PR.

@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch 5 times, most recently from 4209f33 to 7455402 Compare February 8, 2024 03:40
@linliu-code
Copy link
Contributor Author

@hudi-bot run azure

@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch from 7455402 to 215e06a Compare February 8, 2024 05:59
@linliu-code
Copy link
Contributor Author

@hudi-bot run azure

@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch 2 times, most recently from 931f696 to 17c685d Compare February 9, 2024 18:45
@linliu-code linliu-code force-pushed the HUDI-6902_fix_azure_tests branch from 17c685d to 87ca223 Compare February 9, 2024 22:26
@hudi-bot
Copy link

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@vinothchandar vinothchandar merged commit 81cddbb into apache:master Feb 10, 2024
31 checks passed
linliu-code added a commit to linliu-code/hudi that referenced this pull request Feb 13, 2024
@linliu-code
Revert "[HUDI-6902] Containerize the Azure CI (apache#10512)"
d7d62da 
This reverts commit 81cddbb.
@linliu-code linliu-code mentioned this pull request Feb 13, 2024
Merged
4 tasks
yihua pushed a commit that referenced this pull request Feb 27, 2024
@linliu-code @yihua
[HUDI-6902] Containerize the Azure CI (#10512)
a0ebac8 
* [HUDI-6902] Containerize the Azure tests

* remove warning message
yihua pushed a commit that referenced this pull request May 3, 2024
@linliu-code @yihua
[HUDI-6902] Containerize the Azure CI (#10512)
2a5fd3c 
* [HUDI-6902] Containerize the Azure tests

* remove warning message
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vinothchandar vinothchandar vinothchandar approved these changes

@codope codope Awaiting requested review from codope

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@linliu-code @hudi-bot @vinothchandar @codope