Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[VL] Optimize GHA workflow with nightly built docker image (part-2) #4348

Merged
merged 23 commits into from
Jan 15, 2024
Merged

[VL] Optimize GHA workflow with nightly built docker image (part-2) #4348

merged 23 commits into from
Jan 15, 2024

Conversation

zedong-peng
Copy link
Contributor

@zedong-peng zedong-peng commented Jan 10, 2024 •
edited
Loading

update .github/workflows/velox_be.yml

@github-actions GitHub Actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

PHILO-HE
PHILO-HE reviewed Jan 10, 2024
View reviewed changes
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

3 similar comments
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

4 similar comments
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

1 similar comment
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

zhztheplayer
zhztheplayer reviewed Jan 11, 2024
View reviewed changes
.github/workflows/velox_be.yml
steps:
- uses: actions/checkout@v4
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, this may still be required... Since the script files are stored in Gluten's code base?

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

zhouyuan
zhouyuan reviewed Jan 11, 2024
View reviewed changes
.github/workflows/velox_be.yml
- name: Build Gluten velox third party
run: |
docker exec ubuntu2004-test-$GITHUB_RUN_ID bash -c '
$PATH_TO_GLUTEN_TE/$OS_IMAGE_NAME/gha/gha-checkout/exec.sh '
cd /opt/gluten/ep/build-velox/src && \
./get_velox.sh --velox_home=/opt/velox && \
./build_velox.sh --velox_home=/opt/velox --enable_ep_cache=ON --build_test_utils=ON'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to set run_setup_scipts=OFF as all dependencies are already installed

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@zhztheplayer
Revert "fixup"
a807029 
This reverts commit cced802.
@zhztheplayer
Revert "fixup"
5cab97a 
This reverts commit 4947016.
@zhztheplayer
Revert "fixup"
e3aad11 
This reverts commit 428e64c.
@zhztheplayer
fixup
72ac729
@zhztheplayer
fixup
97bcf9b
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@zhztheplayer
Revert "fixup"
19e3342 
This reverts commit 73b9511.
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@zhztheplayer
Copy link
Member

zhztheplayer commented Jan 15, 2024 •
edited
Loading

I am changing caching to pulling in this PR via the added option DOCKER_PULL_REGISTRY. This should work stably despites the pulled registry could technically be built using different config comparing to local config.

However the caching mechanism is still not working as expected, e.g., it behaves differently on different environments, probably involves different docker versions or something. So let's use pulling first and we can try with caching later once we get it working smoothly for Gluten's images.

@PZD-CHINA @PHILO-HE

PHILO-HE
PHILO-HE approved these changes Jan 15, 2024
View reviewed changes
Copy link
Contributor

@PHILO-HE PHILO-HE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PZD-CHINA, @zhztheplayer, thanks to the both of you!

@PHILO-HE PHILO-HE merged commit 8fb2c35 into apache:main Jan 15, 2024
18 of 19 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_4348_time.csv log/native_master_01_14_2024_00db209eb_time.csv difference percentage
q1 34.41 33.95 -0.467 98.64%
q2 24.65 24.88 0.227 100.92%
q3 38.81 38.01 -0.797 97.95%
q4 37.90 38.25 0.352 100.93%
q5 72.09 71.46 -0.635 99.12%
q6 7.39 7.30 -0.087 98.82%
q7 84.57 85.49 0.920 101.09%
q8 85.53 85.37 -0.163 99.81%
q9 125.58 122.58 -3.003 97.61%
q10 44.98 44.98 -0.002 100.00%
q11 19.59 19.89 0.302 101.54%
q12 29.80 28.31 -1.485 95.02%
q13 46.47 47.25 0.777 101.67%
q14 17.14 19.45 2.308 113.46%
q15 30.60 29.55 -1.043 96.59%
q16 15.26 14.42 -0.838 94.51%
q17 102.19 102.43 0.237 100.23%
q18 145.83 148.49 2.662 101.83%
q19 14.68 13.56 -1.126 92.33%
q20 27.40 27.19 -0.216 99.21%
q21 232.38 228.47 -3.906 98.32%
q22 14.06 13.86 -0.206 98.54%
total 1251.33 1245.14 -6.188 99.51%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_01_15_2024_time.csv log/native_master_01_14_2024_00db209eb_time.csv difference percentage
q1 34.48 33.95 -0.530 98.46%
q2 24.98 24.88 -0.099 99.60%
q3 38.86 38.01 -0.850 97.81%
q4 37.85 38.25 0.404 101.07%
q5 70.70 71.46 0.765 101.08%
q6 5.78 7.30 1.521 126.34%
q7 85.94 85.49 -0.446 99.48%
q8 84.86 85.37 0.509 100.60%
q9 123.88 122.58 -1.297 98.95%
q10 46.04 44.98 -1.059 97.70%
q11 20.63 19.89 -0.736 96.43%
q12 25.93 28.31 2.379 109.17%
q13 46.63 47.25 0.620 101.33%
q14 23.32 19.45 -3.870 83.41%
q15 31.44 29.55 -1.891 93.99%
q16 14.88 14.42 -0.457 96.93%
q17 100.72 102.43 1.709 101.70%
q18 147.03 148.49 1.462 100.99%
q19 13.73 13.56 -0.169 98.77%
q20 27.44 27.19 -0.252 99.08%
q21 227.79 228.47 0.678 100.30%
q22 13.83 13.86 0.022 100.16%
total 1246.73 1245.14 -1.588 99.87%

@zedong-peng zedong-peng deleted the GHA-workflow-2 branch January 16, 2024 05:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhouyuan zhouyuan zhouyuan left review comments

@zhztheplayer zhztheplayer zhztheplayer left review comments

@PHILO-HE PHILO-HE PHILO-HE approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@zedong-peng @zhztheplayer @GlutenPerfBot @zhouyuan @PHILO-HE