feat: use gcs for stream file storage #994

JeffreyDallas · 2024-12-17T03:02:48Z

Description

This pull request changes the following:

New flags for cloud storage type, keys and endpoint
Generate new secrets with solo
Refactor create-topic example to also check submitted message from mirror node
move create_test_account to individual helper file to shared between solo relay test and gcs test
Add test workflow to verify using google storage for stream files upload and download

Related Issues

Closes Charts[Network]: expose variables to values.yaml for enabling and configuring mirror s3 and gcs backups full-stack-testing#711

Test result

https://github.com/hashgraph/solo/actions/runs/12473809302/job/34814887772

Signed-off-by: Jeffrey Tang <[email protected]>

github-actions · 2024-12-17T03:04:23Z

Unit Test Results - Linux

1 files ±0 58 suites ±0 3s ⏱️ -1s
227 tests ±0 227 ✅ ±0 0 💤 ±0 0 ❌ ±0
232 runs ±0 232 ✅ ±0 0 💤 ±0 0 ❌ ±0

Results for commit f29bdf6. ± Comparison against base commit 5b9d937.

♻️ This comment has been updated with latest results.

github-actions · 2024-12-17T03:06:20Z

Unit Test Results - Windows

1 files ±0 58 suites ±0 15s ⏱️ -1s
227 tests ±0 227 ✅ ±0 0 💤 ±0 0 ❌ ±0
232 runs ±0 232 ✅ ±0 0 💤 ±0 0 ❌ ±0

Results for commit f29bdf6. ± Comparison against base commit 5b9d937.

♻️ This comment has been updated with latest results.

github-actions · 2024-12-17T03:44:09Z

E2E Test Report

16 files ±0 121 suites ±0 1h 18m 41s ⏱️ - 2m 23s
256 tests ±0 256 ✅ ±0 0 💤 ±0 0 ❌ ±0
259 runs ±0 259 ✅ ±0 0 💤 ±0 0 ❌ ±0

Results for commit ca46661. ± Comparison against base commit a5fab1e.

github-actions · 2024-12-17T03:44:10Z

E2E Test Report

16 files 121 suites 1h 23m 19s ⏱️
258 tests 258 ✅ 0 💤 0 ❌
261 runs 261 ✅ 0 💤 0 ❌

Results for commit f29bdf6.

♻️ This comment has been updated with latest results.

codacy-production · 2024-12-17T03:44:25Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
Report missing for `5b9d937`¹	✅ 60.78%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`5b9d937`)	Report Missing	Report Missing	Report Missing
Head commit (`f29bdf6`)	20293	17353	85.51%

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#994)	255	155	60.78%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct. ↩

codecov · 2024-12-17T03:48:48Z

Codecov Report

Attention: Patch coverage is 60.00000% with 102 lines in your changes missing coverage. Please review.

Project coverage is 84.67%. Comparing base (362df60) to head (f29bdf6).
Report is 48 commits behind head on main.

Files with missing lines	Patch %	Lines
src/commands/network.ts	40.44%	81 Missing ⚠️
src/commands/mirror_node.ts	44.44%	20 Missing ⚠️
src/core/config_manager.ts	88.88%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #994      +/-   ##
==========================================
- Coverage   85.03%   84.67%   -0.36%     
==========================================
  Files          75       75              
  Lines       20043    20293     +250     
  Branches     1333     1286      -47     
==========================================
+ Hits        17044    17184     +140     
- Misses       2946     3092     +146     
+ Partials       53       17      -36

Files with missing lines	Coverage Δ
src/commands/flags.ts	`74.61% <100.00%> (+0.89%)`	⬆️
src/core/constants.ts	`97.20% <100.00%> (+0.15%)`	⬆️
src/core/k8.ts	`87.95% <100.00%> (-0.50%)`	⬇️
version.ts	`100.00% <100.00%> (ø)`
src/core/config_manager.ts	`94.27% <88.88%> (-0.27%)`	⬇️
src/commands/mirror_node.ts	`73.90% <44.44%> (-1.70%)`	⬇️
src/commands/network.ts	`67.92% <40.44%> (-4.56%)`	⬇️

... and 18 files with indirect coverage changes

Signed-off-by: Jeffrey Tang <[email protected]>

…google-storage

Signed-off-by: Jeffrey Tang <[email protected]>

src/commands/network.ts

…google-storage Signed-off-by: Jeffrey Tang <[email protected]> # Conflicts: # src/commands/network.ts

Signed-off-by: Jeffrey Tang <[email protected]>

Co-authored-by: Jeromy Cannon <[email protected]> Signed-off-by: JeffreyDallas <[email protected]>

Signed-off-by: Jeffrey Tang <[email protected]>

…/solo into 00711-D-google-storage

Signed-off-by: Jeffrey Tang <[email protected]>

src/commands/network.ts

Signed-off-by: Jeffrey Tang <[email protected]>

mishomihov00

Review applies to:
.github/workflows/flow-gcs-test.yaml

src/commands/network.ts

Signed-off-by: Jeromy Cannon <[email protected]>

Signed-off-by: Jeffrey Tang <[email protected]>

…google-storage

## [0.33.0](v0.32.0...v0.33.0) (2025-01-13) ### Features * add support for genesis throttle configuration ([#1026](#1026)) ([d8903a8](d8903a8)) * enable Blockstreams and uploader ([#1109](#1109)) ([ed6527b](ed6527b)) * inspect existing cluster resources during cluster setup ([#1094](#1094)) ([808516c](808516c)) * production readiness pass custom mirror node postgresql credentials and server address ([#1128](#1128)) ([f8b7e41](f8b7e41)) * support using GCS for backup uploader ([#1130](#1130)) ([5fe2f4e](5fe2f4e)) * update genesis-network.json to add roster entries and more ([#1122](#1122)) ([c27d1d6](c27d1d6)) * Updating solo to leverage changes after moving acme-cluster-issuer and haproxy-ingress out of mirror-node-explorer ([#1079](#1079)) ([7965894](7965894)) * use gcs for stream file storage ([#994](#994)) ([04d5806](04d5806)) ### Bug Fixes * add missing images, fix debug page images ([#1092](#1092)) ([4073d31](4073d31)) * Bug when running `solo cluster connect` after fresh install ([#1123](#1123)) ([1cd0e28](1cd0e28)) * bump solo-charts to 0.42.2 to pull in blockStreams fix ([#1138](#1138)) ([c482895](c482895)) * bump versions of hedera, json rpc relay, and mirror node ([#1147](#1147)) ([0784b46](0784b46)) * clarify how to use nvm ([#1125](#1125)) ([bdc1873](bdc1873)) * commit message of auto update ([#1106](#1106)) ([a3231f9](a3231f9)) * external link rendering ([#1118](#1118)) ([28af1fe](28af1fe)) * file size empty error when throttle file is provided on command line ([#1140](#1140)) ([5180a76](5180a76)) * fixed release artifact version issue and updated readme link ([#1090](#1090)) ([84e6611](84e6611)) * **lease:** reworks the previous fix addressing renewal issues ([#1105](#1105)) ([5edb9c9](5edb9c9)) * should run `npm install` before build ([#1117](#1117)) ([3398875](3398875)) * solo deployment create should use email address in local config if it is already there instead of prompting the user for it ([#1112](#1112)) ([362df60](362df60)) * started adding a yachli example and taskfile ([#1132](#1132)) ([8753723](8753723)) * update to AccountManager._getNodeClient() to parallel node connection setup ([#1091](#1091)) ([4959a12](4959a12))

swirlds-automation · 2025-01-13T21:31:40Z

🎉 This PR is included in version 0.33.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

swirlds-automation · 2025-01-21T02:02:36Z

🎉 This PR is included in version 0.33.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

JeffreyDallas added 2 commits December 16, 2024 13:36

Add test

81d89c4

Signed-off-by: Jeffrey Tang <[email protected]>

Save

ca46661

Signed-off-by: Jeffrey Tang <[email protected]>

JeffreyDallas requested review from a team as code owners December 17, 2024 03:02

JeffreyDallas requested review from rbarkerSL and nathanklick December 17, 2024 03:02

JeffreyDallas self-assigned this Dec 17, 2024

JeffreyDallas marked this pull request as draft December 17, 2024 03:02

JeffreyDallas added 2 commits December 18, 2024 11:30

Save

cb79a05

Signed-off-by: Jeffrey Tang <[email protected]>

change prefix

2e84977

Signed-off-by: Jeffrey Tang <[email protected]>

JeffreyDallas marked this pull request as ready for review December 19, 2024 16:30

save

5ce7874

Signed-off-by: Jeffrey Tang <[email protected]>

JeffreyDallas marked this pull request as draft December 19, 2024 17:05

JeffreyDallas added 9 commits December 19, 2024 15:19

save

105cfbe

Signed-off-by: Jeffrey Tang <[email protected]>

Merge commit '861c579e353be7d452ff52e12c152123f5a84aef' into 00711-D-…

16ac154

…google-storage

update

43ceb61

Signed-off-by: Jeffrey Tang <[email protected]>

save

05174b9

Signed-off-by: Jeffrey Tang <[email protected]>

update bucket name

30b9b94

Signed-off-by: Jeffrey Tang <[email protected]>

fix id token

e878555

Signed-off-by: Jeffrey Tang <[email protected]>

save

a87fc71

Signed-off-by: Jeffrey Tang <[email protected]>

save

d13b2b6

Signed-off-by: Jeffrey Tang <[email protected]>

update service account

ae8ddd2

Signed-off-by: Jeffrey Tang <[email protected]>

jeromy-cannon reviewed Jan 6, 2025

View reviewed changes

src/commands/network.ts Outdated Show resolved Hide resolved

jeromy-cannon reviewed Jan 6, 2025

View reviewed changes

src/commands/network.ts Outdated Show resolved Hide resolved

Merge commit '6e73f5798098766b9cf31cfd5995767a33565ad5' into 00711-D-…

4e0f3c6

…google-storage Signed-off-by: Jeffrey Tang <[email protected]> # Conflicts: # src/commands/network.ts

JeffreyDallas dismissed mishomihov00’s stale review via 4e0f3c6 January 6, 2025 15:08

JeffreyDallas and others added 7 commits January 6, 2025 09:11

update flag comment

d18254c

Signed-off-by: Jeffrey Tang <[email protected]>

reuse secret name

fad4e1e

Signed-off-by: Jeffrey Tang <[email protected]>

Update src/commands/network.ts

158849e

Co-authored-by: Jeromy Cannon <[email protected]> Signed-off-by: JeffreyDallas <[email protected]>

Update src/commands/network.ts

460a6b7

Co-authored-by: Jeromy Cannon <[email protected]> Signed-off-by: JeffreyDallas <[email protected]>

add matrix for workflow

18588ed

Signed-off-by: Jeffrey Tang <[email protected]>

Merge branch '00711-D-google-storage' of https://github.com/hashgraph…

63694d9

…/solo into 00711-D-google-storage

update

cecdec9

Signed-off-by: Jeffrey Tang <[email protected]>

jeromy-cannon reviewed Jan 6, 2025

View reviewed changes

src/commands/network.ts Outdated Show resolved Hide resolved

revert meaning

ae45773

Signed-off-by: Jeffrey Tang <[email protected]>

mishomihov00 previously approved these changes Jan 7, 2025

View reviewed changes

jeromy-cannon reviewed Jan 7, 2025

View reviewed changes

src/commands/network.ts Outdated Show resolved Hide resolved

Update src/commands/network.ts

1076232

Signed-off-by: Jeromy Cannon <[email protected]>

jeromy-cannon dismissed mishomihov00’s stale review via 1076232 January 7, 2025 14:12

bump solo-charts version to pick up gcs/s3 changes in charts

fcc4fd5

Signed-off-by: Jeromy Cannon <[email protected]>

jeromy-cannon previously approved these changes Jan 7, 2025

View reviewed changes

fix jobname

45f0969

Signed-off-by: Jeffrey Tang <[email protected]>

JeffreyDallas dismissed jeromy-cannon’s stale review via 45f0969 January 7, 2025 15:09

Merge commit '5b9d937cd5db4d7ed63bbe695671a4857e6abda6' into 00711-D-…

f29bdf6

…google-storage

jeromy-cannon approved these changes Jan 7, 2025

View reviewed changes

jeromy-cannon merged commit 04d5806 into main Jan 7, 2025
41 of 42 checks passed

jeromy-cannon deleted the 00711-D-google-storage branch January 7, 2025 18:19

swirlds-automation added the released label Jan 13, 2025

swirlds-automation added the released on @0.33.x label Jan 21, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: use gcs for stream file storage #994

feat: use gcs for stream file storage #994

JeffreyDallas commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024

github-actions bot commented Dec 17, 2024 •

edited

Loading

codacy-production bot commented Dec 17, 2024 •

edited

Loading

codecov bot commented Dec 17, 2024 •

edited

Loading

mishomihov00 left a comment

swirlds-automation commented Jan 13, 2025

swirlds-automation commented Jan 21, 2025

feat: use gcs for stream file storage #994

feat: use gcs for stream file storage #994

Conversation

JeffreyDallas commented Dec 17, 2024 • edited Loading

Description

Related Issues

Test result

github-actions bot commented Dec 17, 2024 • edited Loading

Unit Test Results - Linux

github-actions bot commented Dec 17, 2024 • edited Loading

Unit Test Results - Windows

github-actions bot commented Dec 17, 2024

E2E Test Report

github-actions bot commented Dec 17, 2024 • edited Loading

E2E Test Report

codacy-production bot commented Dec 17, 2024 • edited Loading

Coverage summary from Codacy

See diff coverage on Codacy

See your quality gate settings Change summary preferences

Footnotes

codecov bot commented Dec 17, 2024 • edited Loading

Codecov Report

mishomihov00 left a comment

Choose a reason for hiding this comment

swirlds-automation commented Jan 13, 2025

swirlds-automation commented Jan 21, 2025

JeffreyDallas commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024 •

edited

Loading

codacy-production bot commented Dec 17, 2024 •

edited

Loading

codecov bot commented Dec 17, 2024 •

edited

Loading