Add Support for GCS and Azure backups #401

gunnarsundberg · 2022-07-20T00:06:07Z

pgBackRest already supports backups to GCS buckets and Azure Storage. This PR removes the default S3 config to allow for alternate providers, adds a simple method of configuring a GCP service account key through secrets, and documentation for how to configure the Helm Chart to use S3, GCS, or Azure.

Issue: #272 and #389

agronholm · 2022-07-20T07:49:09Z

charts/timescaledb-single/templates/statefulset-timescaledb.yaml

@@ -510,6 +516,11 @@ spec:
          name: {{ template "timescaledb.fullname" . }}-pgbackrest
          defaultMode: 416 # 0640 permissions
          optional: true
+      - name: pgbackrest-secrets


You just made this secret mandatory, not just for those wanting to enable GCS backups, but for everyone.

Would it be mandatory since line 523 is optional: true?

I stand corrected. I didn't see that and didn't know about the optional option.

charts/timescaledb-single/README.md

agronholm · 2022-07-21T18:50:52Z

Question: doesn't #389 affect this use case at all?

gunnarsundberg · 2022-07-21T19:16:37Z

Yes it does. #390 would fix this, but adds Azure documentation through comments in values.yaml and does not store the Azure key through secrets. It may be confusing to have GCS and S3 documented in the admin guide, with Azure documented in values.yaml and a different key config method from GCS.

agronholm · 2022-07-21T19:21:15Z

Yes, it would be preferable to get these changes coordinated (I am an Azure user myself).

gunnarsundberg · 2022-07-21T19:32:09Z

Would you suggest I wait to see how 390 shakes out, or just go ahead and add Azure to this PR? Seems like 390 is hung up on something unrelated.

agronholm · 2022-07-21T19:33:57Z

You should probably coordinate with its author. I'm just a concerned citizen :)

paulfantom

Awesome work! 💯 Could you just fix those lint issues?

agronholm · 2022-09-01T08:24:00Z

@gunnarsundberg Do you intend to finish this PR? I kinda need it. Let me know if I can help in any way, or if you'd like me to finish it myself.

gunnarsundberg · 2022-09-12T22:30:35Z

@agronholm I intend to finish it but have been quite busy. If you'd like it finished in the immediate future I'm cool with you doing it. If not I'll pick it back up at a later time 🙂

agronholm · 2022-09-13T14:48:07Z

@agronholm I intend to finish it but have been quite busy. If you'd like it finished in the immediate future I'm cool with you doing it. If not I'll pick it back up at a later time slightly_smiling_face

The easiest way would be if you granted me push access to your fork. Otherwise I could just copy everything to my fork and make a new PR, but I see that as unnecessary hassle.

agronholm · 2022-09-14T08:29:05Z

@paulfantom would you mind approving the test run? I believe I've fixed the issue.

DanChugani · 2022-12-14T02:20:44Z

Is there anyway that we could rerun the tests to see the failure - would be nice to have this rolled into the main branch.

shooit · 2023-01-02T14:13:51Z

Would also love to have this merged 🤞. We want to go to production soon on GKE and need support for backups to GCS.

agronholm · 2023-01-02T17:51:51Z

Would also love to have this merged crossed_fingers. We want to go to production soon on GKE and need support for backups to GCS.

We're probably moving to the StackGres operator for deploying TimescaleDB.

Signed-off-by: Gunnar Sundberg <[email protected]>

nhudson · 2023-01-19T08:33:51Z

@gunnarsundberg can you fix the linting issues? Looks like you just need to bump the helm chart version. 0.30.0 -> 0.31.0 should be fine.

Linting charts...

------------------------------------------------------------------------------------------------------------------------
 Charts to be processed:
------------------------------------------------------------------------------------------------------------------------
 timescaledb-single => (version: "0.30.0", path: "charts/timescaledb-single")
------------------------------------------------------------------------------------------------------------------------

Linting chart "timescaledb-single => (version: \"0.30.0\", path: \"charts/timescaledb-single\")"
Checking chart "timescaledb-single => (version: \"0.30.0\", path: \"charts/timescaledb-single\")" for a version bump...
Old chart version: 0.30.0
Error: failed linting charts: failed processing charts
New chart version: 0.30.0
------------------------------------------------------------------------------------------------------------------------
 ✖︎ timescaledb-single => (version: "0.30.0", path: "charts/timescaledb-single") > chart version not ok. Needs a version bump! 
make: *** [Makefile:[16](https://github.com/timescale/helm-charts/actions/runs/3955995463/jobs/6776051445#step:6:17): lint] Error 1
------------------------------------------------------------------------------------------------------------------------
failed linting charts: failed processing charts
Error: Process completed with exit code 2.

gunnarsundberg · 2023-01-19T16:19:02Z

@nhudson done!

nhudson · 2023-01-23T14:38:57Z

@gunnarsundberg looks like there is a lint issue and if you don't mind rebasing again?

* add pgbackrest-secrets for gcs service key * add documentation for backing up to gcs * clean up gcs backup documentation and fix example * revert autogenerated toc changes * check for empty pgbackrest secret (for gcs) * remove s3 defaults and add examples for backup providers * add documentation for s3 azure and gcs changes * Fixed linting error * bump chart version * remove trailing space Signed-off-by: Gunnar Sundberg <[email protected]> Co-authored-by: Alex Grönholm <[email protected]>

gunnarsundberg added 4 commits July 19, 2022 18:56

add pgbackrest-secrets for gcs service key

23aebd8

add documentation for backing up to gcs

9f6dd17

clean up gcs backup documentation and fix example

4b33d2f

Merge branch 'master' into master

0262640

agronholm reviewed Jul 20, 2022

View reviewed changes

revert autogenerated toc changes

d5f70e8

gunnarsundberg mentioned this pull request Jul 21, 2022

Fix 382 and 389 #390

Open

gunnarsundberg added 3 commits July 21, 2022 18:06

check for empty pgbackrest secret (for gcs)

b1577aa

remove s3 defaults and add examples for backup providers

6a452e7

add documentation for s3 azure and gcs changes

17072cd

gunnarsundberg changed the title ~~Add Support for GCS backups~~ Add Support for GCS and Azure backups Jul 21, 2022

paulfantom approved these changes Aug 8, 2022

View reviewed changes

Fixed linting error

c5623a2

agronholm requested a review from a team as a code owner September 13, 2022 20:29

github-actions bot added the area/timescaledb-single label Sep 13, 2022

Merge branch 'main' into azure

966dc1e

Merge branch 'main' into master

aefb020

Signed-off-by: Gunnar Sundberg <[email protected]>

bump chart version

83d5354

gunnarsundberg added 2 commits January 24, 2023 00:28

Merge branch 'main' into master

8b12f1b

remove trailing space

5ac5a2d

nhudson approved these changes Jan 24, 2023

View reviewed changes

nhudson merged commit 3a84626 into timescale:main Jan 24, 2023

renovate bot mentioned this pull request Nov 30, 2023

chore(deps): update helm charts timescale/tobs#684

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Support for GCS and Azure backups #401

Add Support for GCS and Azure backups #401

gunnarsundberg commented Jul 20, 2022 •

edited

Loading

agronholm Jul 20, 2022

gunnarsundberg Jul 21, 2022

agronholm Jul 21, 2022

agronholm commented Jul 21, 2022

gunnarsundberg commented Jul 21, 2022 •

edited

Loading

agronholm commented Jul 21, 2022

gunnarsundberg commented Jul 21, 2022

agronholm commented Jul 21, 2022

paulfantom left a comment •

edited

Loading

agronholm commented Sep 1, 2022

gunnarsundberg commented Sep 12, 2022

agronholm commented Sep 13, 2022

agronholm commented Sep 14, 2022

DanChugani commented Dec 14, 2022

shooit commented Jan 2, 2023

agronholm commented Jan 2, 2023

nhudson commented Jan 19, 2023

gunnarsundberg commented Jan 19, 2023 •

edited

Loading

nhudson commented Jan 23, 2023

Add Support for GCS and Azure backups #401

Add Support for GCS and Azure backups #401

Conversation

gunnarsundberg commented Jul 20, 2022 • edited Loading

agronholm Jul 20, 2022

Choose a reason for hiding this comment

gunnarsundberg Jul 21, 2022

Choose a reason for hiding this comment

agronholm Jul 21, 2022

Choose a reason for hiding this comment

agronholm commented Jul 21, 2022

gunnarsundberg commented Jul 21, 2022 • edited Loading

agronholm commented Jul 21, 2022

gunnarsundberg commented Jul 21, 2022

agronholm commented Jul 21, 2022

paulfantom left a comment • edited Loading

Choose a reason for hiding this comment

agronholm commented Sep 1, 2022

gunnarsundberg commented Sep 12, 2022

agronholm commented Sep 13, 2022

agronholm commented Sep 14, 2022

DanChugani commented Dec 14, 2022

shooit commented Jan 2, 2023

agronholm commented Jan 2, 2023

nhudson commented Jan 19, 2023

gunnarsundberg commented Jan 19, 2023 • edited Loading

nhudson commented Jan 23, 2023

gunnarsundberg commented Jul 20, 2022 •

edited

Loading

gunnarsundberg commented Jul 21, 2022 •

edited

Loading

paulfantom left a comment •

edited

Loading

gunnarsundberg commented Jan 19, 2023 •

edited

Loading