Backups with errors aren't marked as partially complete

[carlpett]

I had a permission issue with my cloud credentials which resulted in Ark not being able to snapshot volumes. However, without going through the logs, it seems like there is no indication of this failure: ark backup get lists it as Completed, and ark backup describe my-backup only has a field for Validation errors, which is empty (reasonably enough).

Shouldn't there be some more indication that errors happened at least? Since I didn't see any errors, I restored the backup, and get to a semi-broken state since the pv/pvc objects aren't created.

[ncdc]

This should show up as an error. Could you please let us know what type of issue you had with your credentials so we can try to reproduce the problem? Could you please also share any relevant lines from the backup’s log and from the ark server pod’s log that show snapshot issues? Thanks!

…

On Mon, May 21, 2018 at 2:11 AM Calle Pettersson ***@***.***> wrote: I had a permission issue with my cloud credentials which resulted in Ark not being able to snapshot volumes. However, without going through the logs, it seems like there is no indication of this failure: ark backup get lists it as ´Completed, andark backup describe my-backuponly has a field forValidation errors`, which is empty (reasonably enough). Shouldn't there be some more indication that errors happened at least? Since I didn't see any errors, I restored the backup, and get to a semi-broken state since the pv/pvc objects aren't created. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#509>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAABYstei_0NdiKz96i9dMj0Qg1st_FFks5t0lqhgaJpZM4UGevw> .

[carlpett]

There were two issues, actually. Some setup-info for context:
We're running on Azure, with acs-engine to create our clusters (currently k8s v1.9.2). I've set up the Ark storage account in a resource group k8s-infra, and created a service principal k8s-ark. Clusters are running in a separate resource group per environment, eg k8s-dev, k8s-staging, etc.

First, I had not given the service principal access on the right resource groups (I cannot follow the guide entirely, since I cannot give subscription-wide access to service accounts, and made a manual mistake here). So this caused a bunch of 403:s (but still successful backup state).

Then, I had misconfigured the Secret with the environment vars, so AZURE_RESOURCE_GROUP pointed to k8s-infra, rather than the resource group of the cluster. This caused 404:s, since it couldn't find the pvc:s. This also didn't cause the backup to fail.

I'll sanitize the logs and post them as a gist in a few minutes

[carlpett]

Here's the output from the first scenario: https://gist.github.com/carlpett/dbb1d336461a6de8207facdd7685b2c1

[ncdc]

@carlpett I'm surprised this showed up as Completed given the snapshot errors. We'll see if we can reproduce and track down where the issue is.

[carlpett]

@ncdc From a quick read through the code, could it be that the issue is that any error returned is ignored here? (Line 214 in the below snippet)

https://github.com/heptio/ark/blob/dc8c66b30525e54df4d8e177f84b54655cc762f1/pkg/backup/item_backupper.go#L195-L215

[ncdc]

@carlpett this is it! Thank you! We'll get this fixed in master asap (unless you want to do a PR?).

[carlpett]

Nice! I'll gladly make a PR :)

[ncdc]

Somewhat related: #511