Improve the error msg for OCI v1.0 registries or registries that don't support deletion API

[FeynmanZhou]

What is the areas you experience the issue in?

Notation CLI

What is not working as expected?

When using Notation CLI to sign an artifact in OCI v1.0 compliant registries such as GHCR or registries that don't support deletion API (e.g. Docker Hub), Notation will return an unexpected error message and may mislead or confuse users. It happens when signing an artifact for the second time.

What did you expect to happen?

Improve the error message to be more descriptive and user-friendly, and tell users about the right result exactly. Giving an example: Successfully signed <registry>/<repository>@<digest>. Please note that Notation can't delete the referrers index since this is an OCI v1.0 compliant registry or the deletion API is disabled by the registry.

How can we reproduce it?

We have a user who signed a sample image in GHCR. It returned the following error message when this user signed the same image for the second time:

$ notation sign --signature-format cose --key $REMOTE_KEY_NAME ghcr.io/demo-user/python@sha256:xxxx
Error: failed to push signature to registry with error: failed to push manifest: failed to delete dangling referrers index sha256:xxx for referrers tag sha256-1ba1561: DELETE "https://ghcr.io/v2/demo-user/python/manifests/sha256:e1b7d6": response status code 405: unsupported: The operation is unsupported.

It may confuse users why this signing behavior failed due to an old referrers index manifest deletion failure. From an end user's point of view, the user is left with the impression that the signing failed. Now the user is left wondering why this happened and if she/he doesn't know the internals of OCI registries she/he may think the issue is with Notation.

In fact, the signing is successful and the signature has been attached to the image but it doesn't tell users the right result unless users view the referenced signatures using notation list as below.

ghcr.io/demo-user/python@sha256:xxxx
└── application/vnd.cncf.notary.signature
    ├── sha256:85f65b60a6e260d52ba5f8d556d444320b92c2ac140ec8d4944dbd8b03deb18c
    └── sha256:96130e5a5e314ef90e56e24559e48065d2889c39661b9b3a944e6862e1e51c0a

Describe your environment

Ubuntu 20.08

What is the version of your Notation CLI or Notation Library?

Notation CLI v1.0.0-RC.3

[patrickzheng200]

On a second thought, printing out a warning like this in CLI:
Warning: failed to delete dangling referrers index sha256:xxx for referrers tag sha256-1ba1561: DELETE "https://ghcr.io/v2/demo-user/python/manifests/sha256:e1b7d6": response status code 405: unsupported: The operation is unsupported.
might be too technical for users. After all, the user's command was just notation sign xxx. Warning like this is difficult to read and understand. Hence, I think we could keep notation CLI unchanged. Instead, we detect this error in notation-go, and add a log.Warn() there. Once this error occurs, notation-go still returns nil error to caller (CLI or non-CLI users), so in CLI's point of view, everything works fine. But if the user Sign with the -v flag enabled, they will see the warning printed out.
What do you guys think? @priteshbandi @sajayantony @toddysm @shizhMSFT @yizha1 @FeynmanZhou @qweeah

[qweeah]

Agree on not showing this ignorable error to CLI users.

[toddysm]

What will be the response that the CLI user will see?

[patrickzheng200]

What will be the response that the CLI user will see?

Without -v flag set, i.e., notation sign ..., user will see Successfully signed <reference>.
With -v flag set, i.e., notation sign ... -v, user will see:

Warning: xxx
Successfully signed <reference>

[toddysm]

Why do we put the Warning first?

[patrickzheng200]

Why do we put the Warning first?

That's because this Warning is just an ordinary log in the Sign process like other logs we already have. Logs are printed out in the workflow, there may be other logs before and/or after this Warning.
Successfully signed <reference> is not a log. It's the final print out of notation CLI's notation sign command. If -v flag is not turned on, all the logs are hidden but one can still see this final message. (This behavior aligns with other commands in notation CLI.)

[shizhMSFT]

I think the warning message should always be there even if -v is not toggled.

It can be something like

$ notation sign --signature-format cose --key $REMOTE_KEY_NAME ghcr.io/demo-user/python@sha256:xxxx
Warning:  Removal of outdated referrers index is not supported by the remote registry. Garbage collection may be required.

[patrickzheng200]

I think the warning message should always be there even if -v is not toggled.

It can be something like

$ notation sign --signature-format cose --key $REMOTE_KEY_NAME ghcr.io/demo-user/python@sha256:xxxx
Warning:  Removal of outdated referrers index is not supported by the remote registry. Garbage collection may be required.

Discussed offline with @shizhMSFT, we found it might be better to keep the Warning inside notation CLI but with a more user-friendly wording, for example Warning: Removal of outdated referrers index is not supported by the remote registry. Garbage collection may be required. Therefore, reopened this PR 619.

[FeynmanZhou]

Reopen this issue since we need to refine the warning message. It requires another PR to improve the error message

[patrickzheng200]

Reopen this issue since we need to refine the warning message. It requires another PR to improve the error message

I will create another PR to refine the warning message as it only covers one of the possible scenarios at the moment.

[patrickzheng200]

Closing the issue as it's resolved by this PR: #595.