Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add native Azure Blob support to BucketController #513

Closed
wants to merge 56 commits into from
Closed
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
04219b8
Add native support to Azure Blob
laozc Dec 3, 2021
774cf9a
Remove calling deprecated MSI function
laozc Dec 3, 2021
e89ef08
Update CRD
laozc Dec 4, 2021
8e86b57
Introduce more explicit Condition types
hiddeco Jul 30, 2021
c50a170
Implement new runtime interfaces, prepare testenv
hiddeco Jul 30, 2021
65dd61e
Introduce `artifactSet` to replace `hasArtifactUpdated`
hiddeco Jul 30, 2021
c278501
source: Add `GetRequeueAfter`
darkowlzz Nov 24, 2021
2d88469
Use new events and metrics helpers in main.go
darkowlzz Nov 24, 2021
e7f0406
Move Artifact conditions to conditions
darkowlzz Nov 24, 2021
a722ec7
Add gomega matcher for artifact
darkowlzz Nov 25, 2021
0ae7758
Fixes PR comments
laozc Dec 7, 2021
f1302d8
bucket: Replace GetInterval() with GetRequeueAfter()
darkowlzz Nov 24, 2021
795b83d
Rewrite `BucketReconciler` to new standards
hiddeco Jul 31, 2021
f82a906
Consolidate condition types into `FetchFailed`
hiddeco Aug 9, 2021
6f34ea8
BucketReconciler: Add reconcileArtifact tests
darkowlzz Aug 9, 2021
317cd44
Add more reconcileMinioSource test cases
darkowlzz Nov 27, 2021
4814abc
Add bucket controller tests for reconcileGCPSource
darkowlzz Nov 29, 2021
625f850
bucket: Ignore patch error not found on delete
darkowlzz Dec 8, 2021
6931fed
Merge remote-tracking branch 'origin/reconcilers-dev-bucket' into azblob
laozc Dec 11, 2021
4dd7799
Update go.sum
laozc Dec 11, 2021
e9d5353
Introduce more explicit Condition types
hiddeco Jul 30, 2021
10a8cc4
Implement new runtime interfaces, prepare testenv
hiddeco Jul 30, 2021
c1668ef
Introduce `artifactSet` to replace `hasArtifactUpdated`
hiddeco Jul 30, 2021
e5806c3
source: Add `GetRequeueAfter`
darkowlzz Nov 24, 2021
adb131c
Use new events and metrics helpers in main.go
darkowlzz Nov 24, 2021
2b7a308
Move Artifact conditions to conditions
darkowlzz Nov 24, 2021
e82a472
Add gomega matcher for artifact
darkowlzz Nov 25, 2021
b65e9bf
Add internal packages error and result
darkowlzz Dec 19, 2021
a956216
bucket: Replace GetInterval() with GetRequeueAfter()
darkowlzz Nov 24, 2021
27455c4
Rewrite `BucketReconciler` to new standards
hiddeco Jul 31, 2021
009acb6
Consolidate condition types into `FetchFailed`
hiddeco Aug 9, 2021
421f50d
BucketReconciler: Add reconcileArtifact tests
darkowlzz Aug 9, 2021
066f74a
Add more reconcileMinioSource test cases
darkowlzz Nov 27, 2021
18fd284
Add bucket controller tests for reconcileGCPSource
darkowlzz Nov 29, 2021
bd0ee6e
bucket: Ignore patch error not found on delete
darkowlzz Dec 8, 2021
878856c
bucket: Add more reconciler design improvements
darkowlzz Dec 20, 2021
f912cbf
Merge remote-tracking branch 'origin/reconcilers-dev-bucket' into azblob
laozc Jan 6, 2022
c2793ef
Introduce more explicit Condition types
hiddeco Jul 30, 2021
d7e0faa
Implement new runtime interfaces, prepare testenv
hiddeco Jul 30, 2021
7d752b7
Introduce `artifactSet` to replace `hasArtifactUpdated`
hiddeco Jul 30, 2021
d9ad873
source: Add `GetRequeueAfter`
darkowlzz Nov 24, 2021
1c25c8c
Use new events and metrics helpers in main.go
darkowlzz Nov 24, 2021
f927c76
Move Artifact conditions to conditions
darkowlzz Nov 24, 2021
34b5dcf
Add gomega matcher for artifact
darkowlzz Nov 25, 2021
45484ef
api: Embed runtime.Object in Source interface
darkowlzz Jan 19, 2022
c3e2087
Add internal packages error and reconcile
darkowlzz Dec 19, 2021
8908d80
Use runtime-events-fixes branch temporarily
darkowlzz Jan 16, 2022
991692f
bucket: Replace GetInterval() with GetRequeueAfter()
darkowlzz Nov 24, 2021
34b27a0
Rewrite `BucketReconciler` to new standards
hiddeco Jul 31, 2021
3105ba9
Consolidate condition types into `FetchFailed`
hiddeco Aug 9, 2021
cf7dca0
BucketReconciler: Add reconcileArtifact tests
darkowlzz Aug 9, 2021
af946e7
Add more reconcileMinioSource test cases
darkowlzz Nov 27, 2021
8dd51ab
Add bucket controller tests for reconcileGCPSource
darkowlzz Nov 29, 2021
4a1818d
bucket: Ignore patch error not found on delete
darkowlzz Dec 8, 2021
a1e5067
bucket: Add more reconciler design improvements
darkowlzz Dec 20, 2021
f73df89
Merge remote-tracking branch 'origin/reconcilers-dev-bucket' into azblob
laozc Jan 21, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion api/go.mod
Original file line number Diff line number Diff line change
@@ -4,25 +4,46 @@ go 1.17

require (
github.com/fluxcd/pkg/apis/acl v0.0.3
github.com/fluxcd/pkg/apis/meta v0.10.2
github.com/fluxcd/pkg/apis/meta v0.11.0-rc.3
// TODO(hidde): introduction of the runtime package is temporary, and the dependency should be removed as soon as
// all APIs have been updated to the runtime standards (more specifically; have dropped their condition modifying
// functions).
github.com/fluxcd/pkg/runtime v0.13.0-rc.6
k8s.io/apimachinery v0.23.1
sigs.k8s.io/controller-runtime v0.11.0
)

require (
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/evanphx/json-patch v4.12.0+incompatible // indirect
github.com/go-logr/logr v1.2.2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/google/go-cmp v0.5.6 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/googleapis/gnostic v0.5.5 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/onsi/gomega v1.17.0 // indirect
github.com/pkg/errors v0.9.1 // indirect
golang.org/x/net v0.0.0-20211215060638-4ddde0e984e9 // indirect
golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f // indirect
golang.org/x/sys v0.0.0-20211029165221-6e7872819dc8 // indirect
golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b // indirect
golang.org/x/text v0.3.7 // indirect
golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/protobuf v1.27.1 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
k8s.io/api v0.23.0 // indirect
k8s.io/client-go v0.23.0 // indirect
k8s.io/klog/v2 v2.30.0 // indirect
k8s.io/kube-openapi v0.0.0-20211115234752-e816edb12b65 // indirect
k8s.io/utils v0.0.0-20211208161948-7d6a63dca704 // indirect
sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2 // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.2.0 // indirect
sigs.k8s.io/yaml v1.3.0 // indirect
)
34 changes: 32 additions & 2 deletions api/go.sum

Large diffs are not rendered by default.

73 changes: 26 additions & 47 deletions api/v1beta1/bucket_types.go
Original file line number Diff line number Diff line change
@@ -17,7 +17,8 @@ limitations under the License.
package v1beta1

import (
apimeta "k8s.io/apimachinery/pkg/api/meta"
"time"

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

"github.com/fluxcd/pkg/apis/acl"
@@ -29,10 +30,17 @@ const (
BucketKind = "Bucket"
)

const (
GenericBucketProvider string = "generic"
AmazonBucketProvider string = "aws"
GoogleBucketProvider string = "gcp"
AzureBlobProvider string = "azure"
)

// BucketSpec defines the desired state of an S3 compatible bucket
type BucketSpec struct {
// The S3 compatible storage provider name, default ('generic').
// +kubebuilder:validation:Enum=generic;aws;gcp
// +kubebuilder:validation:Enum=generic;aws;gcp;azure
// +kubebuilder:default:=generic
// +optional
Provider string `json:"provider,omitempty"`
@@ -82,12 +90,6 @@ type BucketSpec struct {
AccessFrom *acl.AccessFrom `json:"accessFrom,omitempty"`
}

const (
GenericBucketProvider string = "generic"
AmazonBucketProvider string = "aws"
GoogleBucketProvider string = "gcp"
)

// BucketStatus defines the observed state of a bucket
type BucketStatus struct {
// ObservedGeneration is the last observed generation.
@@ -119,61 +121,38 @@ const (
BucketOperationFailedReason string = "BucketOperationFailed"
)

// BucketProgressing resets the conditions of the Bucket to metav1.Condition of
// type meta.ReadyCondition with status 'Unknown' and meta.ProgressingReason
// reason and message. It returns the modified Bucket.
func BucketProgressing(bucket Bucket) Bucket {
bucket.Status.ObservedGeneration = bucket.Generation
bucket.Status.URL = ""
bucket.Status.Conditions = []metav1.Condition{}
meta.SetResourceCondition(&bucket, meta.ReadyCondition, metav1.ConditionUnknown, meta.ProgressingReason, "reconciliation in progress")
return bucket
// GetConditions returns the status conditions of the object.
func (in Bucket) GetConditions() []metav1.Condition {
return in.Status.Conditions
}

// BucketReady sets the given Artifact and URL on the Bucket and sets the
// meta.ReadyCondition to 'True', with the given reason and message. It returns
// the modified Bucket.
func BucketReady(bucket Bucket, artifact Artifact, url, reason, message string) Bucket {
bucket.Status.Artifact = &artifact
bucket.Status.URL = url
meta.SetResourceCondition(&bucket, meta.ReadyCondition, metav1.ConditionTrue, reason, message)
return bucket
// SetConditions sets the status conditions on the object.
func (in *Bucket) SetConditions(conditions []metav1.Condition) {
in.Status.Conditions = conditions
}

// BucketNotReady sets the meta.ReadyCondition on the Bucket to 'False', with
// the given reason and message. It returns the modified Bucket.
func BucketNotReady(bucket Bucket, reason, message string) Bucket {
meta.SetResourceCondition(&bucket, meta.ReadyCondition, metav1.ConditionFalse, reason, message)
return bucket
// GetRequeueAfter returns the duration after which the source must be reconciled again.
func (in Bucket) GetRequeueAfter() time.Duration {
return in.Spec.Interval.Duration
}

// BucketReadyMessage returns the message of the metav1.Condition of type
// meta.ReadyCondition with status 'True' if present, or an empty string.
func BucketReadyMessage(bucket Bucket) string {
if c := apimeta.FindStatusCondition(bucket.Status.Conditions, meta.ReadyCondition); c != nil {
if c.Status == metav1.ConditionTrue {
return c.Message
}
}
return ""
// GetInterval returns the interval at which the source is reconciled.
// Deprecated: use GetRequeueAfter instead.
func (in Bucket) GetInterval() metav1.Duration {
return in.Spec.Interval
}

// GetArtifact returns the latest artifact from the source if present in the
// status sub-resource.
// GetArtifact returns the latest artifact from the source if present in the status sub-resource.
func (in *Bucket) GetArtifact() *Artifact {
return in.Status.Artifact
}

// GetStatusConditions returns a pointer to the Status.Conditions slice
// GetStatusConditions returns a pointer to the Status.Conditions slice.
// Deprecated: use GetConditions instead.
func (in *Bucket) GetStatusConditions() *[]metav1.Condition {
return &in.Status.Conditions
}

// GetInterval returns the interval at which the source is updated.
func (in *Bucket) GetInterval() metav1.Duration {
return in.Spec.Interval
}

// +genclient
// +genclient:Namespaced
// +kubebuilder:object:root=true
16 changes: 16 additions & 0 deletions api/v1beta1/condition_types.go
Original file line number Diff line number Diff line change
@@ -18,6 +18,22 @@ package v1beta1

const SourceFinalizer = "finalizers.fluxcd.io"

const (
// ArtifactUnavailableCondition indicates there is no Artifact available for the Source.
// This is a "negative polarity" or "abnormal-true" type, and is only present on the resource if it is True.
ArtifactUnavailableCondition string = "ArtifactUnavailable"

// ArtifactOutdatedCondition indicates the current Artifact of the Source is outdated.
// This is a "negative polarity" or "abnormal-true" type, and is only present on the resource if it is True.
ArtifactOutdatedCondition string = "ArtifactOutdated"

// FetchFailedCondition indicates a transient or persistent fetch failure of an upstream Source.
// If True, observations on the upstream Source revision may be impossible, and the Artifact available for the
// Source may be outdated.
// This is a "negative polarity" or "abnormal-true" type, and is only present on the resource if it is True.
FetchFailedCondition string = "FetchFailed"
)

const (
// URLInvalidReason represents the fact that a given source has an invalid URL.
URLInvalidReason string = "URLInvalid"
90 changes: 60 additions & 30 deletions api/v1beta1/gitrepository_types.go
Original file line number Diff line number Diff line change
@@ -17,11 +17,14 @@ limitations under the License.
package v1beta1

import (
"time"

apimeta "k8s.io/apimachinery/pkg/api/meta"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

"github.com/fluxcd/pkg/apis/acl"
"github.com/fluxcd/pkg/apis/meta"
"github.com/fluxcd/pkg/runtime/conditions"
)

const (
@@ -34,6 +37,22 @@ const (
LibGit2Implementation = "libgit2"
)

const (
// CheckoutFailedCondition indicates a transient or persistent checkout failure. If True, observations on the
// upstream Source revision are not possible, and the Artifact available for the Source may be outdated.
// This is a "negative polarity" or "abnormal-true" type, and is only present on the resource if it is True.
CheckoutFailedCondition string = "CheckoutFailed"

// SourceVerifiedCondition indicates the integrity of the Source has been verified. If True, the integrity check
// succeeded. If False, it failed. The Condition is only present on the resource if the integrity has been verified.
SourceVerifiedCondition string = "SourceVerified"

// IncludeUnavailableCondition indicates one of the includes is not available. For example, because it does not
// exist, or does not have an Artifact.
// This is a "negative polarity" or "abnormal-true" type, and is only present on the resource if it is True.
IncludeUnavailableCondition string = "IncludeUnavailable"
)

// GitRepositorySpec defines the desired state of a Git repository.
type GitRepositorySpec struct {
// The repository URL, can be a HTTP/S or SSH address.
@@ -42,10 +61,8 @@ type GitRepositorySpec struct {
URL string `json:"url"`

// The secret name containing the Git credentials.
// For HTTPS repositories the secret must contain username and password
// fields.
// For SSH repositories the secret must contain identity, identity.pub and
// known_hosts fields.
// For HTTPS repositories the secret must contain username and password fields.
// For SSH repositories the secret must contain 'identity', 'identity.pub' and 'known_hosts' fields.
// +optional
SecretRef *meta.LocalObjectReference `json:"secretRef,omitempty"`

@@ -63,16 +80,16 @@ type GitRepositorySpec struct {
// +optional
Reference *GitRepositoryRef `json:"ref,omitempty"`

// Verify OpenPGP signature for the Git commit HEAD points to.
// Verification defines the configuration to verify the OpenPGP signature for the Git commit HEAD points to.
// +optional
Verification *GitRepositoryVerification `json:"verify,omitempty"`

// Ignore overrides the set of excluded patterns in the .sourceignore format
// (which is the same as .gitignore). If not provided, a default will be used,
// consult the documentation for your version to find out what those are.
// Ignore overrides the set of excluded patterns in the .sourceignore format (which is the same as .gitignore).
// If not provided, a default will be used, consult the documentation for your version to find out what those are.
// +optional
Ignore *string `json:"ignore,omitempty"`

// Suspend tells the controller to suspend the reconciliation of this source.
// This flag tells the controller to suspend the reconciliation of this source.
// +optional
Suspend bool `json:"suspend,omitempty"`
@@ -84,13 +101,13 @@ type GitRepositorySpec struct {
// +optional
GitImplementation string `json:"gitImplementation,omitempty"`

// When enabled, after the clone is created, initializes all submodules within,
// using their default settings.
// When enabled, after the clone is created, initializes all submodules within, using their default settings.
// This option is available only when using the 'go-git' GitImplementation.
// +optional
RecurseSubmodules bool `json:"recurseSubmodules,omitempty"`

// Extra git repositories to map into the repository
// Include defines a list of GitRepository resources which artifacts should be included in the artifact produced for
// this resource.
Include []GitRepositoryInclude `json:"include,omitempty"`

// AccessFrom defines an Access Control List for allowing cross-namespace references to this object.
@@ -144,11 +161,11 @@ type GitRepositoryRef struct {

// GitRepositoryVerification defines the OpenPGP signature verification process.
type GitRepositoryVerification struct {
// Mode describes what git object should be verified, currently ('head').
// Mode describes what Git object should be verified, currently ('head').
// +kubebuilder:validation:Enum=head
Mode string `json:"mode"`

// The secret name containing the public keys of all trusted Git authors.
// SecretRef containing the public keys of all trusted Git authors.
SecretRef meta.LocalObjectReference `json:"secretRef,omitempty"`
}

@@ -162,8 +179,7 @@ type GitRepositoryStatus struct {
// +optional
Conditions []metav1.Condition `json:"conditions,omitempty"`

// URL is the download link for the artifact output of the last repository
// sync.
// URL is the download link for the artifact output of the last repository sync.
// +optional
URL string `json:"url,omitempty"`

@@ -179,12 +195,10 @@ type GitRepositoryStatus struct {
}

const (
// GitOperationSucceedReason represents the fact that the git clone, pull
// and checkout operations succeeded.
// GitOperationSucceedReason represents the fact that the git clone, pull and checkout operations succeeded.
GitOperationSucceedReason string = "GitOperationSucceed"

// GitOperationFailedReason represents the fact that the git clone, pull or
// checkout operations failed.
// GitOperationFailedReason represents the fact that the git clone, pull or checkout operations failed.
GitOperationFailedReason string = "GitOperationFailed"
)

@@ -196,7 +210,7 @@ func GitRepositoryProgressing(repository GitRepository) GitRepository {
repository.Status.ObservedGeneration = repository.Generation
repository.Status.URL = ""
repository.Status.Conditions = []metav1.Condition{}
meta.SetResourceCondition(&repository, meta.ReadyCondition, metav1.ConditionUnknown, meta.ProgressingReason, "reconciliation in progress")
conditions.MarkUnknown(&repository, meta.ReadyCondition, meta.ProgressingReason, "reconciliation in progress")
return repository
}

@@ -207,15 +221,15 @@ func GitRepositoryReady(repository GitRepository, artifact Artifact, includedArt
repository.Status.Artifact = &artifact
repository.Status.IncludedArtifacts = includedArtifacts
repository.Status.URL = url
meta.SetResourceCondition(&repository, meta.ReadyCondition, metav1.ConditionTrue, reason, message)
conditions.MarkTrue(&repository, meta.ReadyCondition, reason, message)
return repository
}

// GitRepositoryNotReady sets the meta.ReadyCondition on the given GitRepository
// to 'False', with the given reason and message. It returns the modified
// GitRepository.
func GitRepositoryNotReady(repository GitRepository, reason, message string) GitRepository {
meta.SetResourceCondition(&repository, meta.ReadyCondition, metav1.ConditionFalse, reason, message)
conditions.MarkFalse(&repository, meta.ReadyCondition, reason, message)
return repository
}

@@ -230,22 +244,38 @@ func GitRepositoryReadyMessage(repository GitRepository) string {
return ""
}

// GetArtifact returns the latest artifact from the source if present in the
// status sub-resource.
// GetConditions returns the status conditions of the object.
func (in GitRepository) GetConditions() []metav1.Condition {
return in.Status.Conditions
}

// SetConditions sets the status conditions on the object.
func (in *GitRepository) SetConditions(conditions []metav1.Condition) {
in.Status.Conditions = conditions
}

// GetRequeueAfter returns the duration after which the source must be reconciled again.
func (in GitRepository) GetRequeueAfter() time.Duration {
return in.Spec.Interval.Duration
}

// GetInterval returns the interval at which the source is reconciled.
// Deprecated: use GetRequeueAfter instead.
func (in GitRepository) GetInterval() metav1.Duration {
return in.Spec.Interval
}

// GetArtifact returns the latest artifact from the source if present in the status sub-resource.
func (in *GitRepository) GetArtifact() *Artifact {
return in.Status.Artifact
}

// GetStatusConditions returns a pointer to the Status.Conditions slice
// GetStatusConditions returns a pointer to the Status.Conditions slice.
// Deprecated: use GetConditions instead.
func (in *GitRepository) GetStatusConditions() *[]metav1.Condition {
return &in.Status.Conditions
}

// GetInterval returns the interval at which the source is updated.
func (in *GitRepository) GetInterval() metav1.Duration {
return in.Spec.Interval
}

// +genclient
// +genclient:Namespaced
// +kubebuilder:object:root=true
Loading