Help on Understanding a Dependency Misconfiguration

[emanuele-leopardi]

Hello everyone.
First of all, let me show you what was the initial state of the cluster.

This is a classical directory structure for repositories hosting flux

.
└── replica
    ├── flux-system
    │ ├── gotk-components.yaml
    │ ├── gotk-sync.yaml
    │ └── kustomization.yaml
    ├── infra
    │ ├── configurations
    │ │ ├── cluster-secret
    │ │ ├── datadog-agent
    │ │ ├── istio
    │ │ ├── karpenter
    │ │ ├── ks.yaml
    │ │ └── weaveworks-dashboard
    │ ├── controllers
    │ │ ├── alb
    │ │ ├── cert-manager
    │ │ ├── cluster-secret
    │ │ ├── datadog-operator
    │ │ ├── istio
    │ │ ├── karpenter
    │ │ ├── kiali
    │ │ ├── ks.yaml
    │ │ ├── locust
    │ │ └── metrics-server
    │ └── kustomization.yaml
    └── README.md

So nothing special here.
The kustomization file looks like this

replica/flux-system/gotk-sync.yaml

# This manifest was generated by flux. DO NOT EDIT.
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: master
  secretRef:
    name: flux-system
  url: https://github.com/KeylessTech/kl_kubernetes.git
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./clusters/staging/replica
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

As we are on EKS we run flux with a role to access KMS, again nothing special here.

replica/flux-system/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
patches:
  - patch: |
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: controller
        annotations:
          eks.amazonaws.com/role-arn: "arn:aws:iam::446813827414:role/flux-kms-role"
    target:
      kind: ServiceAccount
      name: "kustomize-controller"

We also have dependencies between controllers and configurations kustomization files, so we created the following configuration

replica/infra/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- configurations/ks.yaml
- controllers/ks.yaml

replica/infra/controllers/ks.yaml

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: controllers
  namespace: flux-system
spec:
  interval: 1m
  path: "./clusters/staging/replica/infra/controllers"
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  decryption:
    provider: sops

replica/infra/configurations/ks.yaml

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: configurations
  namespace: flux-system
spec:
  dependsOn:
    - name: controllers   <------------------------------------------- Notice the dependency here
  interval: 1m
  path: "./clusters/staging/replica/infra/configurations"
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  decryption:
    provider: sops

Here's the simplified directory structure

.
└── replica
    ├── flux-system
    │ └── kustomization.yaml
    ├── infra
    │ ├── configurations
    │ │ └── ks.yaml
    │ ├── controllers
    │ │ └── ks.yaml
    │ └── kustomization.yaml
    └── README.md

This all worked fine as expected.

Now...we wanted to add some applications so that the directory structure would become the following

.
└── replica
    ├── apps
    │ └── my-app
    │   └── foo.yaml
    ├── flux-system
    │ ├── gotk-components.yaml
    │ ├── gotk-sync.yaml
    │ └── kustomization.yaml
    ├── infra
    │ ├── configurations
    │ │ └── ks.yaml
    │ ├── controllers
    │ │ └── ks.yaml
    │ └── kustomization.yaml
    └── README.md

we also wanted to explicitly say that app kustomization has a dependency on controllers so we created the following files

/replica/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- apps/ks.yaml

replica/apps/ks.yaml

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  dependsOn:
    - name: controllers   <------------------------------------------------------
  interval: 1m
  path: "./clusters/staging/replica/apps"
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  decryption:
    provider: sops

.
└── replica
    │ └── kustomization.yaml <------------
    ├── apps
    │ │ └── ks.yaml <---------------
    │ └── my-app
    │   └── foo.yaml
    ├── flux-system
    │ ├── gotk-components.yaml
    │ ├── gotk-sync.yaml
    │ └── kustomization.yaml
    ├── infra
    │ ├── configurations
    │ │ └── ks.yaml
    │ ├── controllers
    │ │ └── ks.yaml
    │ └── kustomization.yaml
    └── README.md

Now, as soon as we merged this commit into our master branch ( tracked by source controller) flux started going mad, trying to reconcile every kustomization, disrupting the service, and even removing the flux-system namespace and of course everything that was deployed there, so I'm reasonably confident the directory structure is not correct.

How should I structure the directories to achieve the dependency schema I want between them?
Id est apps/ks.yaml and infra/configurations/ks.yaml depend on infra/controllers/ks.yaml