Avoid reading all the resources from the cluster

[2opremio]

During syncing we read all the resources from the cluster. This causes a spike in memory consumption for clusters with a lot of resources (see #2633 ).

We read all the resources at once to check for ignore annotations:

flux/pkg/cluster/kubernetes/sync.go

Lines 55 to 57 in dc8c1b9

	// NB we get all resources, since we care about leaving unsynced,
	// _ignored_ resources alone.
	clusterResources, err := c.getAllowedResourcesBySelector("")

The ideal solution would be to query resources by annotation, but Kubernetes doesn't support this.

To improve this, we could either or:

1. Get only the resources whose annotations we explicitly need to check, on demand. This may be tricky, since we don't currently keep track of apiVersions in the resource IDs.
2. (1) may result in an increase of cluster queries (since we may be syncing more individual resources than api groups exist in the cluster). We could create a smart query by api group, but only including relevant api groups (i.e. only groups including one of the resources we want to sync).
3. To save memory, we could only keep the ignore annotation of resources. When querying by api-group, this won't help that much without quite a bit of effort (to make sure that the k8s client only deserializes the annotations), otherwise Flux will still need enough memory to hold all the resources in an api group.
4. We could only care about ignoring resources previously created by Flux (by reusing the selector query used by garbage collection). This would allow us to considerably reduce the amount of resources deserialized. As a downside, a user won't be able indicate that a manually created cluster resource should be ignored by Flux when given the same resource in git.