Standardize Helms Charts

[XciD]

• Extract secret to use the helm chart without already existing secrets
• Abstract storage to PV/PVC
• Some helm refactoring

cc: @n1t0

[severo]

To be able to fully test (and then deploy to prod), I would need to have access to the pvc. I currently get

$ kubectl get pvc
Error from server (Forbidden): persistentvolumeclaims is forbidden: User "AWSReservedSSO_EKS-HUB-Tensorboard_855674a9053d4044:sylvain.lesage-huggingface.co" cannot list resource "persistentvolumeclaims" in API group "" in the namespace "datasets-server"

both in the prod and ephemeral clusters.

Could you help me with that @huggingface/infra-team?

[severo]

Done, thx @co42 :

$ kubectl get pvc
NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-datasets-server-dev-mongodb-0   Bound    pvc-a2eb0016-4d7b-4125-9f72-52dd790c5cce   8Gi        RWO            ebs-gp2        113d
nfs-datasets-server-pvc                 Bound    nfs-datasets-server-pv                     100Gi      RWX            nfs            7d

[severo]

Still an issue:

$ k exec -it datasets-server-dev-worker-first-rows-76f94b7f8d-8z45t -- mkdir /assets/test
Defaulted container "datasets-server-worker-first-rows" out of: datasets-server-worker-first-rows, prepare-assets (init), prepare-cache (init), prepare-numba-cache (init)
mkdir: cannot create directory ‘/assets/test’: Read-only file system
command terminated with exit code 1

Edit: It's my fault, due to the code factorization! 7f88e66

[severo]

OK, tested on ephemeral and deployed in prod. It works as expected, the first-rows workers could write images in the new dataset I created to test: https://huggingface.co/datasets/severo/mnist/viewer/mnist/test