New kubernetes backend #21796

grihabor · 2024-12-22T23:24:32Z

While helm backend already exists and works fine for kubernetes, creating a helm chart might be an overkill for a small thing like a single configmap or a secret. New kubernetes backend can deploy single object easily given a yaml file.

See docs for more details on usage.

I'm also planning to contribute our integration with python_format_string target (I will open a pr) that can handle simple python templating in text files.

> fd _category_ docs/docs/ --max-depth 2 --exec rg position '{}' --with-filename --line-number | sort -k 3,3 -n docs/docs/introduction/_category_.json:3: "position": 1 docs/docs/getting-started/_category_.json:3: "position": 2 docs/docs/using-pants/_category_.json:3: "position": 3 docs/docs/python/_category_.json:3: "position": 4 docs/docs/go/_category_.json:3: "position": 5 docs/docs/jvm/_category_.json:3: "position": 6 docs/docs/shell/_category_.json:3: "position": 7 docs/docs/docker/_category_.json:3: "position": 8 docs/docs/kubernetes/_category_.json:3: "position": 9 docs/docs/helm/_category_.json:3: "position": 10 docs/docs/terraform/_category_.json:3: "position": 11 docs/docs/sql/_category_.json:3: "position": 12 docs/docs/ad-hoc-tools/_category_.json:3: "position": 13 docs/docs/javascript/_category_.json:3: "position": 13 docs/docs/writing-plugins/_category_.json:3: "position": 14 docs/docs/releases/_category_.json:3: "position": 15 docs/docs/contributions/_category_.json:3: "position": 16 docs/docs/tutorials/_category_.json:3: "position": 17

cburroughs · 2025-01-07T20:08:56Z

cc @tgolsson who I know has done some k8s related work at https://github.com/tgolsson/pants-backends/tree/main/pants-plugins/k8s

Broad topic not intended to detail this whole PR: Something we have struggled with internally is that we would like to be able to generate the final reified k8s yaml for inspection or alternative deployments. (I have a PoC plugin that does this for helm) This is along the lines of 'package' but not not quite what helm means by 'package', and I'm unsure what to call it. It would be nice if all the Pant k8s generators could eventually both "spit out the yaml" and "deploy".

grihabor · 2025-01-08T11:16:04Z

Something we have struggled with internally is that we would like to be able to generate the final reified k8s yaml for inspection or alternative deployments.

Haha, we have exactly this problem, I've called the goal render and implemented it for helm_deployment and k8s_bundle

lilatomic

Looks good!
My only real sticking point is about the semantics of the requirement for context

lilatomic · 2025-01-08T14:03:22Z

src/python/pants/backend/k8s/goals/tailor.py

+    putative_targets = []
+
+    if k8s.tailor_source_targets:
+        all_k8s_files_globs = req.path_globs("*.yaml")


include "*.yml" too? or maybe make this a customisable option? Not necessary for a first pass

Personally, I want consistent .yaml or .yml in the whole repo, so probably it should be customisable, yes

lilatomic · 2025-01-08T14:17:02Z

src/python/pants/backend/k8s/goals/tailor.py

+    if k8s.tailor_source_targets:
+        all_k8s_files_globs = req.path_globs("*.yaml")
+        all_k8s_files = await Get(Paths, PathGlobs, all_k8s_files_globs)
+        unowned_k8s_files = set(all_k8s_files.files) - set(all_owned_sources)


I think that globbing every yaml file will be too broad, but I don't know that there's a standard way of detecting if a manifest is a k8s manifest (kubectl apply --validate=true --dry-run=client seems to need a server). Worst case people can turn off tailoring.

I'm thinking 2 most common use cases here:

Developer's repo with code will probably use some kind of templating to generate k8s_sources (I'm planning to open couple more prs for that), so tailor for yaml won't be that useful. In this case I expect devs to disable tailoring.

DevOps's repo with mostly k8s yaml files. In this case I expect tailor to work good

lilatomic · 2025-01-08T14:36:50Z

build-support/bin/external_tool_versions.py

+
+
+@dataclass(frozen=True)
+class VersionHash:


You could use pants.core.util_rules.external_tool.ExternalToolVersion

It's not quite the same thing, because platform here is not a predefined platform that pants uses, but a platform that needs to be mapped using url_platform_mapping. But I can make it work like this 6efda67

lilatomic · 2025-01-08T14:49:36Z

build-support/bin/external_tool_versions.py

+            )
+
+    backward_platform_mapping = {v: k for k, v in platform_mapping.items()}
+    for result in results:


you could have these output in semver order (instead of lexical order) by using from packaging.version import Version, by collecting the versions and then sorting with something like sorted(versions, key=lambda e: Version(e.version))

Sure 9d84c9e

build-support/bin/external_tool_versions.py

lilatomic · 2025-01-08T15:19:47Z

src/python/pants/backend/k8s/goals/deploy.py

+    kubectl_tool = await Get(
+        DownloadedExternalTool, ExternalToolRequest, kubectl.get_request(platform)
+    )
+    digest = await Get(Digest, MergeDigests([kubectl_tool.digest, request.input_digest]))


you could use Process.immutable_input_digest for the tool. Here's how it's done in the Helm backend

pants/src/python/pants/backend/helm/util_rules/tool.py

Lines 454 to 457 in 1d1e93e

immutable_input_digests = {

**request.extra_immutable_input_digests,

**helm_binary.immutable_input_digests,

}

Sure 72722dd

lilatomic · 2025-01-08T15:22:09Z

src/python/pants/backend/k8s/goals/deploy.py

+) -> DeployProcess:
+    context = field_set.context.value
+    if context is None:
+        raise ValueError(f"Missing `{K8sBundleContextField.alias}` field")


we could make this more helpful by including the FS's address with field_set.address.spec

Good suggestion, thanks ed0be1f

lilatomic · 2025-01-08T15:27:06Z

src/python/pants/backend/k8s/goals/deploy.py

+    platform: Platform,
+) -> DeployProcess:
+    context = field_set.context.value
+    if context is None:


I think I've missed something in the logic here. Why do we force the field_set.context to have a value even when kubectl.pass_context is False?

The idea is that setting context on a target is required. This is because we had recurring issues because somebody forgot to set the correct context on the target and it got deployed to a different context by accident. However, we've got CI agents running in the cluster itself, so they can only deploy k8s objects to the specific cluster specified with KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT, that's why we need to disable --context argument in pants.ci.toml.

I understand this is kinda custom setup, so people might need some other behavior. However, in general, I think context requirement is a good thing that can prevent misdeployments. I can probably move this context validation to a small linter and check context field there, wdyt?

cburroughs · 2025-01-09T20:43:00Z

Haha, we have exactly this problem, I've called the goal render and implemented it for helm_deployment and k8s_bundle

I'd be excited to see that!

Co-authored-by: Daniel Goldman <[email protected]>

grihabor

@lilatomic Thank you for review! Ready for another round

grihabor · 2025-01-11T14:20:56Z

src/python/pants/backend/k8s/goals/tailor.py

+    if k8s.tailor_source_targets:
+        all_k8s_files_globs = req.path_globs("*.yaml")
+        all_k8s_files = await Get(Paths, PathGlobs, all_k8s_files_globs)
+        unowned_k8s_files = set(all_k8s_files.files) - set(all_owned_sources)


I'm thinking 2 most common use cases here:

Developer's repo with code will probably use some kind of templating to generate k8s_sources (I'm planning to open couple more prs for that), so tailor for yaml won't be that useful. In this case I expect devs to disable tailoring.

DevOps's repo with mostly k8s yaml files. In this case I expect tailor to work good

grihabor · 2025-01-11T14:22:18Z

src/python/pants/backend/k8s/goals/tailor.py

+    putative_targets = []
+
+    if k8s.tailor_source_targets:
+        all_k8s_files_globs = req.path_globs("*.yaml")


Personally, I want consistent .yaml or .yml in the whole repo, so probably it should be customisable, yes

grihabor · 2025-01-11T14:28:41Z

build-support/bin/external_tool_versions.py

+
+
+@dataclass(frozen=True)
+class VersionHash:


It's not quite the same thing, because platform here is not a predefined platform that pants uses, but a platform that needs to be mapped using url_platform_mapping. But I can make it work like this 6efda67

grihabor · 2025-01-11T21:04:17Z

build-support/bin/external_tool_versions.py

+            )
+
+    backward_platform_mapping = {v: k for k, v in platform_mapping.items()}
+    for result in results:


Sure 9d84c9e

grihabor · 2025-01-11T21:06:58Z

src/python/pants/backend/k8s/goals/deploy.py

+) -> DeployProcess:
+    context = field_set.context.value
+    if context is None:
+        raise ValueError(f"Missing `{K8sBundleContextField.alias}` field")


Good suggestion, thanks ed0be1f

grihabor · 2025-01-11T21:14:27Z

src/python/pants/backend/k8s/goals/deploy.py

+    platform: Platform,
+) -> DeployProcess:
+    context = field_set.context.value
+    if context is None:


The idea is that setting context on a target is required. This is because we had recurring issues because somebody forgot to set the correct context on the target and it got deployed to a different context by accident. However, we've got CI agents running in the cluster itself, so they can only deploy k8s objects to the specific cluster specified with KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT, that's why we need to disable --context argument in pants.ci.toml.

I understand this is kinda custom setup, so people might need some other behavior. However, in general, I think context requirement is a good thing that can prevent misdeployments. I can probably move this context validation to a small linter and check context field there, wdyt?

grihabor · 2025-01-11T21:27:18Z

src/python/pants/backend/k8s/goals/deploy.py

+    kubectl_tool = await Get(
+        DownloadedExternalTool, ExternalToolRequest, kubectl.get_request(platform)
+    )
+    digest = await Get(Digest, MergeDigests([kubectl_tool.digest, request.input_digest]))


Sure 72722dd

grihabor added 15 commits December 21, 2024 13:48

Copy internal plugin as is

a594341

Fix linters

c13360d

Fix env var

e9cbcbf

Add a test

420d6b9

Fixing the test...

9409599

Download kubectl via TemplatedExternalTool

7a0d00d

Fix the tests

8ff08ab

Implement tailor goal

596e947

Add a script to fetch kubectl versions

fb2c8e5

Add more known versions

db2d216

Add more docs

f5e11d6

Add notes

4e5ba6b

pants fix

f5df270

Fix mypy

4b8a446

lilatomic reviewed Jan 8, 2025

View reviewed changes

grihabor and others added 6 commits January 11, 2025 15:38

Reuse ExternalToolVersion

6efda67

Update build-support/bin/external_tool_versions.py

6bcb4e4

Co-authored-by: Daniel Goldman <[email protected]>

Sort versions, show progress via tqdm

9d84c9e

Better error message

ed0be1f

Use immutable_input_digests for kubectl tool

72722dd

Add types-tqdm

a50b099

grihabor commented Jan 11, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New kubernetes backend #21796

New kubernetes backend #21796

grihabor commented Dec 22, 2024 •

edited

Loading

cburroughs commented Jan 7, 2025

grihabor commented Jan 8, 2025

lilatomic left a comment

lilatomic Jan 8, 2025

grihabor Jan 11, 2025

lilatomic Jan 8, 2025

grihabor Jan 11, 2025

lilatomic Jan 8, 2025

grihabor Jan 11, 2025

lilatomic Jan 8, 2025

grihabor Jan 11, 2025

lilatomic Jan 8, 2025

grihabor Jan 11, 2025

lilatomic Jan 8, 2025

grihabor Jan 11, 2025

lilatomic Jan 8, 2025

grihabor Jan 11, 2025

cburroughs commented Jan 9, 2025

grihabor left a comment

grihabor Jan 11, 2025

grihabor Jan 11, 2025

grihabor Jan 11, 2025

grihabor Jan 11, 2025

grihabor Jan 11, 2025

grihabor Jan 11, 2025

grihabor Jan 11, 2025

	immutable_input_digests = {
	**request.extra_immutable_input_digests,
	**helm_binary.immutable_input_digests,
	}



		@dataclass(frozen=True)
		class VersionHash:



		@dataclass(frozen=True)
		class VersionHash:

New kubernetes backend #21796

Are you sure you want to change the base?

New kubernetes backend #21796

Conversation

grihabor commented Dec 22, 2024 • edited Loading

cburroughs commented Jan 7, 2025

grihabor commented Jan 8, 2025

lilatomic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cburroughs commented Jan 9, 2025

grihabor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grihabor commented Dec 22, 2024 •

edited

Loading