🌱 Configure ENVTEST Binaries for IDE Debugging #1454

camilamacedo86 · 2024-11-13T12:43:38Z

This change introduces downloading the specific binaries required to run ENVTEST-based tests into the project/bin directory. This setup makes configuring tests for debugging directly in an IDE easier.

Motivation

Ease of IDE Debugging:
- By setting the BinaryAssetsDirectory dynamically, contributors can debug tests in their preferred IDEs (e.g., GoLand, VSCode) without requiring additional environment configuration such as setting KUBEBUILDER_ASSETS.

Project-Specific Binaries:

Installing binaries inside the project avoids conflicts with other projects or system-wide installations, particularly useful for contributors with restricted environments (e.g., corporate laptops).

The binaries installed by envtest-setup are no longer placed in the HOME/global directory. See the default behavior here:
https://github.com/kubernetes-sigs/controller-runtime/blob/e3347b5405cdd0da5bff527af3d406117938ba6b/tools/setup-envtest/README.md?plain=1#L57-L70

This change ensures binaries are placed locally, avoiding the conflicts and clutter caused by global installations.

I’m referring to the binaries required for ENVTEST (not those managed by BINGO).

Before this PR those are placed for example in a Mac OS at: /Users/camilam/Library/Application Support/io.kubebuilder.envtest/k8s/1.31.0-darwin-arm64

$ pwd
/Users/camilam/Library/Application Support/io.kubebuilder.envtest/k8s/1.31.0-darwin-arm64

ls -la
total 337856
dr-xr-xr-x  5 camilam  staff       160 19 Nov 09:20 .
drwxr-xr-x  3 camilam  staff        96 19 Nov 09:20 ..
-r-xr-xr-x  1 camilam  staff  26058562 19 Nov 09:20 etcd
-r-xr-xr-x  1 camilam  staff  90359954 19 Nov 09:20 kube-apiserver
-r-xr-xr-x  1 camilam  staff  56560754 19 Nov 09:20 kubectl

Simplified Contributor Onboarding:
- New contributors often face a learning curve to:
  - Understand ENVTEST and its dependency on binaries.
  - Configure KUBEBUILDER_ASSETS manually.
  - Set up their IDE for debugging tests.
- This setup minimizes friction and ensures tests can be run consistently across different environments.

While setting KUBEBUILDER_ASSETS is sufficient for experienced contributors, this change makes it easier for newcomers and supports debugging directly in IDEs without additional configuration.

netlify · 2024-11-13T12:43:53Z

✅ Deploy Preview for olmv1 ready!

Name	Link
🔨 Latest commit	`450d120`
🔍 Latest deploy log	https://app.netlify.com/sites/olmv1/deploys/6761547aaf2b600008a23e39
😎 Deploy Preview	https://deploy-preview-1454--olmv1.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Makefile

codecov · 2024-11-17T12:01:48Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.68%. Comparing base (7ad65d6) to head (450d120).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1454      +/-   ##
==========================================
- Coverage   74.87%   74.68%   -0.19%     
==========================================
  Files          42       42              
  Lines        3271     3271              
==========================================
- Hits         2449     2443       -6     
- Misses        648      652       +4     
- Partials      174      176       +2

Flag	Coverage Δ
e2e	`52.15% <ø> (-0.19%)`	⬇️
unit	`57.99% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

camilamacedo86 · 2024-11-17T12:07:31Z

Hi @m1kola,

Thank you for catching that—really good observation! 🙌

Your comment highlights an additional motivation for this change: addressing scenarios like those outlined in kubernetes-sigs/controller-runtime#2744 and kubernetes-sigs/controller-runtime#2646.

The required changes have been addressed, and we no longer use bingo here as intended. Please feel free to review this one.

camilamacedo86 · 2024-11-19T00:52:12Z

Hi @joelanford, @m1kola, @perdasilva, @bentito,

Apologies for the confusion earlier!

This PR is NOT about Bingo. The main goal is to make debugging ENVTEST-based tests in the IDE simpler and more straightforward.

I suspect that some team members might think it’s not possible to debug with Ginkgo because the tests in question are actually ENVTEST-based. This change addresses that scenario and clarifies the process.

I’ve addressed all comments, closed all as done or outdated to avoid confusion, reverted the BINGO changes, and kept the suggestions.

@m1kola, could you please revert your block?

Thanks in advance! 😊

camilamacedo86 · 2024-11-19T01:14:56Z

internal/controllers/suite_test.go

+	//
+	// To ensure the binaries are in the expected path without manual configuration, run:
+	// `make setup-envtest`
+	if getFirstFoundEnvTestBinaryDir() != "" {


Hi @perdasilva,

You added this comment, and I deviated—sorry about that!

Would it be cleaner if we just used an environment variable like ENV_TEST_BINARY_DIR or something like that?
Then, it doesn't really matter where the binaries are placed.

We already have an environment variable (KUBEBUILDER_ASSETS) that can be used, but it’s not very convenient and involves a steep learning curve:

Know that ENVTEST requires binaries.

Then, by default, ENVTEST is expected to find the bins in local/....

We can override this using the KUBEBUILDER_ASSETS environment variable.

Note that those bins never was in the $GOPATH/bin because they are placed via the setup-invest in the global place (see here) which indeed cause issues for those which have restrictive envs.

Then, there’s the added challenge of knowing how to configure the tests properly in an IDE.

This PR simplifies the process so everything “just works” without extra setup or deep knowledge.
Making tests and debugging smoother and hassle-free.

I’ve also updated the code comments to clarify things further.
Hope this clears everything up! 😊

Might also be nice to update the developer docs: https://github.com/operator-framework/operator-controller/blob/main/docs/contribute/developer.md

Both are done 👍
Thank you 🥇

camilamacedo86 · 2024-11-19T01:23:13Z

Makefile

 test-unit: $(SETUP_ENVTEST) #HELP Run the unit tests
 	rm -rf $(COVERAGE_UNIT_DIR) && mkdir -p $(COVERAGE_UNIT_DIR)
-	eval $$($(SETUP_ENVTEST) use -p env $(ENVTEST_VERSION) $(SETUP_ENVTEST_BIN_DIR_OVERRIDE)) && \
+	KUBEBUILDER_ASSETS="$(shell $(SETUP_ENVTEST) use -p path $(ENVTEST_VERSION) $(SETUP_ENVTEST_BIN_DIR_OVERRIDE))" \


Hi @m1kola,

Just to clarify, the binaries installed by envtest-setup are no longer placed in the HOME/global directory. See the default behavior here:
https://github.com/kubernetes-sigs/controller-runtime/blob/e3347b5405cdd0da5bff527af3d406117938ba6b/tools/setup-envtest/README.md?plain=1#L57-L70

This change ensures binaries are placed locally, avoiding the conflicts and clutter caused by global installations.

To clarify, I’m referring to the binaries required for ENVTEST (not those managed by BINGO).

Hope this clears things up! 😊

@perdasilva ^ That might clarify better.

Makefile

camilamacedo86 · 2024-11-19T09:38:52Z

Hi @m1kola

Thank you. Regards #1454 (comment) see : #1454 (comment)

That is not the goal of the PR, and this point can be addressed in a follow-up.

m1kola

Looks good to me overall.

Just one question about $(SETUP_ENVTEST) and setup-envtest targets.

Makefile

m1kola · 2024-12-12T14:57:21Z

Makefile

+	$(SETUP_ENVTEST) use -p env $(ENVTEST_VERSION) $(SETUP_ENVTEST_BIN_DIR_OVERRIDE)
+
+.PHONY: test-unit
+test-unit: $(SETUP_ENVTEST) setup-envtest #HELP Run the unit tests


A bit strange to have $(SETUP_ENVTEST) and setup-envtest targets.

Is it beneficial to have a separate setup-envtest target here?

Maybe we can just do this?

Suggested change

test-unit: $(SETUP_ENVTEST) setup-envtest #HELP Run the unit tests

test-unit: $(SETUP_ENVTEST) #HELP Run the unit tests

mkdir -p $(ROOT_DIR)/bin

$(SETUP_ENVTEST) use -p env $(ENVTEST_VERSION) $(SETUP_ENVTEST_BIN_DIR_OVERRIDE)

$(SETUP_ENVTEST) -> Downloads the binary for setting up envtest.

setup-envtest -> Downloads the Kubernetes binaries and actually sets up envtest.

If we make the suggested change where we are not calling setup-envtest and we do not have an option to do only that, it would allow us to have the required binaries to run the IDE without running the WHOLE unit tests.

If we make the suggested change where we are not calling setup-envtest and we do not have an option to do only that, it would allow us to have the required binaries to run the IDE without running the WHOLE unit tests.

Got it. I still don't like the naming, but I don't think we should block on this. And I can't come up with a meaningful alternative name for setup-envtest as well :)

maybe we could call the target: setup-local-env or something like that. Then, it could be a generic target for developers to bootstrap their local development environment. Right now, that may only be the envtest binaries, but there could me more in the future. Maybe we could also add an @echo "adding envtest binaries to bin/ - see [link to developer docs] for more details"?

m1kola · 2024-12-17T09:11:33Z

Makefile

+	$(SETUP_ENVTEST) use -p env $(ENVTEST_VERSION) $(SETUP_ENVTEST_BIN_DIR_OVERRIDE)
+
+.PHONY: test-unit
+test-unit: $(SETUP_ENVTEST) setup-envtest #HELP Run the unit tests


If we make the suggested change where we are not calling setup-envtest and we do not have an option to do only that, it would allow us to have the required binaries to run the IDE without running the WHOLE unit tests.

Got it. I still don't like the naming, but I don't think we should block on this. And I can't come up with a meaningful alternative name for setup-envtest as well :)

camilamacedo86 · 2024-12-17T09:20:24Z

Hi @m1kola

Thank you very much for the very detailed review.
If we see fit, we can also continuously improve things on follow-ups.
Sometimes, it is easier.

Again, thank you a lot for your review 🥇

This change introduces downloading the specific binaries required to run ENVTEST-based tests into the project/bin directory. This setup makes it easier to configure tests for debugging directly in an IDE. Furthermore, no longer install binaries required for ENVTEST based tests on the project/bin instead of global path.

perdasilva

lgtm - thank you <3

camilamacedo86 · 2024-12-17T11:32:29Z

Thank you @perdasilva and @m1kola

I am moving forward with this one
I think all suggestions and etc are addressed

However, we can always to push enhancements in a follow-up as well.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 13, 2024

camilamacedo86 changed the title ~~WIP: Allow debug tests with IDE~~ (Blocked by #1453) - WIP: 🌱 Allow debug tests with IDE Nov 13, 2024

camilamacedo86 force-pushed the allow-debug-test branch 2 times, most recently from 6ccb706 to bec5b49 Compare November 13, 2024 12:50

camilamacedo86 commented Nov 13, 2024

View reviewed changes

Makefile Show resolved Hide resolved

camilamacedo86 changed the title ~~(Blocked by #1453) - WIP: 🌱 Allow debug tests with IDE~~ WIP: 🌱 Allow debug tests with IDE Nov 13, 2024

camilamacedo86 force-pushed the allow-debug-test branch 5 times, most recently from ff5d402 to 94cd7d3 Compare November 13, 2024 19:30

camilamacedo86 changed the title ~~WIP: 🌱 Allow debug tests with IDE~~ 🌱 (fix): configure ENVTEST binaries for IDE execution and project-specific setup Nov 13, 2024

camilamacedo86 changed the title ~~🌱 (fix): configure ENVTEST binaries for IDE execution and project-specific setup~~ WIP - 🌱 (fix): configure ENVTEST binaries for IDE execution and project-specific setup Nov 13, 2024

camilamacedo86 force-pushed the allow-debug-test branch 5 times, most recently from 703e583 to 9ba4f50 Compare November 13, 2024 20:40

camilamacedo86 changed the title ~~WIP - 🌱 (fix): configure ENVTEST binaries for IDE execution and project-specific setup~~ 🌱 (fix): configure ENVTEST binaries for IDE execution and project-specific setup Nov 13, 2024

camilamacedo86 requested a review from joelanford November 13, 2024 20:48

camilamacedo86 marked this pull request as ready for review November 13, 2024 20:49

camilamacedo86 requested a review from a team as a code owner November 13, 2024 20:49

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 13, 2024

m1kola reviewed Nov 14, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

camilamacedo86 force-pushed the allow-debug-test branch from 9ba4f50 to 3563506 Compare November 17, 2024 11:51

camilamacedo86 force-pushed the allow-debug-test branch from 3563506 to ba71f8b Compare November 17, 2024 12:03

camilamacedo86 requested a review from m1kola November 17, 2024 12:04

camilamacedo86 changed the title ~~WIP (came back to use bingo)🌱 (fix): configure ENVTEST binaries for IDE execution and project-specific setup~~ WIP (TODO: came back to use bingo)🌱 (fix): configure ENVTEST binaries for IDE execution and project-specific setup Nov 18, 2024

This comment was marked as resolved.

Sign in to view

camilamacedo86 force-pushed the allow-debug-test branch from 70048e4 to 038669f Compare November 19, 2024 00:43

camilamacedo86 changed the title ~~WIP (TODO: came back to use bingo)🌱 (fix): configure ENVTEST binaries for IDE execution and project-specific setup~~ 🌱 Configure ENVTEST Binaries for IDE Debugging Nov 19, 2024

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 19, 2024

camilamacedo86 requested a review from m1kola November 19, 2024 00:46

camilamacedo86 force-pushed the allow-debug-test branch from 038669f to c5cdc3d Compare November 19, 2024 01:02

camilamacedo86 commented Nov 19, 2024

View reviewed changes

m1kola reviewed Nov 19, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

Makefile Show resolved Hide resolved

Makefile Show resolved Hide resolved

camilamacedo86 requested a review from m1kola November 19, 2024 09:39

camilamacedo86 force-pushed the allow-debug-test branch from c5cdc3d to 7f4439e Compare November 19, 2024 15:49

m1kola reviewed Dec 12, 2024

View reviewed changes

camilamacedo86 force-pushed the allow-debug-test branch from 7f4439e to 18fa9ac Compare December 16, 2024 18:01

camilamacedo86 requested a review from m1kola December 16, 2024 18:01

m1kola previously approved these changes Dec 17, 2024

View reviewed changes

camilamacedo86 dismissed m1kola’s stale review via 450d120 December 17, 2024 10:37

camilamacedo86 force-pushed the allow-debug-test branch from 18fa9ac to 450d120 Compare December 17, 2024 10:37

perdasilva approved these changes Dec 17, 2024

View reviewed changes

camilamacedo86 added this pull request to the merge queue Dec 17, 2024

Merged via the queue into operator-framework:main with commit d3f267a Dec 17, 2024
18 of 19 checks passed

camilamacedo86 deleted the allow-debug-test branch December 17, 2024 12:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🌱 Configure ENVTEST Binaries for IDE Debugging #1454

🌱 Configure ENVTEST Binaries for IDE Debugging #1454

camilamacedo86 commented Nov 13, 2024 •

edited

Loading

netlify bot commented Nov 13, 2024 •

edited

Loading

codecov bot commented Nov 17, 2024 •

edited

Loading

camilamacedo86 commented Nov 17, 2024

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

camilamacedo86 commented Nov 19, 2024

camilamacedo86 Nov 19, 2024 •

edited

Loading

perdasilva Dec 17, 2024

camilamacedo86 Dec 17, 2024

camilamacedo86 Nov 19, 2024 •

edited

Loading

camilamacedo86 Nov 19, 2024

camilamacedo86 commented Nov 19, 2024

m1kola left a comment

m1kola Dec 12, 2024

camilamacedo86 Dec 16, 2024

m1kola Dec 17, 2024

perdasilva Dec 17, 2024 •

edited

Loading

m1kola Dec 17, 2024

camilamacedo86 commented Dec 17, 2024

perdasilva left a comment

camilamacedo86 commented Dec 17, 2024

🌱 Configure ENVTEST Binaries for IDE Debugging #1454

🌱 Configure ENVTEST Binaries for IDE Debugging #1454

Conversation

camilamacedo86 commented Nov 13, 2024 • edited Loading

Motivation

netlify bot commented Nov 13, 2024 • edited Loading

✅ Deploy Preview for olmv1 ready!

codecov bot commented Nov 17, 2024 • edited Loading

Codecov Report

camilamacedo86 commented Nov 17, 2024

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

camilamacedo86 commented Nov 19, 2024

camilamacedo86 Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

perdasilva Dec 17, 2024

Choose a reason for hiding this comment

camilamacedo86 Dec 17, 2024

Choose a reason for hiding this comment

camilamacedo86 Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

camilamacedo86 Nov 19, 2024

Choose a reason for hiding this comment

camilamacedo86 commented Nov 19, 2024

m1kola left a comment

Choose a reason for hiding this comment

m1kola Dec 12, 2024

Choose a reason for hiding this comment

camilamacedo86 Dec 16, 2024

Choose a reason for hiding this comment

m1kola Dec 17, 2024

Choose a reason for hiding this comment

perdasilva Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

m1kola Dec 17, 2024

Choose a reason for hiding this comment

camilamacedo86 commented Dec 17, 2024

perdasilva left a comment

Choose a reason for hiding this comment

camilamacedo86 commented Dec 17, 2024

camilamacedo86 commented Nov 13, 2024 •

edited

Loading

netlify bot commented Nov 13, 2024 •

edited

Loading

codecov bot commented Nov 17, 2024 •

edited

Loading

camilamacedo86 Nov 19, 2024 •

edited

Loading

camilamacedo86 Nov 19, 2024 •

edited

Loading

perdasilva Dec 17, 2024 •

edited

Loading