Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let Rake spans be disabled programmatically #200

Merged
merged 3 commits into from
Jun 1, 2022

Conversation

ajvondrak
Copy link
Contributor

Which problem is this PR solving?

When running a Rake task, the Honeycomb integration typically creates the root node of the trace (since the task is the entry point into the program). For short-lived tasks like rake db:migrate, this works great. However, tasks might be long-lived. For example, rake resque:work runs forever, dequeueing background jobs on a loop. Ideally, we'd like a separate trace for each job that gets dequeued. But with the Rake integration enabled, all the jobs' traces point back up at the auto-generated root span, which will be missing until the worker process finally dies. This has been discussed before in Pollinators: https://honeycombpollinators.slack.com/archives/CJR134U2F/p1643989199035879

As-is, the only way to disable the Rake integration is to remove it from the HONEYCOMB_INTEGRATIONS env var. There's a bug in the implementation of Rake::Application#honeycomb_client such that setting it to nil doesn't disable the Rake tracing (same sort of bug as in #60). Moreover, we can't pick & choose which Rake tasks generate a span. So if we disable the Rake integration altogether, rake resque:work might look good, but something like rake db:migrate will just generate a bunch of orphans that can't be tied back to a single parent.

This PR allows you to programmatically disable the Rake integration both on the application level and the individual task level.

Short description of the changes

  • Fixed bug in Rake::Application#honeycomb_client so that it will honor when we set it to nil.
  • Modified Rake::Task#honeycomb_client so that it can be customized similarly to the application-level client.
  • Overhauled the specs, which had not been very extensive previously. You can still see the original test in the form of the test:client:access task.

With this, you can selectively disable individual Rake tasks' auto-generated spans by saying

Rake::Task["resque:work"].honeycomb_client = nil

Alternatively, you could disable Rake at the application level and opt-in to the tracing at the task level:

Rake.application.honeycomb_client = nil
Rake::Task["db:migrate"].honeycomb_client = Honeycomb.client

Other thoughts:

  • I refrained from elevating this to a Honeycomb::Configuration option just to keep the change as simple as possible. It's perhaps not the most user-friendly, but it is maximally flexible: use whatever arbitrary logic you want to toggle the client to nil.
  • As common as the Resque use case is, I wonder if it might be worth adding something to the Rails generator. 🤔 But then, I haven't tested out which exact Resque spans to disable this way, yet.

@ajvondrak ajvondrak requested a review from a team April 16, 2022 01:55
@ajvondrak ajvondrak requested a review from martin308 as a code owner April 16, 2022 01:55
@@ -25,7 +25,7 @@ class << self
extend Forwardable
attr_reader :client

def_delegators :@client, :libhoney, :start_span, :add_field,
def_delegators :client, :libhoney, :start_span, :add_field,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is sort of a subtle change. By going through :client instead of :@client, this allows us to use RSpec to stub Honeycomb.client like

allow(Honeycomb).to receive(:client).and_return(...)

And then Honeycomb.start_span & company will go through the stubbed client.

This is important because there's no way (short of instance_variable_set) to reset the client after a Honeycomb.configure. Thus, a Honeycomb.configure in some stray before block could leak into subsequent tests, which might break things depending on the order they're run. It's probably worth going through the rest of the suite and cleaning up the few stray uses of Honeycomb.configure.

@JamieDanielson JamieDanielson added the status: oncall Flagged for awareness from Honeycomb Telemetry Oncall label May 11, 2022
Copy link
Member

@robbkidd robbkidd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

handy with a rake

@MikeGoldsmith MikeGoldsmith merged commit c9ff63f into honeycombio:main Jun 1, 2022
@MikeGoldsmith MikeGoldsmith added the version: bump minor A PR that adds behavior, but is backwards-compatible. label Jun 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: oncall Flagged for awareness from Honeycomb Telemetry Oncall version: bump minor A PR that adds behavior, but is backwards-compatible.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants