Dashboard search doesn't always match kwargs

[gap777]

We sprinkle metadata into our job kwargs to enable lookup in the Dashboard, like this:

{"ad_id"=>"ad_qTB3kTjQeZEmptyx8JZ5", "ad_ref"=>"A-1012679", "line_item_gid"=>"gid://supply-side-platform/Orders::Types::ReservationLineItem/rli_Dd2LpagvXuqaZNAZt4yu", "purchase_order_id"=>"po_8o9yOwc88sfMQWFmjE8F", "_aj_ruby2_keywords"=>["purchase_order_id", "line_item_gid", "ad_ref", "ad_id"]}

• Searching for po_8o9yOwc88sfMQWFmjE8F finds the job.
• Searching for 8o9yOwc88sfMQWFmjE8F (a substring of the previous arg) finds the job
• Searching for A-1012679 does NOT find the job
• Searching for gid://supply-side-platform/Orders::Types::ReservationLineItem/rli_Dd2LpagvXuqaZNAZt4yu does NOT.
• Searching for rli_Dd2LpagvXuqaZNAZt4yu (a substring of the previous arg) does NOT.

Why can some args (po_8o9yOwc88sfMQWFmjE8F) be used for searching and others (A-1012679) can not be?

[bensheldon]

I think this might be a side effect of Postgres Full Text search. This is the query I'm using:

good_job/app/models/concerns/good_job/filterable.rb

Lines 37 to 42 in b557525

	# TODO: turn this into proper bind parameters in Arel
	tsvector = "(to_tsvector('english', id::text) || to_tsvector('english', COALESCE(active_job_id::text, '')) || to_tsvector('english', serialized_params) || to_tsvector('english', COALESCE(serialized_params->>'arguments', '')) || to_tsvector('english', COALESCE(error, '')) || to_tsvector('english', COALESCE(array_to_string(labels, ' '), '')))"
	to_tsquery_function = database_supports_websearch_to_tsquery? ? 'websearch_to_tsquery' : 'plainto_tsquery'
	where("#{tsvector} @@ #{to_tsquery_function}(?)", query)
	.order(sanitize_sql_for_order([Arel.sql("ts_rank(#{tsvector}, #{to_tsquery_function}(?))"), query]) => 'DESC')
	end)

Maybe english is the wrong stemmer to use, or something else.

[gap777]

Right... if pg dictionaries are losing characters b/c these aren't english words, that could do it....
any interest in expanding the kinds of strings that can be used to find jobs via this mechanism?
or is there another way we can make this metadata available from a job in order to enable search?

[bensheldon]

yep! I think we should try it. I feel pretty good about the current regression tests, so we can be creative. I think probably the complexity of these arguments can be expanded to cover your needs:

good_job/spec/app/models/concerns/good_job/filterable_spec.rb

Lines 8 to 23 in b557525

	model_class.create!(
	active_job_id: SecureRandom.uuid,
	queue_name: "default",
	job_class: "ExampleJob",
	scheduled_at: Time.current,
	serialized_params: { example_key: 'example_value', arguments: ["hello-arg", 998899] },
	labels: %w[buffalo gopher],
	error: "ExampleJob::ExampleError: a message",
	error_event: "retried"
	)
	end
	describe '.search_test' do
	it 'searches serialized params' do
	expect(model_class.search_text('example_value')).to include(job)
	end

...and then we just gotta find a query that matches them.