Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move default task build_enumerator logic back into the job #967

Merged
merged 1 commit into from
Feb 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 37 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -436,7 +436,7 @@ module Maintenance
def collection
Post.where(created_at: 2.days.ago...1.hour.ago)
end

def process(post)
post.update!(content: "updated content")
end
Expand Down Expand Up @@ -626,6 +626,42 @@ module Maintenance
end
```

### Writing tests for a Task that uses a custom enumerator

Tests for tasks that use custom enumerators need to instantiate the task class
in order to call `#build_enumerator`. Once the task instance is set up, validate
that `#build_enumerator` returns an enumerator yielding pairs of [item, cursor]
as expected.

```ruby
# test/tasks/maintenance/custom_enumerating_task.rb

require "test_helper"

module Maintenance
class CustomEnumeratingTaskTest < ActiveSupport::TestCase
setup do
@task = CustomEnumeratingTask.new
end

test "#build_enumerator returns enumerator yielding pairs of [item, cursor]" do
enum = @task.build_enumerator(cursor: 0)
expected_items = [:b, :c]

assert_equal 2, enum.size

enum.each_with_index do |item, cursor|
assert_equal expected_items[cursor], item
end
end

test "#process performs a task iteration" do
...
end
end
end
```

### Running a Task

#### Running a Task from the Web UI
Expand Down
39 changes: 39 additions & 0 deletions app/jobs/concerns/maintenance_tasks/task_job_concern.rb
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,45 @@ def retry_on(*, **)
def build_enumerator(_run, cursor:)
cursor ||= @run.cursor
@collection_enum = @task.enumerator_builder(cursor: cursor)

@collection_enum ||= case (collection = @task.collection)
when :no_collection
enumerator_builder.build_once_enumerator(cursor: nil)
when ActiveRecord::Relation
enumerator_builder.active_record_on_records(collection, cursor: cursor, columns: @task.cursor_columns)
when ActiveRecord::Batches::BatchEnumerator
if collection.start || collection.finish
raise ArgumentError, <<~MSG.squish
#{@task.class.name}#collection cannot support
a batch enumerator with the "start" or "finish" options.
MSG
end

# For now, only support automatic count based on the enumerator for
# batches
enumerator_builder.active_record_on_batch_relations(
collection.relation,
cursor: cursor,
batch_size: collection.batch_size,
columns: @task.cursor_columns,
)
when Array
enumerator_builder.build_array_enumerator(collection, cursor: cursor&.to_i)
when BatchCsvCollectionBuilder::BatchCsv
JobIteration::CsvEnumerator.new(collection.csv).batches(
batch_size: collection.batch_size,
cursor: cursor&.to_i,
)
when CSV
JobIteration::CsvEnumerator.new(collection).rows(cursor: cursor&.to_i)
else
raise ArgumentError, <<~MSG.squish
#{@task.class.name}#collection must be either an
Active Record Relation, ActiveRecord::Batches::BatchEnumerator,
Array, or CSV.
MSG
end

throttle_enumerator(@collection_enum)
end

Expand Down
44 changes: 2 additions & 42 deletions app/models/maintenance_tasks/task.rb
Original file line number Diff line number Diff line change
Expand Up @@ -260,55 +260,15 @@ def count
self.class.collection_builder_strategy.count(self)
end

# Default enumeration builder. You may override this method to return any
# Default enumerator builder. You may override this method to return any
# Enumerator yielding pairs of `[item, item_cursor]`.
#
# @param cursor [String, nil] cursor position to resume from, or nil on
# initial call.
#
# @return [Enumerator]
def enumerator_builder(cursor:)
collection = self.collection

job_iteration_builder = JobIteration::EnumeratorBuilder.new(nil)

case collection
when :no_collection
job_iteration_builder.build_once_enumerator(cursor: nil)
when ActiveRecord::Relation
job_iteration_builder.active_record_on_records(collection, cursor: cursor, columns: cursor_columns)
when ActiveRecord::Batches::BatchEnumerator
if collection.start || collection.finish
raise ArgumentError, <<~MSG.squish
#{self.class.name}#collection cannot support
a batch enumerator with the "start" or "finish" options.
MSG
end

# For now, only support automatic count based on the enumerator for
# batches
job_iteration_builder.active_record_on_batch_relations(
collection.relation,
cursor: cursor,
batch_size: collection.batch_size,
columns: cursor_columns,
)
when Array
job_iteration_builder.build_array_enumerator(collection, cursor: cursor&.to_i)
when BatchCsvCollectionBuilder::BatchCsv
JobIteration::CsvEnumerator.new(collection.csv).batches(
batch_size: collection.batch_size,
cursor: cursor&.to_i,
)
when CSV
JobIteration::CsvEnumerator.new(collection).rows(cursor: cursor&.to_i)
else
raise ArgumentError, <<~MSG.squish
#{self.class.name}#collection must be either an
Active Record Relation, ActiveRecord::Batches::BatchEnumerator,
Array, or CSV.
MSG
end
nil
end
end
end
3 changes: 2 additions & 1 deletion test/dummy/app/tasks/maintenance/custom_enumerating_task.rb
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ def count
3
end

def process(_)
def process(letter)
Rails.logger.debug("letter: #{letter}")
end
end
end
Loading