Skip to content

Commit

Permalink
Use cgroups aware processor count by default
Browse files Browse the repository at this point in the history
In containerized environments, a number of CPU cores isn't the
same as the available CPUs. In this case, we need to consider cgroups.

`concurrent-ruby` now has the method to get that since v1.3.1. I think it's better
to use this for setting more container environment friendly default value.
Ref: ruby-concurrency/concurrent-ruby#1038

I keep `Parallel.processor_count` as is to keep supporting setting
processor count via env(`PARALLEL_PROCESSOR_COUNT`).
  • Loading branch information
y-yagi committed Aug 8, 2024
1 parent 9644d13 commit f7bfe1b
Show file tree
Hide file tree
Showing 4 changed files with 7 additions and 9 deletions.
3 changes: 2 additions & 1 deletion Gemfile.lock
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ PATH
remote: .
specs:
parallel (1.25.1)
concurrent-ruby (~> 1.0, >= 1.3.1)

GEM
remote: https://rubygems.org/
Expand All @@ -19,7 +20,7 @@ GEM
zeitwerk (~> 2.3)
ast (2.4.2)
bump (0.10.0)
concurrent-ruby (1.1.9)
concurrent-ruby (1.3.3)
diff-lcs (1.5.0)
i18n (1.10.0)
concurrent-ruby (~> 1.0)
Expand Down
7 changes: 4 additions & 3 deletions lib/parallel.rb
Original file line number Diff line number Diff line change
Expand Up @@ -337,10 +337,11 @@ def physical_processor_count
end
end

# Number of processors seen by the OS, used for process scheduling
# Number of processors seen by the OS or value considering CPU quota if the process is inside a cgroup,
# used for process scheduling
def processor_count
require 'etc'
@processor_count ||= Integer(ENV['PARALLEL_PROCESSOR_COUNT'] || Etc.nprocessors)
require 'concurrent-ruby'
@processor_count ||= Integer(ENV['PARALLEL_PROCESSOR_COUNT'] || Concurrent.available_processor_count.to_i)
end

def worker_number
Expand Down
1 change: 1 addition & 0 deletions parallel.gemspec
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,6 @@ Gem::Specification.new name, Parallel::VERSION do |s|
}
s.files = `git ls-files lib MIT-LICENSE.txt`.split("\n")
s.license = "MIT"
s.add_runtime_dependency "concurrent-ruby", "~> 1.0", ">= 1.3.1"
s.required_ruby_version = '>= 2.7'
end
5 changes: 0 additions & 5 deletions spec/parallel_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,6 @@ def without_ractor_warning(out)
(1..999).should include(Parallel.processor_count)
end
end

it 'uses Etc.nprocessors in Ruby 2.2+' do
defined?(Etc).should == "constant"
Etc.respond_to?(:nprocessors).should == true
end
end

describe ".physical_processor_count" do
Expand Down

0 comments on commit f7bfe1b

Please sign in to comment.