Add parallelization to tootctl search deploy (#12051)

* Add parallel gem

* Modify parallel option in tootctl search deploy

* Add paralell option to tootctl search deploy

* Change 1 to false

* Clean up

* Rename --parallel to --processes
This commit is contained in:
Takeshi Umeda 2019-10-03 04:50:43 +09:00 committed by Eugen Rochko
parent 4e1afef6f9
commit 0ce0baa9b5
3 changed files with 23 additions and 2 deletions

View File

@ -67,6 +67,7 @@ gem 'oj', '~> 3.9'
gem 'ostatus2', '~> 2.0' gem 'ostatus2', '~> 2.0'
gem 'ox', '~> 2.11' gem 'ox', '~> 2.11'
gem 'parslet' gem 'parslet'
gem 'parallel', '~> 1.17'
gem 'posix-spawn', git: 'https://github.com/rtomayko/posix-spawn', ref: '58465d2e213991f8afb13b984854a49fcdcc980c' gem 'posix-spawn', git: 'https://github.com/rtomayko/posix-spawn', ref: '58465d2e213991f8afb13b984854a49fcdcc980c'
gem 'pundit', '~> 2.1' gem 'pundit', '~> 2.1'
gem 'premailer-rails' gem 'premailer-rails'

View File

@ -748,6 +748,7 @@ DEPENDENCIES
ox (~> 2.11) ox (~> 2.11)
paperclip (~> 6.0) paperclip (~> 6.0)
paperclip-av-transcoder (~> 0.6) paperclip-av-transcoder (~> 0.6)
parallel (~> 1.17)
parallel_tests (~> 2.29) parallel_tests (~> 2.29)
parslet parslet
pg (~> 1.1) pg (~> 1.1)

View File

@ -6,6 +6,7 @@ require_relative 'cli_helper'
module Mastodon module Mastodon
class SearchCLI < Thor class SearchCLI < Thor
option :processes, default: 2, aliases: [:p]
desc 'deploy', 'Create or update an ElasticSearch index and populate it' desc 'deploy', 'Create or update an ElasticSearch index and populate it'
long_desc <<~LONG_DESC long_desc <<~LONG_DESC
If ElasticSearch is empty, this command will create the necessary indices If ElasticSearch is empty, this command will create the necessary indices
@ -13,10 +14,28 @@ module Mastodon
This command will also upgrade indices if the underlying schema has been This command will also upgrade indices if the underlying schema has been
changed since the last run. changed since the last run.
With the --processes option, parallelize execution of the command. The
default is 2. If "auto" is specified, the number is automatically
derived from available CPUs.
LONG_DESC LONG_DESC
def deploy def deploy
processed = Chewy::RakeHelper.upgrade processed = Chewy::RakeHelper.upgrade(parallel: processes)
Chewy::RakeHelper.sync(except: processed) Chewy::RakeHelper.sync(except: processed, parallel: processes)
end
private
def processes
return true if options[:processes] == 'auto'
num = options[:processes].to_i
if num < 2
nil
else
num
end
end end
end end
end end