Feature Request: improve parallel processing

[tilo]

@xjlin0 commented on May 25

Smarter_csv is a great gem! Save me ton's of time by parallel processing.

One possible improvement I am hoping here, is to let smarter_csv sending out the chunks before finishing reading the entire files. Smarter_csv use readline to read csv files, smartly avoiding reading the entire csv files into memory. However the it seems cannot sending chunks out before finishing the entire csv files.

[tilo]

this looks related to issue #32

[tilo]

needs to be addressed in 2.0

https://felipeelias.github.io/ruby/2017/01/02/fast-file-processing-ruby.html

File.open("test.txt", "r").each_line do |row|
  puts row
end        

#each_line is a streaming method and should be faster

https://stackoverflow.com/questions/39031541/how-to-read-a-large-text-file-line-by-line-and-append-this-stream-to-a-file-line

https://dalibornasevic.com/posts/68-processing-large-csv-files-with-ruby

[tbolender]

I recently stumbled upon this gem when processing a ~400MB large CSV file. Your gem helped me a lot speeding the process up, thank you @tilo a lot for this!

However, it left me a bit helpless when it came to parallel processing. When studying the linked examples like https://xjlin0.github.io/tech/2015/05/25/faster-parsing-csv-with-parallel-processing/, I noticed that they assume that the file is small enough to load it completely in memory. That is not feasible nor practical in my case.

For actual parallel processing of arbitrary large files, I suggest some kind of Enumerable implementation on entry or chunk base. This would, e.g., allow the usage in the lambda syntax of parallel or the manual distribution over a worker infrastructure.

EDIT: If you have anything planned or sketched out already, I am happy to help.

[tilo]

ear-marked for 2.0