ARROW-5505: [R] Normalize file and class names, stop masking base R functions, add vignette, improve documentation

[nealrichardson]

The main thrust of the changes are summarized in the new vignette:

C++ is an object-oriented language, so the core logic of the Arrow library is encapsulated in classes and methods. In the R package, these classes are implemented as R6 reference classes, most of which are exported from the namespace.

In order to match the C++ naming conventions, the R6 classes are in TitleCase, e.g. RecordBatch. This makes it easy to look up the relevant C++ implementations in the code or documentation. To simplify things in R, the C++ library namespaces are generally dropped or flattened; that is, where the C++ library has arrow::io::FileOutputStream, it is just FileOutputStream in the R package. One exception is for the file readers, where the namespace is necessary to disambiguate. So arrow::csv::TableReader becomes CsvTableReader, and arrow::json::TableReader becomes JsonTableReader.

Some of these classes are not meant to be instantiated directly; they may be base classes or other kinds of helpers. For those that you should be able to create, use the $create() method to instantiate an object. For example, rb <- RecordBatch$create(int = 1:10, dbl = as.numeric(1:10)) will create a RecordBatch. Many of these factory methods that an R user might most often encounter also have a snake_case alias, in order to be more familiar for contemporary R users. So record_batch(int = 1:10, dbl = as.numeric(1:10)) would do the same as RecordBatch$create() above.

The typical user of the arrow R package may never deal directly with the R6 objects. We provide more R-friendly wrapper functions as a higher-level interface to the C++ library. An R user can call read_parquet() without knowing or caring that they're instantiating a ParquetFileReader object and calling the $ReadFile() method on it. The classes are there and available to the advanced programmer who wants fine-grained control over how the C++ library is used.

There are a few other fixes and cleanups rolled in here, named in the individual commit messages below.

I stopped short of more documentation consolidation because (1) this patch is already huge and (2) R6 classes are really tedious to document because it's all manual. I did some searching around and found open issues from 2014 and 2015 about supporting R6 better in roxygen2.

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (master@4f7ead8). Click here to learn what that means.
The diff coverage is 81.58%.

@@            Coverage Diff            @@
##             master    #5279   +/-   ##
=========================================
  Coverage          ?   76.06%           
=========================================
  Files             ?       56           
  Lines             ?     3572           
  Branches          ?        0           
=========================================
  Hits              ?     2717           
  Misses            ?      855           
  Partials          ?        0

Impacted Files	Coverage Δ
r/R/enums.R	0% <ø> (ø)
r/R/arrow-package.R	63.63% <ø> (ø)
r/R/memory_pool.R	0% <0%> (ø)
r/src/recordbatch.cpp	86.5% <100%> (ø)
r/R/list.R	100% <100%> (ø)
r/src/array_from_vector.cpp	78.89% <100%> (ø)
r/R/compute.R	100% <100%> (ø)
r/R/compression.R	100% <100%> (ø)
r/R/json.R	100% <100%> (ø)
r/R/buffer.R	81.81% <100%> (ø)
... and 22 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4f7ead8...0b866c3. Read the comment docs.

[pitrou]

I can't speak about this PR, but as a R neophyte, I find the quoted "vignette" quite useful.

[nealrichardson]

In order to show the documentation changes, I published the pkgdown site from this branch at https://enpiar.com/arrow-r-site/. Decent examples include https://enpiar.com/arrow-r-site/reference/ParquetFileReader.html and https://enpiar.com/arrow-r-site/reference/FeatherTableReader.html

[nealrichardson]

Also preview the vignette at https://enpiar.com/arrow-r-site/articles/arrow.html

[nealrichardson]

I don't know what's up with that Ursabot builder but I'm wondering if it's related to d7ef11f ? While this patch is big, it's pretty superficial and shouldn't affect anything build/path related. Travis and Appveyor have been steadily passing on this branch, and those are the ones I pay attention to.

[wesm]

Could be related, but it's definitely weird because this is passing on master. Maybe rebase?

Note that you're mixing underscore- and hyphen-based file names for the R files, do you want to make them consistent?

[romainfrancois]

This is huge indeed, thanks for doing this. Few comments to follow up on our conversation:

• R does not have a way to nest namespaces, so we can't have e.g. arrow::csv::FileReader on the R side, the choice of squashing the namespace in the name, camel casing it along the way looks "fine for now" because there aren't many cases, and it's better looking than faux namespacing as in arrow::csv_FileReader. I'm just worried about the future proofing of this, it makes it somewhat harder to map C++ class names with their R stand ins. Given classes are not used "directly" most of the time, maybe going with predictable faux namespacing is more useful 🤷‍♂ ?

• I guess DataType$dispatch() should be DataType$create() as well.

[romainfrancois]

Maybe we could leave the fully class names ?

[romainfrancois]

I mean "arrow::DataType" as the R6class(classname=)

[romainfrancois]

Suggested change

	chunks = function() map(ChunkedArray__chunks(self), ~ Array$create(.x)),
	chunks = function() map(ChunkedArray__chunks(self), Array$create),

That probably works as well

[nealrichardson]

I understand the buildbot failure now: because there is a vignette, R CMD build is installing the package to build the vignette, and that's failing because LD_LIBRARY_PATH isn't set in the build command, only the check command: https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/builders.py#L262

Trying to confirm locally now.

[kou]

It seems that "Ursabot / AMD64 Ubuntu 18.04 R" is failed since this merge: https://ci.ursalabs.org/#/builders/96/builds/1246

[nealrichardson]

@kou I believe I fixed that in voltrondata-labs/ursabot#176, but maybe something has to happen for that change to be deployed?

[kou]

Thanks!