Submission: tidync

[mdsumner]

Summary

• What does this package do? (explain in 50 words or less):

Explore the contents of a NetCDF source (file or URL) presented as variables organized by grids with a database-like interface. The hyper_filter verb shows the effects of array-slicing expressions by value or index. Actual data read is delayed until explicitly requested, as a data frame or list of arrays.

• Paste the full DESCRIPTION file inside a code block below:

Package: tidync
Title: A Tidy Approach to 'NetCDF' Data Exploration and Extraction
Version: 0.0.3
Authors@R: c(person("Michael", "Sumner", email = "[email protected]", role = c("aut", "cre")), 
             person("Simon", "Wotherspoon", role = "ctb"), 
             person("Tomas", "Remenyi", role = "ctb"), 
             person("Ben", "Raymond", role = "ctb"))
Description: Tidy tools for 'NetCDF' data sources. Explore the contents of a NetCDF source (file or URL) 
 presented as variables organized by grid with a database-like interface. The 'hyper_filter' verb 
 shows the effects of array-slicing expressions by value or index. Actual data read is delayed until explicitly 
 requested, as a data frame or list of arrays. 
Depends: R (>= 3.3.0)
License: GPL-3
Encoding: UTF-8
LazyData: true
ByteCompile: true
Imports:
    dplyr (>= 0.7.0), 
    forcats,
    magrittr,
    ncdf4,
    ncmeta (>= 0.0.2), 
    purrr, 
    RNetCDF  (>= 1.9-1),
    raster,
    rlang, 
    tabularaster,
    tibble,
    yesno
RoxygenNote: 6.0.1
Suggests: ggplot2,
    knitr,
    memoise,
    ncdump,
    palr,
    rmarkdown,
    testthat,
    covr
Remotes: hypertidy/ncmeta
Roxygen: list(markdown = TRUE)
URL: https://github.com/hypertidy/tidync
BugReports: https://github.com/hypertidy/tidync/issues

• URL for the package (the development repository, not a stylized html page):

https://github.com/hypertidy/tidync

• Please indicate which category or categories from our [package fit policies]

1. data munging because the NetCDF model is so general that no one size fits all, tidync provides ease-of-exploration for development of domain-specific access tools as well as for understanding a new source
2. data extraction because the data can be actually read by hyper_slice in raw form, with a higher-level output provided by the hyper_tibble wrapper

•   Who is the target audience and what are scientific applications of this package?  

The target audience is users who learn so much about their particular sub-domain that they become programmers helping others in that arena. They either wrap around tidync to build an interface to a NetCDF source-family, or simply use it to learn to craft lower level calls more directly to the API (with packages RNetCDF, ncdf4, rgdal, rhdf5, etc).

• Are there other R packages that accomplish the same thing? If so, how does
  yours differ or meet [our criteria for best-in-category]

The dplyr tbl_cube is the nearest and the in-development stars does have some overlap. but I think the virtual table abstraction in tidync is novel, albeit very heavily inspired by the "laziness" of ggplot2 and the multiple-tables approach in tidygraph. In terms of useability overall - but not in terms of generality - the raster package is the best but it does not handle non-geographic arrays well, can only deal with 2D or 3D slices from higher forms, and does not support grids with non-regular (non-affine) coordinates.

•   If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted.

Here's the pre-submission enquiry:

#167

Requirements

Confirm each of the following by checking the box. This package:

• does not violate the Terms of Service of any service it interacts with.
• has a CRAN and OSI accepted license.
• contains a README with instructions for installing the development version.
• includes documentation with examples for all functions.
• contains a vignette with examples of its essential functions and uses.
• has a test suite.
• has continuous integration, including reporting of test coverage, using services such as Travis CI, Coeveralls and/or CodeCov.
• I agree to abide by ROpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

Publication options

• Do you intend for this package to go on CRAN?
• Do you wish to automatically submit to the Journal of Open Source Software? If so:
  • The package has an obvious research application according to JOSS's definition.
  • The package contains a paper.md matching JOSS's requirements with a high-level description in the package root or in inst/.
  • The package is deposited in a long-term repository with the DOI:
  • (Do not submit your package separately to JOSS)
• Do you wish to submit an Applications Article about your package to Methods in Ecology and Evolution? If so:
  • The package is novel and will be of interest to the broad readership of the journal.
  • The manuscript describing the package is no longer than 3000 words.
  • You intend to archive the code for the package in a long-term repository which meets the requirements of the journal.
  • (Please do not submit your package separately to Methods in Ecology and Evolution)

Detail

• Does R CMD check (or devtools::check()) succeed? Paste and describe any errors or warnings:

Passing locally (on Linux), but still having problems on travis.

• Does the package conform to rOpenSci packaging guidelines? Please describe any exceptions:

• If this is a resubmission following rejection, please explain the change in circumstances:

• If possible, please provide recommendations of reviewers - those with experience with similar packages and/or likely users of your package - and their GitHub user names:

[karthik]

👋 @mdsumner We have been on break (and for a couple more days). I will advise with next steps upon my return next week.

[sckott]

Editor checks:

• Fit: The package meets criteria for fit and overlap
• Automated tests: Package has a testing suite and is tested via Travis-CI or another CI service.
• License: The package has a CRAN or OSI accepted license
• Repository: The repository link resolves correctly
• Archive (JOSS only, may be post-review): The repository DOI resolves correctly
• Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

---

Editor comments

Thanks for your submission @mdsumner !!

• You can put a ropensci review badge in your README

[![](https://badges.ropensci.org/174_status.svg)](https://github.com/ropensci/onboarding/issues/174)

• Goodpractice output for you and reviewers to consider:

It is good practice to

  ✖ write unit tests for all functions, and all package code
    in general. 60% of code lines are covered by test cases.

    R/activate.R:32:NA
    R/activate.R:33:NA
    R/activate.R:35:NA
    R/activate.R:36:NA
    R/activate.R:54:NA
    ... and 132 more lines

  ✖ avoid long code lines, it is bad for readability. Also,
    many people prefer editor windows that are about 80 characters
    wide. Try make your lines shorter than 80 characters

    R/activate.R:54:1
    R/activate.R:56:1
    R/activate.R:59:1
    R/activate.R:73:1
    R/activate.R:75:1
    ... and 53 more lines

  ✖ fix this R CMD check ERROR: Packages suggested but not
    available: ‘ncdump’ ‘palr’ The suggested packages are required for
    a complete check. Checking can be attempted without them by setting
    the environment variable _R_CHECK_FORCE_SUGGESTS_ to a false value.
    See section ‘The DESCRIPTION file’ in the ‘Writing R Extensions’
    manual.

Seeking reviewers now 🕐

---

Reviewers: @timcdlucas @Nowosad
Due date: 2018-02-08

[mdsumner]

Thanks @sckott ! I added the rOpenSci badge, and I'll fix the lines and lack of tests.

I expect the "suggested but not available" error goes away if you install palr and ncdump from CRAN.

Editor, and reviewers: please note there's a Remotes: hypertidy/ncmeta which will need to go on CRAN soon as well.

[sckott]

Reviewers assigned

Reviewers: @timcdlucas @Nowosad
Due date: 2018-02-08

[Nowosad]

Package Review

Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide

• As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Documentation

The package includes all the following forms of documentation:

• A statement of need clearly stating problems the software is designed to solve and its target audience in README
• Installation instructions: for the development version of package and any non-standard dependencies in README
• Vignette(s) demonstrating major functionality that runs successfully locally
• Function Documentation: for all exported functions in R help
• Examples for all exported functions in R Help that run successfully locally
• Community guidelines including contribution guidelines in the README or CONTRIBUTING, and DESCRIPTION with URL, BugReports and Maintainer (which may be autogenerated via Authors@R).

For packages co-submitting to JOSS

• The package has an obvious research application according to JOSS's definition

The package contains a paper.md matching JOSS's requirements with:

• A short summary describing the high-level functionality of the software
• Authors: A list of authors with their affiliations
• A statement of need clearly stating problems the software is designed to solve and its target audience.
• References: with DOIs for all those that have one (e.g. papers, datasets, software).

Functionality

• Installation: Installation succeeds as documented.
• Functionality: Any functional claims of the software been confirmed.
• Performance: Any performance claims of the software been confirmed.
• Automated tests: Unit tests cover essential functions of the package
  and a reasonable range of inputs and conditions. All tests pass on the local machine.
• Packaging guidelines: The package conforms to the rOpenSci packaging guidelines

Final approval (post-review)

• The author has responded to my review and made changes to my satisfaction. I recommend approving this package.

Estimated hours spent reviewing:

• 4

---

Review Comments

• The tidync package provides tools to explore NetCDF files and allows to extract the data as an array, tibble, or tbl_cube.
  While I think this package give ease the exploration of NetCDFs, there is still a room for improvement.
  It includes, among others, expanded vignette, improved examples in the help files, more unified function names.

Installation

• I had no trouble installing/building the package from GitHub and a local clone of the repository.

Automated tests

• devtools::check() run time is rather long - ~5 minutes (Running ‘testthat.R’ [11s/211s]).
  Additionally, devtools::test() shows that seven tests were skipped (see below).

══ Results ════════════════════════════════════════════════════════════════════════════════════════════════════
Duration: 239.6 s

OK:       8
Failed:   0
Warnings: 0
Skipped:  7

• The raadtools package is used in tests but it's not included in the Description file.
  This non-CRAN package is also used in some of examples in the help files.
• devtools::spell_check indicates a typo in the axis_transforms.Rd file:

spell_check()
  WORD        FOUND IN
dimesion    axis_transforms.Rd:30

README

• The order of elements in the README file is not consistent with the rOpenSci Packaging Guide.
  E.g. some badges are below the package name, while some are above the installation code.
• The README file has an appveyor badge, but the build status is unknown.
• Another packages supporting NetCDF are mentioned in the README file.
  Maybe it would be worthwhile to describe how the package compares to other similar packages.
• I would suggest to rewrite the Terminology section.
  There are a few problems with it, including a mix of first and third person narrative and complex stories behind the terminology used.
  How about recreating it as a glossary, where each important term is explained in a few sentences?
• Also consider adding the code of conduct badge with devtools::use_code_of_conduct.

Vignette

• The tidync package has one vignette called "Tidy NetCDF examples".
  While it give an overview of some of the package's capabilities, it also look unfinished.
• I was unable to see any difference between the first and the second example in the tidync section of the vignette.
  It seems that tidync(f) and tidync(f) %>% activate(concentration) have the same active variable.
• The vignette also ends abruptly with a small uncommented chunk of code relating to the hyper_tibble function.
• I would suggest to extend the vignette (especially the data extraction section) and to add functions that are not mentioned in the vignette or README, such as axis_transforms, hyper_tbl_cube, tidyfile.

Examples

I would suggest improving some of the examples in the help files:

• The tidyfile and tidync functions examples require files from the raadfiles package.
• The var_names example looks unhelpful as it gives NULL as a result.

Function names

I found two small inconsistencies in the function names:

1. The main function is called tidync, next there is a large group of hyper_* functions, and finally there also some called differently: activate, dim_names, var_names.
   Why this inconsistency exists?
   Could the last group also have the hyper_ prefix, e.g. hyper_activate?
2. hyper_tibble() returns tibble, hyper_tbl_cube() returns tbl_cube, and hyper_slice() returns... array.
   Have you thought about changing the name of the last function to hyper_array?

Questions

• Interactive mode allows for subseting dimensions by values or indices.
  Is it possible to get actual values (or their ranges) in the interactive mode?
  It could ease the subsetting process.

@mdsumner thanks for developing the package and let me know if you have any questions regarding to my comments.

[sckott]

thanks for your review @Nowosad !

[mdsumner]

thanks @Nowosad - lots of good points here! I'm trying a few things to come up with a better naming scheme, your feedback is very helpful

In terms of the Question "Is it possible to get actual values (or their ranges) in the interactive mode?" , I guess you mean seeing the range of "chlor_a" in this output?

 library(tidync)
 f <- "S20092742009304.L3m_MO_CHL_chlor_a_9km.nc"
 l3file <- system.file("extdata/oceandata", f, package= "tidync")
 ## filter by value
 tidync(l3file) %>% hyper_filter(lon = lon < 100)

The problem is that this filtering is completely lazy, there's only information about the metadata (axis names and step values) - so it's effectively free in computation terms. Polling the variables for values implies quite a lot of i/o. It's possible to get the "valid range" for some variables, but I'm not sure how useful that would be (not affected by filter expressions, not always present afaik).

[mdsumner]

Ah, it didn't occur to me until now that the ranges of 1D axis variables is useful, that definitely has to be in the summary - big multi-d variables are hard to scan, but the axis ranges are readily available:

x <- tidync(l3file)
axes <- axis_transforms(x)
lapply(names(axes), function(nm) range(axes[[nm]][[nm]]))

I couldn't see a good purrrish way to do that but I'll figure it out and make sure those ranges are in the summary print.

[Nowosad]

@mdsumner That would definitely be an improvement.

[timcdlucas]

Just a check in to say sorry for being slow. I'll be done by the 3 week deadline.

[timcdlucas]

tidync review

Author: Tim Lucas

Package Review

Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide

• As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Documentation

The package includes all the following forms of documentation:

• A statement of need clearly stating problems the software is designed to solve and its target audience in README
• Installation instructions: for the development version of package and any non-standard dependencies in README
• Vignette(s) demonstrating major functionality that runs successfully locally
• Function Documentation: for all exported functions in R help
• Examples for all exported functions in R Help that run successfully locally
• Community guidelines including contribution guidelines in the README or CONTRIBUTING, and DESCRIPTION with URL, BugReports and Maintainer (which may be autogenerated via Authors@R).

Functionality

• Installation: Installation succeeds as documented.
• Functionality: Any functional claims of the software been confirmed.
• Performance: Any performance claims of the software been confirmed.
• Automated tests: Unit tests cover essential functions of the package
  and a reasonable range of inputs and conditions. All tests pass on the local machine.
• Packaging guidelines: The package conforms to the rOpenSci packaging guidelines

Final approval (post-review)

• The author has responded to my review and made changes to my satisfaction. I recommend approving this package.

Estimated hours spent reviewing: 4 (lots of time wasted trying to install NetCDF)

---

Review Comments

Overview

This package provides a nice interface for dealing with NetCDF files. It seems to work as stated showing summaries of NetCDF files without reading the whole file and allowing slices from the data to be loaded and munged into a number of useful data formats.

Specific comments

While it could possibly be assumed that the user has NetCDF installed (why else would they want to use the package?) some details on how to install it would still be useful in the README. I have switched computers since I last used NetCDF and it did take me quite a while to get it installed.

Large block of commented out examples in hyper_tbl_cube and tidyfile.

raadtools in tidync example is a github only package.
This is quite annoying for the example of the main function.

And then I get:

> cf <- raadtools::cpolarfiles()
Error in UseMethod("filter_") : 
  no applicable method for 'filter_' applied to an object of class "NULL"

ggplot2 example in hyper_tibble would be better as

> ggplot(ht %>% filter(!is.na(chlor_a)), 
+        aes(x = lon, y = lat, fill = chlor_a)) + geom_tile()

Example in var_names doesn't seem to work.

> l3file <- "S20092742009304.L3m_MO_CHL_chlor_a_9km.nc"
> rnc <- tidync(system.file("extdata", "oceandata", l3file, package= "tidync"))
> var_names(rnc)
NULL

Furthermore the description for var_names is very terse. Even though it is fairly clear what the function does just from the function name.

The tests for the function activate are currently commented out.

raadtools is not mentioned in DESCRIPTION even though it is used in many examples and unit tests.

The function we_are_raady is used in tests to skip. But as far as I can see the function isn't defined in raadtools, raadfiles or tidync.

As mentioned above, after installing raadtools I still get for example

> oisst_dayfile <- raadtools::sstfiles()$fullname[1]
Error in UseMethod("filter_") : 
  no applicable method for 'filter_' applied to an object of class "NULL"

so I cannot run any of the unit tests.

+1 for the request for variable ranges. It is hard to know how to subset until you know this!

Conclusion

Overall, this looks like a promising package with a clear and useful set of functionality.
However, at the moment the documentation and tests need some work. Thanks for the work so far @mdsumner and others.

[mdsumner]

Thanks both @Nowosad and @timcdlucas I'm working through these issues, I appreciate the reviews!

One request I have is if there were NetCDF files/sources of interest that you used? Are there sources that would be compelling if I built them into the tests, or as example data in the package? I totally agree it's important to remove the raadtools stuff, but I otherwise find it hard to get enough interesting data to test with - I'd lost sight of how dependent on that local system I am, so apologies for that.

Would it be suitable to have the tests download files, perhaps using rappdirs and just skip those when on CRAN, but link to them in examples?

[mdsumner]

One other question, @timcdlucas were you working on Ubuntu? Is that where you have problems installing? I'm going through a fresh install on 16.04 to make sure it works, but I should do the same with Mac somehow too.

[sckott]

thanks @timcdlucas

@Nowosad when you get a chance can you look at the new version and see if you have any more feedback?

[Nowosad]

I've already took a look at the changes and they are 👍 . The only thing that still can be improved is the vignette.

[sckott]

thanks @Nowosad

[timcdlucas]

Hi all,

Great work @mdsumner.

Hope I'm looking at the right version. I'm using bfb975ce565fc807e482497c7a7b953b6e66463c

I still get 1 test failure.

>   expect_equal(active(tnc1),  "D1,D12")

> active(tnc2)
[1] "D1,D12"

in test-activate.R.

Other than that the only thing I can see is that the readme isn't currently runeable. filename is not defined for example. This chunk is set as eval = FALSE so I guess it's deliberate, but I think it's more useful if the readme is something that can just be copied into an R session.

Otherwise I'm happy!

Tim

[mdsumner]

Awesome, thanks so much @timcdlucas

I'll clean those things up!

[mdsumner]

I think I've addressed everything, thanks again for the responses! I've done a run over the documentation and it's a lot tighter and uses links better now, every bit of feedback here has been really helpful and I appreciate the patience provided to me. I didn't foresee this being wrapped up so nicely a few weeks ago.

(For CRAN it's down to needing a release of ncmeta, which I'm stuck on. )

[sckott]

nice work @mdsumner - any issues we can help on with ncmeta?

[mdsumner]

It comes down to revdepcheck which I just can't get to finish ATM. I need to make sure a release doesn't disrupt stars,and I don't feel confident on that without revdep. If you revdepcheck ncmeta and share the results that would be helpful?

[sckott]

problems and readme from revdepcheck aattached

problems.txt
README.txt

[mdsumner]

That was fast! Awesome, that's very helpful - I think I only need to condition on version pre and post and it'll be safe. THANKS (hopefully I can get revdep working again too ...)

[mdsumner]

I've got this sorted now, ncmeta will have to follow the next version of stars onto CRAN.

[mdsumner]

And now on CRAN.

I still see stuff that needs fixing but functionally I think all is well. THANK YOU @sckott @timcdlucas and @Nowosad ! A while back I just didn't think this was going to work, but the whole process has been extremely helpful and I appreciate all the patience.

It occurred to me I haven't credited efforts from this review enough in the package itself, but I will rectify that.

While it's fresh, the things I got wrong during the CRAN submission:

• ALL Rd files should have examples (still not sure if package_doc Rd needs that technically, but I did it)
• I had print() calls in tidync(), they have to be message() or warning(). print() or cat() is fine in print methods, i.e. in print.tidync() and print.tidync_data()
• special words in the Decription need single quotes e.g. 'NetCDF' but function names must have parentheses and no quotes e.g.

Description: The function hyper_array() is wicked awesome for 'NetCDF' stuffs.

[timcdlucas]

Thanks, glad to have helped.

Didn't know print was explicitly banned. Will remember that.

[mdsumner]

@sckott are we good for rOpenSci here? I found some personal time to push this out to CRAN and kind of lost track of the status here -can we go to the next steps? Apologies if I've overlooked something I needed to do

[sckott]

@mdsumner All is good, was waiting i think for ncmeta to get on CRAN, now it's on CRAN.

Approved! Thanks again for your submission @mdsumner ! And thanks for your reviews @Nowosad and @timcdlucas 👌

To-dos:

• Please transfer the package to ropensci- I've invited you to a team (you need to accept that invitation first, before transferring). You'll be made admin once you transfer
• add rOpenSci footer to README
  [![ropensci_footer](https://ropensci.org/public_images/ropensci_footer.png)](https://ropensci.org)
• Change any needed links, such those for CI badges
• Travis CI should update to the new location automatically - you may have to update other CI systems manually
• it's a good idea to add issue and PR template files to your .github folder, egs here https://github.com/ropensci/dotgithubfiles/
• We're starting to roll out software metadata files to all ropensci pkgs via the Codemeta initiative, see https://github.com/ropensci/codemetar/#codemetar for how to include it in your pkg, after installing the pkg - should be easy as running codemetar::write_codemeta() in the root of your pkg

---

We've started putting together a bookdown with our best practice and tips, this chapter starts the 3d section that's about guidance for after onboarding. Please tell us what could be improved. The repo is at https://github.com/ropensci/dev_guide

Are you interested in doing a blog post for our blog https://ropensci.org/blog/ ? either a short-form intro to it (https://ropensci.org/technotes/) or long-form post with more narrative about its development (https://ropensci.org/blog/). If so, we'll have our community manager @stefaniebutland get in touch with you on that

[mdsumner]

Great thanks! All done apart from

• admin for me on ropensci/tidync (change pkgdown doc link)
• appveyor badge

tidync 0.2.0 just hit CRAN!

[sckott]

try again, admin should be fixed

[stefaniebutland]

Are you considering a blog post @mdsumner ?

[mdsumner]

Oh definitely!

[stefaniebutland]

Most excellent!
Guidelines are here: https://github.com/ropensci/roweb2#contributing-a-blog-post and worth checking as we've made some updates. Earliest publication date would be June 11, 2019. If that works for you, please submit a draft via pull request by June 4, 2019.

Once published, it will show up on your author page

Happy to answer any questions.

[sckott]

sorry to change recommendation @mdsumner - we recommend continuing to maintain your appveyor badge under your own acct

[mdsumner]

Oh that's all good, the change is only that appveyor creates a new link (another random suffix), but the badge hangs off github/ropensci:

https://ci.appveyor.com/api/projects/status/github/ropensci/tidync?branch=master&svg=true

https://ci.appveyor.com/project/mdsumner/tidync-nvwwt

[sckott]

@mdsumner so is that okay?

[mdsumner]

Yes, my only Todo now is the blog!