Date format differs by Station

[mdwhitby]

Hello,

Great package, almost everything I need!

I have found a problem that appears to have happened with a recent update I did (not sure if it was tidyverse or Rnoaa).

When I download data from different ISD stations I get 2 different date formats. One is formated as a character, but the other comes in as a date format. An example is below. I used lapply to download stations by year. data[[2]] date is a chr, while data[[1]] it is a date.

I think this might have something to do with reading or converting the data to a tibble where it is autoformating.

data[[2]][1:10,1:5]

A tibble: 10 × 5

total_chars usaf_station wban_station date time

1 0126 720308 04992 20160101 0015
2 0126 720308 04992 20160101 0035
3 0185 720308 04992 20160101 0055
4 0185 720308 04992 20160101 0115
5 0126 720308 04992 20160101 0135
6 0185 720308 04992 20160101 0155
7 0166 720308 04992 20160101 0215
8 0185 720308 04992 20160101 0235
9 0126 720308 04992 20160101 0255
10 0137 720308 04992 20160101 0315

data[[1]][1:10,1:5]

A tibble: 10 × 5

total_chars usaf_station wban_station date time

1 126 720308 04992 2015-01-01 0015
2 139 720308 04992 2015-01-01 0035
3 126 720308 04992 2015-01-01 0055
4 126 720308 04992 2015-01-01 0115
5 126 720308 04992 2015-01-01 0135
6 126 720308 04992 2015-01-01 0155
7 137 720308 04992 2015-01-01 0215
8 126 720308 04992 2015-01-01 0235
9 137 720308 04992 2015-01-01 0255
10 126 720308 04992 2015-01-01 0315

sessionInfo()
R version 3.3.2 (2016-10-31)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C LC_TIME=English_United States.1252

attached base packages:
[1] parallel stats graphics grDevices utils datasets methods base

other attached packages:
[1] lubridate_1.6.0 rnoaa_0.6.5 dplyr_0.5.0 purrr_0.2.2 readr_1.0.0
[6] tidyr_0.6.0 tibble_1.2 ggplot2_2.1.0 tidyverse_1.0.0 RevoUtilsMath_10.0.0

loaded via a namespace (and not attached):
[1] Rcpp_0.12.7 xml2_1.0.0 magrittr_1.5 rappdirs_0.3.1 munsell_0.4.3 colorspace_1.2-7 R6_2.2.0
[8] stringr_1.1.0 httr_1.2.1 plyr_1.8.4 tools_3.3.2 grid_3.3.2 gtable_0.2.0 DBI_0.5-1
[15] lazyeval_0.2.0 assertthat_0.1 gridExtra_2.2.1 curl_2.2 mime_0.5 stringi_1.1.2 RevoUtils_10.0.2
[22] scales_0.4.0 XML_3.98-1.4 jsonlite_1.1 foreign_0.8-67

[sckott]

thank @mdwhitby - please share what function you are referring to and example code that gives you the two different results

[mdwhitby]

I download the data with the isd() function. It actually appears to be the same station, but different years (720308-04992 in 2015 and 2016). The relevant output should be in the printed header I included.

[sckott]

First thing is: update rnoaa to the latest version on CRAN v0.6.6 https://cran.rstudio.com/web/packages/rnoaa/ then try again and tell me what you get.

[mdwhitby]

updating did not seem to help. Looking through the source I speculate that it is actually a problem with the isdparser::isd_parse() function. on line 59 you "use tibble::as_data_frame(df)". I think tibble is automatically formating columns, and for some reason the date column is changed only occasionaly. is there a way to force all columns to be character? Or does it have to be a tibble? Personally, since I am pulling 3 years of data from 30+ stations i am not opposed to having them as data.tables. This is the first 2 list elements returned from rnoaa::isd() in an apply() usaf_station [1] "720308" wban_station [1] "04992"

class(data1[[1]]$date) #2015 data from station

[1] "Date"

class(data1[[2]]$date) #2016 data from station

[1] "character" Michael Whitby [email protected] 609-923-0973

On Mon, Mar 6, 2017 at 3:38 PM, Scott Chamberlain ***@***.***> wrote: First thing is: update rnoaa to the latest version on CRAN v0.6.6

https://cran.rstudio.com/web/packages/rnoaa/ then try again and tell me what you get.

…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[sckott]

@mdwhitby thanks for the update.

are you sure the above report is using the latest rnoaa? i only ask because i'm using the exact same version of rnoaa on CRAN and I don't get the problem you're reporting - i have tibble v1.2 as well so that's not different

I haven't seen all your code - it's possible you're doing something that's affecting the output and/or perhaps you have options that load in your R session that are coming into play here

[mdwhitby]

I think I finally tracked it down. When I originally coded I used "regular" R. This used rnoaa 0.6.6 and cached columns in character formats. I then switched to the MRAN version (for speed in doing a lot of data), it uses rnoaa 0.6.5 and chached any new files with the Date format. By deleteing the cache and not switching between versions the columns remain constaint. You may consider having a cache folder for each version. Michael Whitby [email protected] 609-923-0973

…

On Tue, Mar 7, 2017 at 12:56 PM, Scott Chamberlain ***@***.*** > wrote: @mdwhitby <https://github.com/mdwhitby> thanks for the update. are you sure the above report is using the latest rnoaa? i only ask because i'm using the exact same version of rnoaa on CRAN and I don't get the problem you're reporting - i have tibble v1.2 as well so that's not different I haven't seen all your code - it's possible you're doing something that's affecting the output and/or perhaps you have options that load in your R session that are coming into play here — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#205 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AXdHZvAgTCoLMVy3nT_2OgYipZAT58iPks5rjahLgaJpZM4MUeWU> .

[sckott]

glad you figured it out.

You may consider having a cache folder for each version.

for each version of what?

[mdwhitby]

any new update of rnoaa - say /rnoaa/v066 and when updated /rnoaa/v067. That would force it to use the cache from that version or rnoaa and make sure the cleaned data is in the same format. Michael Whitby [email protected] 609-923-0973

…

On Tue, Mar 7, 2017 at 2:36 PM, Scott Chamberlain ***@***.***> wrote: glad you figured it out. You may consider having a cache folder for each version. for each version of what? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#205 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AXdHZlxtsPNx04qDWYGhthMK2cpiUKavks5rjb_KgaJpZM4MUeWU> .

[sckott]

Hmm, maybe - i might rather just make sure the docs are clear on this - and maybe i could have a message to the user about checking that their cache