fct_lump prop to consider NA

[camdenhu]

Currently, prop_n is calculated as the count divided by sum of counts, which excludes NA.

If prop_n is calculated as count divided by length(f), the proportion would take into account NA values and reflect the true proportion of the level, not the proportion of the level among non-NA values.

Thanks for the great work.

[hadley]

Can you please provide a reproducible example illustrating where this is a problem?

[camdenhu]

f <- factor(c(rep("a",5), "b", rep(NA, 94)))
fct_lump(f, prop = 0.02) keeps "b" as a level, even though it occurs in just 1 of 100 values, because it occurs in 1 of 6 non-NA values.

[hadley]

The problem is that in your example NA isn't actually a level. But if you make it one it illustrates a different bug (this time in levels<-)

f <- factor(c(rep("a", 5), "b", rep(NA, 94)), exclude = NULL)
fct_lump(f, prop = 0.02)
#>   [1] a     a     a     a     a     Other <NA>  <NA>  <NA>  <NA>  <NA> 
#>  [12] <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA> 
#>  [23] <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA> 
#>  [34] <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA> 
#>  [45] <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA> 
#>  [56] <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA> 
#>  [67] <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA> 
#>  [78] <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA> 
#>  [89] <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA>  <NA> 
#> [100] <NA> 
#> Levels: a Other

[camdenhu]

Yes I have realized that there are bugs with NA as a level. The problem goes away when it is replaced by an explicit string. My point is that prop should be considered as a proportion of the length of the vector, not just those with associated levels.