Efficient workflow to decode sentences of corpus

[ablaette]

Getting a list of sentences would be possible by something like corpus() %>% split() %>% get_token_stream(). This may not be very efficient. A nice and efficient workflow might work as follows:

corpus("REUTERS") %>%
  regions(s_attribute = "s") %>%
  get_token_stream(split = TRUE)

Or, alternatively:

corpus("REUTERS") %>%
  segment(s_attribute = "s") %>%
  get_token_stream(split = TRUE)

[ablaette]

A new polmineR version on the dev branch (v0.8.5.9011) now includes an implementation of the first option that occurred to me. Generally speaking, the implementation is much faster than the original approach and performance is satisfactory. The bottleneck is a cut() call. No idea so far how I could speed this up. But as I said, the performance boost is significant already.

[ChristophLeonhardt]

This is a fine solution. However it does not yet work with subcorpora. It splits the entire corpus even if you do insert a subset() in the pipe before regions().

[ablaette]

Good point. I now implemented a regions() method for subcorpus objects. This is an example I played with.

x <- corpus("GERMAPARL_PARLACLARIN_III") %>% 
  subset(year = "2017") %>%
  regions(s_attribute = "s") %>%
  get_token_stream(split = TRUE)

Performance is ok, I think.