Caterpillar Mambo

New Features

• The corpus class has been put in a shape to become the default point of
  departure of most workflows. All core methods are now available for the
  corpus class, and have been implemented newly if necessary, e.g. show()
  and size()-method. The constructor method for a corpus object, the
  corpus() method, will now check whether the character vector with the corpus
  ID refers to an available corpus, whether all letters are upper case and
  issue informative warnings and error messages.
• The s_attributes()-method for corpus objects has been reworked: It will decode
  binary files directly, without reliance on the corpus library functions, which is
  significantly faster.
• The Corpus reference class is now obsolete after the introduction of the
  S4 corpus class. To maintain the functionality not covered otherwise,
  new generics get_info and show_info have been introduced and defined
  for the corpus class.
• Methods available for the subcorpus class have been expanded so that this
  class can supersede the partition class: Methods newly available are
  cpos(), count(), p_attributes(), s_attributes() get_token_stream(),
  and size(). Technically, there is virtual slice-class, from which
  subcorpus inherits (methods called via callNextMethod()).
• A new subset()-method for the corpus and subcorpus classes to generate subcorpora
  (i.e. subcorpus objects) has been introduced. It outperforms the
  partition() method. The subset()-method for corpus and subcorpus objects
  will be the default way to work with non standard evaluation in a manner that
  feels "R-ish" (#40).
• The zoom()-method that has been introduced experimentally has
  been dropped again in favor of the subset()-method to get subcorpus objects
  from corpus and subcorpus objects. A set of experimental methods for an
  initial check of the feasibility of a non-standard evaluation approach to
  the generation of subcorpora has been dropped (methods $, ==, !=,
  zoom for corpus-class).
• To facilitate the transition from the partition class (inheriting from
  the textstat class) to the subcorpus class (inheriting from the textstat
  class), there is a new coerce()-method to turn a partition object into
  a subcorpus object.
• A new remote_corpus-class is the basis for accessing remote
  corpora. A remote_subcorpus can be derived from a remote_corpus. Methods
  available for remote corpora und subcorpora remain limited at this stage.
• Consolidation of the class system: For all the S4 classes in the package, multiple
  contains have been checked, and multiple contains have been removed.
• The subcorpus_bundle class now inherits from partition_bundle. This is not
  intended to be a long-term solution, but facilitates the implementation of new
  workflows based on the subcorpus class rather than the partition class.
• Calling the polmineR shiny app via polmineR did not have safeguards if
  the suggested packages shiny and shinythemes were not installed. Now
  there will be a conditional installation of the packages required for running
  the shiny app.
• The somewhat odd class CorpusOrSubcorpus has been removed. The ngrams-method
  now applies for corpus and subcorpus objects.
• The pipe operator of the magrittr package is imported now, and magrittr has moved
  from a suggested package to a required package.
• The label()-method, present for a while, is superseded by a edit()-method now.
  It will call a shiny gadget either using DataTables or Handsontable. The former
  Labels reference class has been turned into a S4 class, because the
  desired reference logic can also be achieved with a data.table in a slot of
  the labels class.
• The table-slot of the kwic class has been renamed as stat slot (a data.table),
  so that the kwic class can now inherit from the textstat class. The
  enrich()-method for objects of class kwic now includes a new argument
  extra that will add extra tokens to the left of the windows for concordances so
  that qualitative inspections for query hits can work with more context.
• The as.TermDocumentMatrix() and the as.DocumentTermMatrix()-methods are now
  also defined for kwic objects. They work exactly the same as for the context
  class. To avoid having to write new methods, a new neighborhood virtual class has
  been introduced. The aforementioned methods are defined for the virtual class and
  are available for context and kwic class objects.
• Added CQP functionality to count tab in shiny app, and to the dispersion tab.
• There is now a basic implementation of get_token_stream() for a partition_bundle
  object.
• The Cooccurrences()-method is now available for subcorpus-objects (#88).
• There is a new coerce method to turn a kwic-object into a context-object.
  The neighborhood virtual class could be discarded again, and a bug could be removed
  that left an enrich()-operation for kwic objects (argument p_attribute)
  ineffectual (#103).

Minor changes

• Added a new argument regex to the cpos()-method (for corpus objects), which
  will interpret argument query as a regular expression. This may be faster than
  taking query as an outright CQP query.
• The configure-script in the package that would adjust paths in the registry files
  for the corpora included in the package for documentation and testing purposes has
  been removed. Having switched to a temporary registry directory, it has lost
  its function.
• The version of the data.table package now required is 1.12.2, because previous
  versions did not allow adding columns to a new data.table.
• Implemented the possibility to use multiple queries in dispersion-method (#92).
• To keep up with the renaming of functions and arguments in the package, "sAttributes"
  and "pAttributes" in the polmineR shiny app have been renamed ("s_attributes",
  and "p_attributes", respectively).
• The shiny app module for kwic output will not show p_attribute and positivelist
  by default.
• The format()-method is used to create proper output in the cooccurrences of the
  shiny app.
• User names that include non-ASCII characters were a persistent problem on Windows
  machines (#66). The solution now is to check for non-ASCII characters in the path
  to the data directory, and to use the "old" short DOS path if necessary. The worker is
  a modified registry()-function.
• The ordering of the table for ll-method had been somewhat mixed up, which is repaired
  now. Tokens with NA values for the ll-test will show up at the end of the table.
• The registry_move()-function, used only internally at this stage, is exported now
  so that it can be used by other packages.
• The return value of the get_token_stream()-method for regions objects was a
  data.table. The behavior is now in line with the other get_token_stream() methods
• The tempcorpus()-method and the tempcorpus class have been removed from the package,
  having become utterly deprecated.
• The summary()-method for partition-class objects has been turned into a method
  for the count-class, to eliminate an inconsistency. The example of a workflow has been
  moved to the documentation object for the count-class.
• The browse()-method has not proven to be useful and has been removed from the package.
  A new browse()-function is introduced to throw a warning, if browse should be
  called nevertheless.
• A refactoring of the split()-method for partition-objects improved the readability
  of the code, but the performance gain is minimal.
• A new kwic_bundle-class has been introduced, a list of kwic objects can be turned
  into this new class using as.bundle.
• The context()-method will now take again as input character vectors for the arguments
  left and right to expand to the left and right boundaries of the designated
  region (#87).
• Rework of the way messages are printed to make it easy to implement notifications in
  the shiny environment.
• Default highlighting when a positivelist is supplied has been removed from the
  kwic()-method. This ensures that subsequent highlighting operations can assign
  new colors (#38).
• Implemented feature request for dispersion() that results are reported for all
  values of structural attributes, including those with zero matches. (#104)
• Performance improved for the cpos-method for matrix which unfolds a matrix with regions
  of corpus positions, useful for operations that require many calls.
• The count-method for partition_bundle has been reworked and is much faster and more
  memory efficient.
• as.TermDocumentMatrix() for partition_bundle optimized to work efficiently
  with large corpora.
• Introduction of a context,matrix-method to have a unified auxiliary function
  to create contexts.
• The as.corpusEnc()-function uses the localeToCharset()-function from the utils
  package to determine the charset of input strings. On RStudio Server, we have seen
  cases when the return value is NA. Then it will be assumed that the locale is UTF-8.
• Functionality to highlight terms in kwic display has been restored for the shiny app.

Bug fixes

• Removed a bug in the context()/kwic() method that led to superfluous words in the
  right context.
• Removed a bug that occurred with the as.data.frame()-method for kwic-objects
  when no metadata were added.
• The count()-method for partition_bundle-objects did not perform iconv() if
  necessary - this has been corrected.
• Indexing the concordances of a kwic object did not reduce the cpos table
  concurringly. This has been corrected.
• The as.speeches()-method failed to handle situations correctly, when one speaker
  occurring in the corpus only contributed one single region to the entire corpus (#86).
  This behavior has been debugged.
• Counting over a partition_bundle started to throw a warning that an argument arrives
  at the cpos()-method that is not used. The cause for the warning message is removed,
  an additional unit test has been introduced to recognize issues with the
  count-method (#90).
• The kwic()-method threw an error when trimming the matches by using a positivelist
  or a stoplist resulted in no remaining matches. The method will now return a NULL
  object and keep issuing a warning if no matches remain after filtering (#91).
• Chaining subsetting calls on a corpus/subcorpus omitted filling the s_attribute slot
  of the subcorpus object, resulting in false results when counting over
  subcorpora. Fixed.
• Started to remove bugs in the shiny app: kwic starts to work again (bug: slot table
  has been replaced by stat).
• The part of the shiny app for dispersions did not work at all - has been repaired,
  exposing more functionality of dispersion() (#62).
• In the as.speeches()-method, the argument verbose was not used (#64) - this had
  been addressed when solving issue #86.
• Telling messages when sending out emails - on success and error - have been added (#61).
• A shortcoming in coerce method to turn a subcorpus into a String was removed:
  A semicolon was not recognized as a punctuation mark. This makes decoding subcorpora
  as Annotation more robust. The respective unit test has been updated.
• Calling read() on a kwic object works again (#84).
• Checks for the as.VCorpus() method that failed are now ok (#77). The reason was
  that get_token_stream() assumed implicitly that a p-attribute "pos" is present,
  which is not the case for the REUTERS test corpus.
• A minor bug in the s_attributes-method was removed that would make retrieving the
  metadata for the first strucs (index 0) of a s-attribute impossible.
• Fixed an issue for as.DocumentTermMatrix that started to occur with the introduction
  of the subcorpus_bundle class (#100).
• Removed a bug in the kwic-method for character that prevented using different values for
  right and left context (#101).
• Removed a bug that occurred when using as.DocumentTermMatrix() on a corpus stated
  by corpus ID / length-one character vector (#105).
• Removed a bug from the kwic,character-method, and the context,corpus-method that
  would result in odd behavior when either the left or right context is 0.
• An endemic encoding issue for full text output on Windows machines (latin1 encoding)
  has been solved by replacing internally markdown::markdownToHTML by a direct
  call to markdown::renderMarkdown. On this occasion, some overhead preparing
  fulltext output has been removed.
• A bug that prevented getting extra left and right context for kwic objects has
  been removed (#102).
• The as.TermDocumentMatrix()-method for neighborhood-objects returned a
  DocumentTermMatrix (unintendedly), this bug is removed now.

Documentation

• Extended documentation for pmi()-method and t_test()-method.
• New s_attributes()-method for corpus-class.
• The documentation for the corpus-class has been rewritten entirely, and the
  documentation for the remote_corpus-class has been integrated, whereas methods
  applicable to the remote_corpous-class were integrated into the documentation
  objects for the respective methods.
• The documentation for the get_token_stream()-method has been reworked and expanded
  thoroughly (#65). On this occasion, test coverage for the method has been improved
  significantly. (Everything is tested now apart from parallelization.)