[FEATURE] Flexible vocabulary

[leezu]

Currently the gluonnlp.Vocab always assigns 0 as id of <unk> (#393). Further
reserved tokens are not exposed as attributes of the vocab (#572). This PR
addresses both issues.

Checklist

Essentials

• PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented

Changes

• Add token_to_idx argument to gluonnlp.Vocab which allows users to
  optionally specify the index to be associated to a token for a subset of
  tokens in the vocabulary. (Fixes gluonnlp.Vocab hard coded id of <unk> token #393)
• Add identifiers_to_tokens argument to gluonnlp.Vocab which allows users to
  optionally specify a Mapping, specifying identifiers for tokens that will be
  exposed as attributes of the vocabulary (Fixes [Vocab] Support special token registration  #572)

Comments

• The token_to_idx argument can only contain tokens that will actually be part
  of the vocabulary. On the one hand, a user may not know which tokens will be
  part of the vocabulary, eg. when specifying a maximum size argument. On the
  other hand, if the user does not know if a certain token will be part of the
  vocab or not, it seems unlikely that the user would like to pre-specify the
  index of that token. Thus this shouldn't pose any problems.

[codecov]

Codecov Report

❗ No coverage uploaded for pull request head (flexiblevocab@af1aeb0). Click here to learn what that means.
The diff coverage is n/a.

[codecov]

Codecov Report

Merging #732 into master will decrease coverage by 9.07%.
The diff coverage is 94.3%.

@@            Coverage Diff             @@
##           master     #732      +/-   ##
==========================================
- Coverage   90.32%   81.25%   -9.08%     
==========================================
  Files          64       64              
  Lines        6077     6064      -13     
==========================================
- Hits         5489     4927     -562     
- Misses        588     1137     +549

Impacted Files	Coverage Δ
src/gluonnlp/base.py	87.87% <100%> (ø)	⬆️
src/gluonnlp/vocab/bert.py	98.03% <100%> (+12.99%)	⬆️
src/gluonnlp/embedding/token_embedding.py	88.09% <72.22%> (-2.36%)	⬇️
src/gluonnlp/vocab/vocab.py	97.35% <97.56%> (-0.61%)	⬇️
src/gluonnlp/model/train/cache.py	26.19% <0%> (-71.43%)	⬇️
src/gluonnlp/model/train/language_model.py	42.04% <0%> (-55.12%)	⬇️
src/gluonnlp/embedding/evaluation.py	41.8% <0%> (-54.1%)	⬇️
src/gluonnlp/data/batchify/language_model.py	44.03% <0%> (-52.3%)	⬇️
src/gluonnlp/model/translation.py	20.63% <0%> (-50.8%)	⬇️
src/gluonnlp/model/language_model.py	50.38% <0%> (-49.62%)	⬇️
... and 17 more

[mli]

Job PR-732/1 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-732/1/index.html

[mli]

Job PR-732/5 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-732/5/index.html

[mli]

Job PR-732/6 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-732/6/index.html

[eric-haibin-lin]

@rich-junwang you might be interested in this

[rich-junwang]

@rich-junwang you might be interested in this

Thanks, this is helpful!

[leezu]

@rich-junwang feel free to review the code and leave comments

[mli]

Job PR-732/7 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-732/7/index.html

[leezu]

@kenjewu @eric-haibin-lin please take a look at the refactored BERTVocab.

The only user-visible change is that the static default UNKNOWN_TOKEN, etc attributes are moved away from the BERTVocab class. Conceptually it doesn't seem to make sense to expose them as part of the object, given that the object may not be using the default values. If you think this should be preserved, I can add it back.

[szha]

@eric-haibin-lin @leezu if the attributes can be flexibly registered, do we still need BERTVocab?

[leezu]

Only for backwards compatibility; I'm fine with breaking it, but we'll have to replace the Bert artifacts on S3

[eric-haibin-lin]

@szha the BERTVocab is no long necessary. But I suggest we keep it in this version for backward compatibility.
@leezu just to double check, this changes doesn't break backward compatibility and users can still use their existing serialized BERTVocab file, right?

[leezu]

@szha last commit introduces the registration of special tokens via keyword arguments as discussed. For deprecation of padding, bos, eos_tokens I'll open a separate PR.

You may want to take a look at the last commit @eric-haibin-lin

[mli]

Job PR-732/32 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-732/32/index.html