add a template to add missing tokenization test

[SaulLu]

What does this PR do?

In this PR I propose to add a cookie cutter template for tokenization tests.

It is a first version and could be useful to propose in a good first issue to users to add the missing tests to the tokenizers of the following models:

• Flaubert Add test suite for flaubert tokenizer #15137
• LED
• RemBert
• Splinter

and eventually for these tokenizers too (currently they just inherit from BertTokenizer and just re-define the attributes):

• MobileBert
• ConvBert
• Electra
• Longformer
• RetriBert

I plan to give a little more information on how to add the test to the good first issue ticket than is currently shown in the readme. But don't hesitate if you think it is better to say as much as possible in the current readme

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. I would love to have your input on this @LysandreJik , @sgugger , @patrickvonplaten or @patil-suraj 🤗

[SaulLu]

In this template currently I don't propose to implement any additional specific tests but please let me know if you think there are any generic tests - that are not in test_tokenization_common.py - that should be implemented for each tokenizer.

I'm pointing this out in particular because I've noticed that there are some tests that are shared by multiple tokenizers. This is for example the case for test_convert_token_and_id that is implemented in 18 different tests files (out of 63).

transformers/tests/camembert/test_tokenization_camembert.py

Lines 49 to 55 in 4975002

	def test_convert_token_and_id(self):
	"""Test ``_convert_token_to_id`` and ``_convert_id_to_token``."""
	token = "<pad>"
	token_id = 1
	self.assertEqual(self.get_tokenizer()._convert_token_to_id(token), token_id)
	self.assertEqual(self.get_tokenizer()._convert_id_to_token(token_id), token)

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sgugger]

Thanks for adding this!

[LysandreJik]

Thank you @SaulLu!