_convert_id_to_tokens for XLNet not working

[chris-boson]

text = self.tokenizer.convert_ids_to_tokens(token_list)
  File "/home/lambda/repos/pytorch-transformers/pytorch_transformers/tokenization_utils.py", line 444, in convert_ids_to_tokens
    tokens.append(self._convert_id_to_token(index))
  File "/home/lambda/repos/pytorch-transformers/pytorch_transformers/tokenization_xlnet.py", line 170, in _convert_id_to_token
    token = self.sp_model.IdToPiece(index)
  File "/home/lambda/python-envs/research/lib/python3.6/site-packages/sentencepiece.py", line 187, in IdToPiece
    return _sentencepiece.SentencePieceProcessor_IdToPiece(self, id)
TypeError: in method 'SentencePieceProcessor_IdToPiece', argument 2 of type 'int'

I find that if I explicitly convert ids to integers it works fine. In tokenization_xlnet.py

def _convert_id_to_token(self, index, return_unicode=True):
    """Converts an index (integer) in a token (string/unicode) using the vocab."""
    token = self.sp_model.IdToPiece(int(index))
    if six.PY2 and return_unicode and isinstance(token, str):
        token = token.decode('utf-8')
    return token

[thomwolf]

Which command can we use to reproduce the behavior?

[chris-boson]

Upon further testing, looks like this tokenizer doesn't like numpy arrays, the other ones seem to be fine

import numpy as np
from pytorch_transformers import XLNetTokenizer, TransfoXLTokenizer, BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
print(tokenizer.convert_ids_to_tokens(np.array([3, 4, 6, 2356])))
tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
print(tokenizer.convert_ids_to_tokens(np.array([3, 4, 6, 2356])))
tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')
print(tokenizer.convert_ids_to_tokens(np.array([3, 4, 6, 2356]).tolist()))
print(tokenizer.convert_ids_to_tokens(np.array([3, 4, 6, 2356]))) # Above error

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.