batch_encode_plus doesn't work correctly

[tempdeltavalue]

code here:
https://github.com/tempdeltavalue/temp_l/blob/main/finetune_seq2seq.ipynb

https://discuss.huggingface.co/t/repetitive-words-in-model-output/132085/2

[tempdeltavalue]

same with tokenizer() batch encoding

[jonvet]

In your notebook you initialise the tokenizer as follows

tokenizer = AutoTokenizer.from_pretrained("microsoft/DialoGPT-medium", padding_side='left')
tokenizer.pad_token = tokenizer.eos_token

so you're padding with tokens from the left.
the reason why you're getting more pad tokens for the same input sequence when you encode X[0:99] than when you encode X[0:3] is that some sequence in X[3:99] is longer than the longest sequence in X[0:3], and it will make all encoded sequences the same length (due to padding). where is this going wrong in your opinion?