Different results for different lenght of input

[Ascisel]

I'm trying to calculate bertscore for huge corpus, where I'm putting two lists of same length with corresponding texts. I was getting very strange results when changing only the length of input lists. Do you have any ideas why this is happening and how to deal with it?

df = pd.read_json(file_name)

texts1 = df["text1"][:50]
texts2 = df["text2"][:50]
texts3 = df["text1"][:100]
texts4 = df["text2"][:100]

bertscorer = BERTScore(model_name_or_path="roberta-base", truncation=True, batch_size=512)

print(bertscorer(texts1, texts2)["f1"][:4])
tensor([0.8560, 0.799, 0.8931, 0.8693])

print(bertscorer(texts3, texts4)["f1"][:4])
tensor([0.982, 0.6341, 0.9532, 0.9186])