New TruncateTextColumn to truncate the length of texts using the number of tokens or characters

[plaguss]

Description

This PR adds a new TruncateTextColumn step to truncate the text.

While developing pipelines that start from datasets with longer texts, the model can fail for longer texts than the max context length the model can handle. This step can be useful to avoid such errors:

• Truncating a row to a given number of tokens:

from distilabel.steps import TruncateTextColumn

trunc = TruncateTextColumn(
    tokenizer="meta-llama/Meta-Llama-3.1-70B-Instruct",
    max_length=4,
    column="text"
)

trunc.load()

result = next(
    trunc.process(
        [
            {"text": "This is a sample text that is longer than 10 characters"}
        ]
    )
)
# result
# [{'text': 'This is a sample'}]

• Truncating a row to a given number of characters

from distilabel.steps import TruncateTextColumn

trunc = TruncateTextColumn(max_length=10)

trunc.load()

result = next(
    trunc.process(
        [
            {"text": "This is a sample text that is longer than 10 characters"}
        ]
    )
)
# result
# [{'text': 'This is a '}]

[github-actions]

Documentation for this PR has been built. You can view it at: https://distilabel.argilla.io/pr-902/

[codspeed-hq]

CodSpeed Performance Report

Merging #902 will not alter performance

_{Comparing step-truncate (8dd2790) with develop (c8df5a9)}

Summary

✅ 1 untouched benchmarks

[gabrielmbmb]

Maybe TruncateTextColumn is a better name for this step?