Expand Support for Non-OpenAI Models in Token Trimming

[monilpat]

Feature Request

Is your feature request related to a problem? Please describe.

Currently, the system is limited to using TiktokenModel for token trimming, which restricts compatibility with non-OpenAI models. This limitation may hinder the integration of diverse models that could enhance performance and scalability.

Describe the solution you'd like

Implement a flexible token trimming mechanism that supports a variety of models beyond TiktokenModel. This could involve abstracting the token trimming logic to accommodate different model architectures and tokenization strategies.

Describe alternatives you've considered

Continuing with the current model-specific approach, but this would limit the flexibility and potential for optimization across different models.

Additional context

Expanding support for non-OpenAI models will improve the system's adaptability and allow for better optimization of token trimming processes. This aligns with the goal of enhancing algorithm efficiency and scalability.

Related Issues

None at the moment, but tracking this enhancement will facilitate discussions around implementation strategies.

[tcm390]

related: #1439

[tcm390]

@monilpat please kindly try the PR and see if it fixes your issue. #1605

[shakkernerd]

Implemented in #1605