Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable HF nllb conversion #204

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Enable HF nllb conversion #204

wants to merge 4 commits into from

Conversation

francoishernandez
Copy link
Member

@francoishernandez francoishernandez commented Jan 30, 2025

NLLB support has been unclear since switching from OpenNMT to Eole.
This PR facilitates the conversion of official HF NLLB models (e.g. https://huggingface.co/facebook/nllb-200-distilled-1.3B).
This should also facilitate re-enabling support of pre-trained seq2seq models such as BART or T5.

That being said, the current structure of convert_HF needs to be reviewed to better support the encoder/decoder duality.
#156 was a first major step in making convert_HF, more modular, and #153 introduced the support of encoder keys, but now we need to meld all this into a more robust logic.
Also, we should probably define a better "HF settings deduction waterfall" in build_config_dict, but I'm not sure there is any centralized repository of all the possible values, as they are defined per model.

Note: HF tokenization was not tested yet, there might be some edge case to tackle with the prefix transform.
EDIT: HF tokenization patched in 3fd59e6

@francoishernandez francoishernandez changed the title [WIP] Enable HF nllb conversion Enable HF nllb conversion Jan 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant