Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

split: allow --split-max-size option #6259

Open
phymbert opened this issue Mar 23, 2024 · 1 comment · Fixed by #6343
Open

split: allow --split-max-size option #6259

phymbert opened this issue Mar 23, 2024 · 1 comment · Fixed by #6343
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed split GGUF split model sharding

Comments

@phymbert
Copy link
Collaborator

Motivation

we support --split-max-tensors since:

As mentionned by @Artefact2 in this comment:

allowing to split by file size would be more intuitive (and usually more appropriate since file size is usually the limiting factor, eg 4G for FAT or 50G for HF)

Proposition:

Introduce --split-max-size N(M|G) split strategy to split files in file with a max size of N Megabytes or Gigabytes.
As it is not possible to have less than 1 tensor per GGUF, this size is a soft limit.

@phymbert phymbert added enhancement New feature or request split GGUF split model sharding labels Mar 23, 2024
@phymbert phymbert assigned phymbert and unassigned phymbert Mar 23, 2024
@phymbert phymbert added help wanted Extra attention is needed good first issue Good for newcomers labels Mar 23, 2024
@phymbert
Copy link
Collaborator Author

Tests are missing, and we need to show how to use the gguf-split feature, it should be integrated in the CI. This is mandatory as shard feature is more and more used on HF.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed split GGUF split model sharding
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant