Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support lazily generate intermediate files(.wav) and clear them once the corresponding transcribe task is completed #50

Open
MaleicAcid opened this issue Jul 8, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@MaleicAcid
Copy link
Contributor

MaleicAcid commented Jul 8, 2024

Currently, it seems that intermediate files(.wav) are generated uniformly before the all transcription task starts, and cleared after all transcription tasks are completed.

This results in huge temporary disk usage.
In my example, the task of transcribing of 7.8GB MP4 audios (total 120 files) generates about 50GB intermediate files, which is unfriendly to the gpu server in the cloud environment.

@zh-plus
Copy link
Owner

zh-plus commented Jul 8, 2024

I'm working on refactoring the intermediate file generation process, which is currently buggy and poorly structured.
During the refactor:

  1. I will maintain the lazy generation strategy. The project's time bottleneck lies in transcription and translation, so a consumer-worker pattern is used to mitigate this. All intermediate files (preprocessing) should be generated before transcription since both require CPU usage.
  2. I will consider clearing these files as soon as transcription is completed.

I expect to finish the refactor by late July. Feel free to submit a PR if you want to help speed up the process.

@zh-plus zh-plus added the enhancement New feature or request label Jul 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants