Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark script + improvements and bug fixes #46

Merged
merged 23 commits into from
May 8, 2022
Merged

Conversation

juanmc2005
Copy link
Owner

This PR addresses issues #35 and #39

Changelog

  • Add diart.benchmark script for fast inference, evaluation and real-time latency estimation of the pipeline (issue Add batched mode for faster inference on pre-recorded conversations #35). Only for the standard pipeline for now, custom pipelines will have to modify some internals until this is improved.
  • Add GPU compatibility for diart.benchmark and diart.demo with --gpu
  • Add real-time latency estimation to FileAudioSource with the profile argument (issue Benchmarking operator #39)
  • Add ChunkLoader to centralize all file chunking
  • Add tqdm progress operator: observable.pipe(dops.progress()).subscribe(...)
  • Add PipelineConfig to encapsulate the configuration of the pipeline
  • Add --no-plot argument to diart.demo to skip plotting (as mentioned in Question about reproduce the result #36)
  • Make blocks in functional compatible with batched inference
  • Fix resolution bug in FrameWiseModel always using chunk duration from model training instead of the duration of the given chunk
  • Make models compatible with torch.Tensor and numpy.ndarray inputs (SlidingWindowFeature compatibility is kept but only for non-batched mode)
  • Add OnlineSpeakerTracking as a submodule of the pipeline including clustering and output reconstruction
  • Add from_file() method to OnlineSpeakerDiarization to run batched inference, as opposed to from_source() that runs online inference
  • Sources can now provide the number of chunks that they will emit (if known) via the length: Optional[int] property

Notes

  • Benchmarked performance on AMI test with latency=5: DER = 27.3
  • Benchmarked performance on VoxConverse test with latency=5: DER = 17.1

… progress operator. Add chunk number estimation without loading chunks
…sion to disappear. Improve benchmark script help
@juanmc2005 juanmc2005 added bug Something isn't working feature New feature or request labels May 4, 2022
@juanmc2005 juanmc2005 added this to the Version 0.3 milestone May 4, 2022
@juanmc2005 juanmc2005 self-assigned this May 4, 2022
@juanmc2005 juanmc2005 requested a review from hbredin May 4, 2022 16:14
@juanmc2005 juanmc2005 merged commit 7ecae25 into develop May 8, 2022
@juanmc2005 juanmc2005 deleted the feat/batched branch May 8, 2022 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants