Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kunlunxin] fix tacotron2 running error and add 1x1 & 2x8 config #346

Merged
merged 4 commits into from
Dec 6, 2023
Merged

[kunlunxin] fix tacotron2 running error and add 1x1 & 2x8 config #346

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
83560bd
[kunlunxin] fix tacotron2 running error and add 1x1 & 2x8 config
jamesruio Dec 1, 2023
03cc616
[kunlunxin] modify tacotron2 test_config
jamesruio Dec 1, 2023
a1300e7
[kunlunxin] update tacotron2 readme
jamesruio Dec 4, 2023
2f2bd6a
[kunlunxin] modify tacotron2 torch.load()
jamesruio Dec 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion training/benchmarks/tacotron2/pytorch/model/data/data_function.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ def get_mel(self, filename):
melspec = self.stft.mel_spectrogram(audio_norm)
melspec = torch.squeeze(melspec, 0)
else:
melspec = torch.load(filename)
melspec = torch.load(filename, map_location='cpu')
yuzhou03 marked this conversation as resolved.
Show resolved Hide resolved
assert melspec.size(0) == self.stft.n_mel_channels, (
'Mel dimension mismatch: given {}, expected {}'.format(
melspec.size(0), self.stft.n_mel_channels))
Expand Down
29 changes: 24 additions & 5 deletions training/kunlunxin/tacotron2-pytorch/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,31 @@


### 运行情况
| 训练资源 | 配置文件 | 运行时长(s) | 目标val_loss | 收敛val_loss | epoch数 | 性能(samples/s) |
| -------- | --------------- | ----------- | ------------ | ------------ | ------- | --------------- |
| 单机1卡 | config_R300x1x1 | / | | / | | |
| 单机8卡 | config_R300x1x8 | | 0.4852(fp32) | 0.4271(fp32) | 1235 | |
| 两机8卡 | config_R300x2x8 | / | | / | | |
* 通用指标

| 指标名称 | 指标值 | 特殊说明 |
| -------------- | ----------------------- | ------------------------------------------- |
| 任务类别 | SpeechSynthesis | |
| 模型 | tacotron2 | |
| 数据集 | LJSpeech | |
| 数据精度 | precision,见“性能指标” | 可选fp32/amp/fp16/tf32 |
| 超参修改 | fix_hp,见“性能指标” | 跑满硬件设备评测吞吐量所需特殊超参 |
| 硬件设备简称 | Kunlunxin R300 | |
| 硬件存储使用 | mem,见“性能指标” | 通常称为“显存”,单位为GiB |
| 端到端时间 | e2e_time,见“性能指标” | 总时间+Perf初始化等时间 |
| 总吞吐量 | p_whole,见“性能指标” | 实际训练样本数除以总时间(performance_whole) |
| 训练吞吐量 | p_train,见“性能指标” | 不包含每个epoch末尾的评估部分耗时 |
| **计算吞吐量** | **p_core,见“性能指标”** | 不包含数据IO部分的耗时(p3>p2>p1) |
| 训练结果 | val_loss,见“性能指标” | 验证loss |
| 额外修改项 | 无 | |

* 性能指标

| 配置 | precision | fix_hp | e2e_time | p_whole | p_train | p_core | val_loss | mem |
| -------------------- | --------- | --------------- | -------- | ------- | ------- | ------ | -------- | --------- |
| R300单机8卡(1x8) | fp32 | bs=96, lr=0.001 | / | / | / | / | 0.4801 | 26.3/32.0 |
| R300双机8卡(2x8) | fp32 | bs=96, lr=0.001 | / | / | / | / | / | 25.0/32.0 |
| R300单机单卡(1x1) | fp32 | bs=128, lr=0.001 | / | / | / | / | / | 30.5/32.0 |
### 许可证

Apache 2.0 license。
9 changes: 9 additions & 0 deletions training/kunlunxin/tacotron2-pytorch/config/config_R300x1x1.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
from config_common import *

train_batch_size = 128
eval_batch_size = train_batch_size

warmup = 0.2
learning_rate = 1e-3

seed = 23333
2 changes: 1 addition & 1 deletion training/kunlunxin/tacotron2-pytorch/config/config_R300x1x8.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from config_common import *

train_batch_size = 48
train_batch_size = 96
eval_batch_size = train_batch_size

warmup = 0.2
Expand Down
9 changes: 9 additions & 0 deletions training/kunlunxin/tacotron2-pytorch/config/config_R300x2x8.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
from config_common import *

train_batch_size = 96
eval_batch_size = train_batch_size

warmup = 0.2
learning_rate = 1e-3

seed = 23333
4 changes: 4 additions & 0 deletions training/kunlunxin/tacotron2-pytorch/config/environment_variables.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
export XACC=1
export BKCL_PCIE_RING=1
export BKCL_TIMEOUT=1800
export XMLIR_D_XPU_L3_SIZE=10485760
3 changes: 2 additions & 1 deletion training/run_benchmarks/config/test_conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@
# "bert:pytorch:R300:1:8:1": "/raid/dataset/bert_large/train",
# "longformer:pytorch:R300:1:8:1": "/raid/dataset/longformer_train",
# "distilbert:pytorch:R300:1:8:1": "/raid/dataset/distilbert/",
# "swin_transformer:pytorch:R300:1:8:1": "/raid/dataset/ImageNet_1k_2012/"
# "swin_transformer:pytorch:R300:1:8:1": "/raid/dataset/ImageNet_1k_2012/",
# "tacotron2:pytorch:R300:1:8:1": "/raid/dataset/tacotron2/LJSpeech/"
}

Loading