text classification bug fix & support ernie m #3184
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wawltor
reviewed
Sep 5, 2022
| @@ -376,7 +384,7 @@ python prune.py \ | |||
| * `per_device_eval_batch_size`：开发集评测过程批处理大小，请结合显存情况进行调整，若出现显存不足，请适当调低这一参数；默认为32。 | |||
| * `learning_rate`：训练最大学习率；默认为3e-5。 | |||
| * `num_train_epochs`: 训练轮次，使用早停法时可以选择100；默认为10。 | |||
| * `logging_steps`: 训练过程中日志打印的间隔steps数，默认5。 | |||
| * `logging_steps`: 训练过程中日志打印的间隔steps数，默认100。 | |||
There was a problem hiding this comment.
这里的logging steps对于大多数cpu用户来说，是不是太大了
There was a problem hiding this comment.
这是因为trainer里默认的参数就是100，但我在训练时命令行的参数还是设置--logging_steps 5
| 'token_type_ids'], batch['labels'] | ||
| logits = model(input_ids, token_type_ids) | ||
| label = batch.pop("labels") | ||
| logits = model(**batch) |
There was a problem hiding this comment.
为了可以适配erniem模型，erniem模型tokenizer得到的数据和模型输入都没有token_type_ids
| 'token_type_ids'], batch['labels'] | ||
| logits = model(input_ids, token_type_ids) | ||
| label = batch.pop("labels") | ||
| logits = model(**batch) |
| @@ -264,16 +267,15 @@ checkpoint/ | |||
|
|
|||
| * 如需恢复模型训练，则可以设置 `init_from_ckpt` ， 如 `init_from_ckpt=checkpoint/model_state.pdparams` 。 | |||
| * 如需训练英文文本分类任务，只需更换预训练模型参数 `model_name` 。英文训练任务推荐使用"ernie-2.0-base-en"，更多可选模型可参考[Transformer预训练模型](https://paddlenlp.readthedocs.io/zh/latest/model_zoo/index.html#transformer)。 | |||
|
|
|||
| * 英文和中文以外文本分类任务建议使用多语言预训练模型"ernie-m-base","ernie-m-large"， 多语言模型暂不支持文本分类模型部署，相关功能正在加速开发中。 | |||
There was a problem hiding this comment.
ernie-m版本目前在适配压缩API，主要问题是什么了？
There was a problem hiding this comment.
erniem模型tokenizer得到的数据和模型输入都没有token_type_ids，但压缩API训练的时候默认模型输入要有token_type_ids
https://github.com/PaddlePaddle/PaddleNLP/blob/develop/paddlenlp/trainer/trainer_compress.py#L386
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Others
PR changes
Others
Description
修复了文本分类中的bug，新增支持ernie-m模型（裁剪和部署部分暂不支持）