[MODEL] BERT conversion scripts, SciBERT, BioBERT, ClinicalBERT #735

leezu · 2019-05-29T15:58:11Z

This refactors the tf Bert conversion scripts based on #732.

codecov · 2019-05-29T15:58:14Z

Codecov Report

❗ No coverage uploaded for pull request head (refactorbertconversioncode@ef74887). Click here to learn what that means.
The diff coverage is n/a.

codecov · 2019-05-29T15:58:14Z

Codecov Report

Merging #735 into master will increase coverage by 0.65%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #735      +/-   ##
==========================================
+ Coverage   89.95%   90.61%   +0.65%     
==========================================
  Files          64       64              
  Lines        6064     6064              
==========================================
+ Hits         5455     5495      +40     
+ Misses        609      569      -40

Impacted Files	Coverage Δ
src/gluonnlp/data/utils.py	`74.14% <ø> (ø)`	⬆️
src/gluonnlp/model/utils.py	`76.72% <100%> (ø)`	⬆️
src/gluonnlp/model/bert.py	`99.27% <100%> (ø)`	⬆️
src/gluonnlp/data/dataloader.py	`83.62% <0%> (-5.18%)`	⬇️
src/gluonnlp/data/stream.py	`89.61% <0%> (+0.54%)`	⬆️
src/gluonnlp/model/sequence_sampler.py	`92.22% <0%> (+15.9%)`	⬆️

leezu · 2019-05-29T15:59:02Z

@eric-haibin-lin this PR only touches code in the script folder. All other changes are due to #732 and can be ignored while reviewing this.

scripts/bert/index.rst

scripts/bert/compare_tf_gluon_model.py

mli · 2019-06-04T20:58:07Z

Job PR-735/4 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-735/4/index.html

scripts/bert/conversion_tools/convert_tf_model.py

mli · 2019-06-05T16:36:58Z

Job PR-735/5 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-735/5/index.html

Typically Optimizer parameters are not exported for Bert models (eg. https://github.com/google-research/bert). As this does not always hold, we have to handle the case that the parameters are still present (eg. https://github.com/allenai/scibert).

mli · 2019-06-06T21:06:17Z

Job PR-735/7 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-735/7/index.html

mli · 2019-06-07T09:28:58Z

Job PR-735/8 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-735/8/index.html

Confirmed that the scibert_scivocab_uncased loaded from PyTorch produces the same output as the tensorflow version (based on the compare_tf_gluon_model.py).

mli · 2019-06-07T21:47:43Z

Job PR-735/14 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-735/14/index.html

eric-haibin-lin

nice work!

scripts/bert/conversion_tools/convert_pytorch_model.py

scripts/bert/conversion_tools/infer_pytorch_gluon_parameter_name_mapping.py

src/gluonnlp/data/utils.py

mli · 2019-06-08T09:25:37Z

Job PR-735/15 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-735/15/index.html

mli · 2019-06-08T20:53:31Z

Job PR-735/16 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-735/16/index.html

mli · 2019-06-08T21:24:12Z

Job PR-735/17 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-735/17/index.html

leezu requested a review from eric-haibin-lin May 29, 2019 15:58

leezu requested a review from szha as a code owner May 29, 2019 15:58

eric-haibin-lin reviewed May 30, 2019

View reviewed changes

scripts/bert/index.rst Outdated Show resolved Hide resolved

scripts/bert/compare_tf_gluon_model.py Outdated Show resolved Hide resolved

eric-haibin-lin reviewed Jun 1, 2019

View reviewed changes

scripts/bert/compare_tf_gluon_model.py Show resolved Hide resolved

leezu force-pushed the refactorbertconversioncode branch 3 times, most recently from 2c7a7ba to 501189c Compare June 4, 2019 19:20

eric-haibin-lin reviewed Jun 4, 2019

View reviewed changes

scripts/bert/conversion_tools/convert_tf_model.py Outdated Show resolved Hide resolved

szha added the release focus Progress focus for release label Jun 5, 2019

leezu force-pushed the refactorbertconversioncode branch from 501189c to d3a9c68 Compare June 5, 2019 15:28

leezu added 7 commits June 6, 2019 18:50

Fix typos

d34285c

Refactor argparse

88466b2

Validate bert_config.json

6237048

Make use of flexible vocab in scripts/bert/convert_tf_model.py

a41bd81

Support non-standard datasets/vocabularies in compare_tf_gluon_model

cfeac29

Address comments

e1ef91c

leezu force-pushed the refactorbertconversioncode branch from d3a9c68 to 20950a7 Compare June 6, 2019 18:50

Fix lint

c32f538

leezu force-pushed the refactorbertconversioncode branch from 20950a7 to c32f538 Compare June 6, 2019 18:55

leezu added 2 commits June 7, 2019 07:33

Move input.txt

19ff84a

Fix vocabulary mismatch in compare_tf_gluon_model.py

59b2250

leezu added 3 commits June 7, 2019 13:40

Add SciBert

8cb5fc0

Add BioBert

0d0fec7

Add PyTorch Bert support

d724106

Confirmed that the scibert_scivocab_uncased loaded from PyTorch produces the same output as the tensorflow version (based on the compare_tf_gluon_model.py).

szha changed the title ~~Refactor tf bert conversion scripts based on flexible vocab~~ [MODEL] BERT conversion scripts, SciBERT, BioBERT, ClinicalBERT Jun 7, 2019

Fix lint

5614e5b

leezu force-pushed the refactorbertconversioncode branch from d229a41 to 5614e5b Compare June 7, 2019 20:11

Fix clinicalbert test case

cb8344b

eric-haibin-lin reviewed Jun 7, 2019

View reviewed changes

scripts/bert/conversion_tools/convert_pytorch_model.py Outdated Show resolved Hide resolved

scripts/bert/conversion_tools/infer_pytorch_gluon_parameter_name_mapping.py Outdated Show resolved Hide resolved

src/gluonnlp/data/utils.py Outdated Show resolved Hide resolved

Address comments

ff53a57

leezu and others added 2 commits June 8, 2019 19:46

Rename biobert and clinicalbert models

16222f4

Update bert.py

277ceba

eric-haibin-lin approved these changes Jun 8, 2019

View reviewed changes

szha approved these changes Jun 8, 2019

View reviewed changes

szha merged commit 847c415 into dmlc:master Jun 8, 2019

leezu deleted the refactorbertconversioncode branch June 9, 2019 10:18

This was referenced Jun 10, 2019

Port BioBERT to GluonNLP #714

Closed

Port ClinicalBERT to GluonNLP #709

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MODEL] BERT conversion scripts, SciBERT, BioBERT, ClinicalBERT #735

[MODEL] BERT conversion scripts, SciBERT, BioBERT, ClinicalBERT #735

leezu commented May 29, 2019

codecov bot commented May 29, 2019

codecov bot commented May 29, 2019 •

edited

Loading

leezu commented May 29, 2019

mli commented Jun 4, 2019

mli commented Jun 5, 2019

mli commented Jun 6, 2019

mli commented Jun 7, 2019

mli commented Jun 7, 2019

eric-haibin-lin left a comment

mli commented Jun 8, 2019

mli commented Jun 8, 2019

mli commented Jun 8, 2019

[MODEL] BERT conversion scripts, SciBERT, BioBERT, ClinicalBERT #735

[MODEL] BERT conversion scripts, SciBERT, BioBERT, ClinicalBERT #735

Conversation

leezu commented May 29, 2019

codecov bot commented May 29, 2019

Codecov Report

codecov bot commented May 29, 2019 • edited Loading

Codecov Report

leezu commented May 29, 2019

mli commented Jun 4, 2019

mli commented Jun 5, 2019

mli commented Jun 6, 2019

mli commented Jun 7, 2019

mli commented Jun 7, 2019

eric-haibin-lin left a comment

Choose a reason for hiding this comment

mli commented Jun 8, 2019

mli commented Jun 8, 2019

mli commented Jun 8, 2019

codecov bot commented May 29, 2019 •

edited

Loading