[Numpy] Benchmark the backbone models + Some fixes + Always use python3 + Fix conversion tool #1292

sxjscience · 2020-08-10T11:35:27Z

Description

After this PR is merged, you can

Benchmarking the performance of backbone models in GluonNLP by running scripts/benchmarks/benchmark_gluonnlp.py
Always use python3 and pip3 as discussed with @szha and @leezu .
~~Test for the conversion scripts.~~ (Next PR, this one will revise the conversion scripts)

cc @dmlc/gluon-nlp-team

update Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py Create requirements.txt Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py Update requirements.txt update Update README.md Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py Update benchmark_hf.py fix fix Update test_models_bart.py Update test_models_bart.py Update bart.py update Update __init__.py Update electra.py update update Update convert_bert_from_tf_hub.sh update Update unittests.yml fix conversion update fix bert conversion update fix fix Update __init__.py fix bug fix css Update benchmark_utils.py Update benchmark_utils.py update update Update misc.py Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py no multiprocessing Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py fix bug Update benchmark_utils.py Update benchmark_utils.py try to use mxnet profiler Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py fix update Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py fix Update benchmark_utils.py Update bart.py Update bart.py fix fix Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py Update benchmark_gluonnlp.py Update benchmark_gluonnlp.py Update benchmark_gluonnlp.py Update benchmark_utils.py Update benchmark_utils.py Update benchmark_utils.py Update README.md

codecov · 2020-08-10T12:24:31Z

Codecov Report

Merging #1292 into master will increase coverage by 0.74%.
The diff coverage is 61.81%.

@@            Coverage Diff             @@
##           master    #1292      +/-   ##
==========================================
+ Coverage   83.62%   84.36%   +0.74%     
==========================================
  Files          42       42              
  Lines        6382     6397      +15     
==========================================
+ Hits         5337     5397      +60     
+ Misses       1045     1000      -45

Impacted Files	Coverage Δ
setup.py	`0.00% <ø> (ø)`
src/gluonnlp/layers.py	`86.78% <ø> (ø)`
src/gluonnlp/models/albert.py	`95.47% <ø> (ø)`
src/gluonnlp/models/bert.py	`94.86% <ø> (ø)`
src/gluonnlp/models/electra.py	`76.45% <15.38%> (-0.65%)`	⬇️
src/gluonnlp/models/bart.py	`86.25% <50.00%> (+1.08%)`	⬆️
src/gluonnlp/utils/misc.py	`51.89% <93.75%> (+8.49%)`	⬆️
src/gluonnlp/models/__init__.py	`96.66% <100.00%> (+0.11%)`	⬆️
src/gluonnlp/op.py	`95.08% <100.00%> (+35.08%)`	⬆️
src/gluonnlp/models/transformer.py	`98.94% <0.00%> (+0.42%)`	⬆️
... and 3 more

scripts/conversion_toolkits/README.md

scripts/pretraining/pretraining_utils.py

sxjscience · 2020-08-13T04:33:40Z

Initial numbers obtained via the benchmark:

See the shared google sheet: https://docs.google.com/spreadsheets/d/1ohQ2P4PNDxciz1_gWJaGzD-LeXsvX5MSe-qMoqnRaxc/edit?usp=sharing

zheyuye

Looks good overall

sxjscience added 2 commits August 10, 2020 04:30

Update benchmark_utils.py

1955197

sxjscience changed the title ~~[WIP] Benchmark the Backbone Models + Some Fixes~~ [WIP] Benchmark the Backbone Models + Fixes + Always use python3 Aug 10, 2020

sxjscience added 2 commits August 10, 2020 04:38

Update benchmark_utils.py

626739d

Update requirements.txt

1451f03

sxjscience changed the title ~~[WIP] Benchmark the Backbone Models + Fixes + Always use python3~~ [WIP] Benchmark the backbone models + Some fixes + Always use python3 + Test conversion tool Aug 10, 2020

sxjscience added 2 commits August 10, 2020 04:48

Update benchmark_utils.py

8215df6

Update benchmark_utils.py

15272f1

sxjscience changed the title ~~[WIP] Benchmark the backbone models + Some fixes + Always use python3 + Test conversion tool~~ [Numpy][WIP] Benchmark the backbone models + Some fixes + Always use python3 + Test conversion tool Aug 10, 2020

sxjscience added 8 commits August 10, 2020 04:54

Update benchmark_utils.py

1cf5c7b

Update benchmark_utils.py

37f224f

Update benchmark_utils.py

9e0f631

Update benchmark_utils.py

d9daf58

debug

9735edb

Update benchmark_utils.py

f0ae933

Update benchmark_gluonnlp.py

45c58b6

Update benchmark_gluonnlp.py

44f9c8b

zheyuye reviewed Aug 10, 2020

View reviewed changes

scripts/conversion_toolkits/README.md Outdated Show resolved Hide resolved

scripts/pretraining/pretraining_utils.py Outdated Show resolved Hide resolved

scripts/pretraining/pretraining_utils.py Show resolved Hide resolved

sxjscience and others added 11 commits August 10, 2020 08:49

Update benchmark_utils.py

fdd9df5

Update pretraining_utils.py

13db82f

Update benchmark_utils.py

a43e65b

update

b1f5955

Update benchmark_utils.py

9bbc581

Update benchmark_utils.py

9238d56

fix convert

93bd659

tiny fix

d6f3fc4

python3

e097c3b

fix

d651730

lower tolerance for albert large and xlarge

edd6655

sxjscience added 13 commits August 12, 2020 12:53

update

42765c3

Update benchmark_gluonnlp.py

8d8dae6

Update benchmark_gluonnlp.py

3db99e4

Update benchmark_gluonnlp.py

50020bc

Update benchmark_gluonnlp.py

1674df3

Update README.md

525c5eb

update

0f69ff4

Update benchmark_hf.py

fd5209c

Update benchmark_hf.py

5bc2a9c

Update requirements.txt

b38fe6d

Update benchmark_hf.py

f178a8f

Merge remote-tracking branch 'ZheyuYe/conversion_tf2' into benchmark

03cb027

Delete conversion_tool_test.yml

f952613

szha changed the base branch from numpy to master August 13, 2020 02:29

sxjscience added 11 commits August 13, 2020 10:36

Update README.md

4426295

Update README.md

52192a1

Update README.md

af617d7

move python --> python3

ce6a989

try to fix test

348221d

fix test case

c88b5b2

add test cases

7445798

Update README.md

28bc30e

update

c939cfd

update logging config

099be53

fix logging config

3cbf6d7

sxjscience changed the title ~~[Numpy][WIP] Benchmark the backbone models + Some fixes + Always use python3 + Test conversion tool~~ [Numpy] Benchmark the backbone models + Some fixes + Always use python3 + Test conversion tool Aug 14, 2020

sxjscience changed the title ~~[Numpy] Benchmark the backbone models + Some fixes + Always use python3 + Test conversion tool~~ [Numpy] Benchmark the backbone models + Some fixes + Always use python3 + Fix conversion tool Aug 14, 2020

zheyuye approved these changes Aug 14, 2020

View reviewed changes

zheyuye merged commit 32e87d4 into dmlc:master Aug 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Numpy] Benchmark the backbone models + Some fixes + Always use python3 + Fix conversion tool #1292

[Numpy] Benchmark the backbone models + Some fixes + Always use python3 + Fix conversion tool #1292

sxjscience commented Aug 10, 2020 •

edited

Loading

codecov bot commented Aug 10, 2020 •

edited

Loading

sxjscience commented Aug 13, 2020

zheyuye left a comment

[Numpy] Benchmark the backbone models + Some fixes + Always use python3 + Fix conversion tool #1292

[Numpy] Benchmark the backbone models + Some fixes + Always use python3 + Fix conversion tool #1292

Conversation

sxjscience commented Aug 10, 2020 • edited Loading

Description

codecov bot commented Aug 10, 2020 • edited Loading

Codecov Report

sxjscience commented Aug 13, 2020

zheyuye left a comment

Choose a reason for hiding this comment

sxjscience commented Aug 10, 2020 •

edited

Loading

codecov bot commented Aug 10, 2020 •

edited

Loading