Skip to content
This repository has been archived by the owner on Dec 1, 2021. It is now read-only.

Fix a stability about multiple measurements #1001

Merged
merged 10 commits into from
Apr 30, 2020

Conversation

tk26eng
Copy link
Contributor

@tk26eng tk26eng commented Apr 17, 2020

What this patch does to fix the issue.

  • This patch change the order to run the functions for multiple times because the continuous calling same function might estimate the consumed time lesser than actual situation
  • This patch add the standard deviation value to know more statistics information about stabilities of measurement results

Link to any relevant issues or pull requests.

Fix #929

@blueoil-butler blueoil-butler bot added the CI: auto-run Run CI automatically label Apr 17, 2020
@bo-code-review-bot
Copy link

This PR needs Approvals as follows.

  • Ownership Approval for / from iizukak, tkng, ruimashita
  • Readability Approval for Python from tkng, tsawada, tfujiwar

Please choose reviewers and requet reviews!

Click to see how to approve each reviews

You can approve this PR by triggered comments as follows.

  • Approve all reviews requested to you (readability and ownership) and LGTM review
    Approval, LGTM

  • Approve all ownership reviews
    Ownership Approval or OA

  • Approve all readability reviews
    Readability Approval or RA

  • Approve specified review targets

    • Example of Ownership Reviewer of /: Ownership Approval for / or OA for /
    • Example of Readability Reviewer of Python: Readability Approval for Python or RA for Python
  • Approve LGTM review
    LGTM

See all trigger comments

Please replace [Target] to review target

  • Ownership Approval
    • Ownership Approval for [Target]
    • OA for [Target]
    • Ownership Approval
    • OA
    • Approval
  • Readability Approval
    • Readability Approval for [Target]
    • RA for [Target]
    • [Target] Readability Approval
    • [Target] RA
    • Readability Approval
    • RA
    • Approval
  • LGTM
    • LGTM
    • lgtm

@tk26eng tk26eng requested a review from iizukak April 17, 2020 07:55
@tk26eng tk26eng self-assigned this Apr 17, 2020
@iizukak
Copy link
Member

iizukak commented Apr 17, 2020

@tk26eng Thanks. Can you give me the output of run.py to check #929 solved?

@tk26eng
Copy link
Contributor Author

tk26eng commented Apr 17, 2020

This log is from original run.py.
I removed some logs which is not related to this issue because they are too long.

$ python3 python/run.py -m lib/libdlk_x86.so -c meta.yaml -i ../CamVid/val/0016E5_07959.png --trial 10
INFO:__main__:Function _pre_process took 0.002661852864548564 seconds to complete
INFO:__main__:Function _pre_process took 0.0021740971133112907 seconds to complete
INFO:__main__:Function _pre_process took 0.0013471120037138462 seconds to complete
INFO:__main__:Function _pre_process took 0.001183542888611555 seconds to complete
INFO:__main__:Function _pre_process took 0.0016493839211761951 seconds to complete
INFO:__main__:Function _pre_process took 0.0011473170015960932 seconds to complete
INFO:__main__:Function _pre_process took 0.0016200740355998278 seconds to complete
INFO:__main__:Function _pre_process took 0.0011365190148353577 seconds to complete
INFO:__main__:Function _pre_process took 0.0016173268668353558 seconds to complete
INFO:__main__:Function _pre_process took 0.0011295280419290066 seconds to complete
INFO:__main__:Avg(func _pre_process): 0.0015666753752157092 sec.
INFO:__main__:Function _run took 2.8996931849978864 seconds to complete
INFO:__main__:Function _run took 2.915913230041042 seconds to complete
INFO:__main__:Function _run took 2.9033334071282297 seconds to complete
INFO:__main__:Function _run took 2.910078221000731 seconds to complete
INFO:__main__:Function _run took 2.914406050927937 seconds to complete
INFO:__main__:Function _run took 2.899797271937132 seconds to complete
INFO:__main__:Function _run took 2.9151353198103607 seconds to complete
INFO:__main__:Function _run took 2.9041824468877167 seconds to complete
INFO:__main__:Function _run took 2.9097492690198123 seconds to complete
INFO:__main__:Function _run took 2.909641008125618 seconds to complete
INFO:__main__:Avg(func _run): 2.9081929409876466 sec.
INFO:__main__:Output: (before post process)

INFO:__main__:Function _post_process took 1.6513047739863396e-05 seconds to complete
INFO:__main__:Function _post_process took 1.2961681932210922e-06 seconds to complete
INFO:__main__:Function _post_process took 9.359791874885559e-07 seconds to complete
INFO:__main__:Function _post_process took 8.328352123498917e-07 seconds to complete
INFO:__main__:Function _post_process took 7.960479706525803e-07 seconds to complete
INFO:__main__:Function _post_process took 8.700881153345108e-07 seconds to complete
INFO:__main__:Function _post_process took 8.16071406006813e-07 seconds to complete
INFO:__main__:Function _post_process took 8.069910109043121e-07 seconds to complete
INFO:__main__:Function _post_process took 7.75093212723732e-07 seconds to complete
INFO:__main__:Function _post_process took 7.541384547948837e-07 seconds to complete
INFO:__main__:Avg(func _post_process): 2.439646050333977e-06 sec.
INFO:__main__:Output: (after post process)

INFO:__main__:save json: output/output.json
INFO:__main__:save image: output/images/0016E5_07959.png
INFO:__main__:save image: output/images/0016E5_07959.png
INFO:__main__:Benchmark avg result(sec) for 10 trials: pre_process: 0.0015666753752157092  inference: 2.9081929409876466 post_process: 2.439646050333977e-06  Total: 2.909762056008913

@tk26eng
Copy link
Contributor Author

tk26eng commented Apr 17, 2020

This is a log from improved run.py.

$ python3 python/run.py -m lib/libdlk_x86.so -c meta.yaml -i ../CamVid/val/0016E5_07959.png --trial 10
INFO:__main__:Function _pre_process took 0.002802397822961211 seconds to complete
INFO:__main__:Function _run took 2.8791214958764613 seconds to complete
INFO:__main__:Function _post_process took 1.3412907719612122e-05 seconds to complete
INFO:__main__:Function _pre_process took 0.002683616941794753 seconds to complete
INFO:__main__:Function _run took 2.883878685068339 seconds to complete
INFO:__main__:Function _post_process took 1.2026168406009674e-05 seconds to complete
INFO:__main__:Function _pre_process took 0.0026482800021767616 seconds to complete
INFO:__main__:Function _run took 2.8805576048325747 seconds to complete
INFO:__main__:Function _post_process took 1.184898428618908e-05 seconds to complete
INFO:__main__:Function _pre_process took 0.0022290239576250315 seconds to complete
INFO:__main__:Function _run took 2.881930090021342 seconds to complete
INFO:__main__:Function _post_process took 1.1490890756249428e-05 seconds to complete
INFO:__main__:Function _pre_process took 0.0022081630304455757 seconds to complete
INFO:__main__:Function _run took 2.8811711110174656 seconds to complete
INFO:__main__:Function _post_process took 1.1846888810396194e-05 seconds to complete
INFO:__main__:Function _pre_process took 0.0022052910644561052 seconds to complete
INFO:__main__:Function _run took 2.8586047310382128 seconds to complete
INFO:__main__:Function _post_process took 1.1841068044304848e-05 seconds to complete
INFO:__main__:Function _pre_process took 0.0021971690002828836 seconds to complete
INFO:__main__:Function _run took 2.8789779420476407 seconds to complete
INFO:__main__:Function _post_process took 5.37605956196785e-06 seconds to complete
INFO:__main__:Function _pre_process took 0.0015663520898669958 seconds to complete
INFO:__main__:Function _run took 2.8791096380446106 seconds to complete
INFO:__main__:Function _post_process took 1.1652009561657906e-05 seconds to complete
INFO:__main__:Function _pre_process took 0.0022083939984440804 seconds to complete
INFO:__main__:Function _run took 2.8753687757998705 seconds to complete
INFO:__main__:Function _post_process took 5.624955520033836e-06 seconds to complete
INFO:__main__:Function _pre_process took 0.0015324631240218878 seconds to complete
INFO:__main__:Function _run took 2.875194007065147 seconds to complete
INFO:__main__:Function _post_process took 1.1805910617113113e-05 seconds to complete
INFO:__main__:save json: output/output.json
INFO:__main__:save image: output/images/0016E5_07959.png
INFO:__main__:save image: output/images/0016E5_07959.png
INFO:__main__:Benchmark avg result(sec) for 10 trials
INFO:__main__:({'total': {'mean': 2.879630215768702, 'std': 0.006862973546535256}, 'pre': {'mean': 0.0022281151032075286, 'std': 0.00040471588833259145}, 'post': {'mean': 1.0692584328353406e-05, 'std': 2.644066623170189e-06}, 'run': {'mean': 2.8773914080811664, 'std': 0.006762520295850802}},)

@tk26eng
Copy link
Contributor Author

tk26eng commented Apr 17, 2020

  • Results of new _pre_process and new _post_process seems to get better
  • Results of new _run is almost same as original one
measurement mean std
original _pre_process 1.567 ms 0.482 ms
new _pre_process 2.228 ms 0.405 ms
original _run 2.908 s 0.006 s
new _run 2.877 s 0.007 s
original _post_process 2.440 us 4.693 us
new _post_process 10.693 us 2.644 us
original total 2.910 s 0.006 s
new totall 2.880 s 0.007 s

Copy link
Member

@iizukak iizukak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OA

@@ -93,12 +96,12 @@ def _init(model, config):
return nn


def _run(nn, image_data):
def _run(image_data, nn):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I missed this line.
Why did change order of arguments?

Copy link
Contributor Author

@tk26eng tk26eng Apr 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed this line because the similar functions (_pre_process, _run, _post_process) should have same order of argument.
But when I checked my code again, my current code is also not so ruled that I can recover original code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iizukak
The changing order has been recovered to original code

@tk26eng tk26eng requested a review from iizukak April 27, 2020 09:51
"pre": {"mean": np.mean(results_pre), "std": np.std(results_pre), },
"post": {"mean": np.mean(results_post), "std": np.std(results_post), },
"run": {"mean": np.mean(results_run), "std": np.std(results_run), },
},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tk26eng Sorry, I missed this.
This line looks return tuple. This is what you want?

In [1]: a = {"a": 0, "b": 1}
In [2]: b = {"a": 0, "b": 1},
In [3]: type(a)
Out[3]: dict
In [4]: type(b)
Out[4]: tuple

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you.
Dict type is what I wanted to use.
I fixed this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iizukak
Can I go to Readability Review ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tk26eng yes

@tk26eng tk26eng requested review from iizukak and tfujiwar April 28, 2020 13:31
Copy link
Contributor

@tfujiwar tfujiwar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution!
Suggested some small changes.


# add the batch dimension
image_data = np.expand_dims(image_data, axis=0)
funcs = [(_pre_process, image_data), _run, _post_process]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

funcs seems not to be used.

@tk26eng tk26eng requested a review from tfujiwar April 30, 2020 13:11
Copy link
Contributor

@tfujiwar tfujiwar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Readability Approval

Copy link
Member

@iizukak iizukak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OA

@iizukak
Copy link
Member

iizukak commented Apr 30, 2020

/ready

@bo-mergebot
Copy link
Contributor

bo-mergebot bot commented Apr 30, 2020

⏳Merge job is queued...

@bo-mergebot
Copy link
Contributor

bo-mergebot bot commented Apr 30, 2020

😥This PR is not approved yet.

@tfujiwar tfujiwar changed the title Fix a stability about multiple measurements [WIP] Fix a stability about multiple measurements Apr 30, 2020
@tfujiwar tfujiwar changed the title [WIP] Fix a stability about multiple measurements Fix a stability about multiple measurements Apr 30, 2020
@tfujiwar
Copy link
Contributor

/ready

@bo-mergebot
Copy link
Contributor

bo-mergebot bot commented Apr 30, 2020

⏳Merge job is queued...

@bo-mergebot bo-mergebot bot merged commit 2d670bc into blue-oil:master Apr 30, 2020
@tk26eng tk26eng deleted the improve_continuous_count branch May 1, 2020 03:19
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
CI: auto-run Run CI automatically
Projects
None yet
Development

Successfully merging this pull request may close these issues.

inference: run.py time measurement include library load
3 participants