tf_upgrade_v2 for dataset #894

ytfksw · 2020-03-04T05:42:43Z

What this patch does to fix the issue.

I converted only the blueoil/datasets with the command line tool tf_upgrade_v2 to run on tensorflow2.

There is a code ("metaclass = ABCMeta", "yield from") that tf_upgrade_v2 cannot support, so I change it once, convert it, and restore it again.

Here's what I did:

install newest tenosrflow (tensorflow-gpu==1.5.2)
Change once to convert tf_upgrade_v2 8c6db67
Run the upgrade script tf_upgrade_v2 8e31b9c
Undo changes in 2 1eedf62
Check the upgrade report for warnings and errors
Run test (make test)

The only change this time is also to replace the operator with compat.v1.

Link to any relevant issues or pull requests.

This reverts commit 8c6db67.

ytfksw · 2020-03-04T08:51:19Z

This is the whole log file.

TensorFlow 2.0 Upgrade Script
-----------------------------
Converted 24 files
Detected 0 issues that require attention
--------------------------------------------------------------------------------
================================================================================
Detailed log follows:

================================================================================
================================================================================
Input tree: 'datasets'
================================================================================
--------------------------------------------------------------------------------
Processing file 'datasets/mnist.py'
 outputting to 'datasets/mnist.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/pascalvoc_2007_2012.py'
 outputting to 'datasets/pascalvoc_2007_2012.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/caltech101.py'
 outputting to 'datasets/caltech101.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/div2k.py'
 outputting to 'datasets/div2k.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/image_folder.py'
 outputting to 'datasets/image_folder.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/__init__.py'
 outputting to 'datasets/__init__.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/pascalvoc_2007.py'
 outputting to 'datasets/pascalvoc_2007.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/mscoco_2017.py'
 outputting to 'datasets/mscoco_2017.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/tfds.py'
 outputting to 'datasets/tfds.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/camvid.py'
 outputting to 'datasets/camvid.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/delta_mark.py'
 outputting to 'datasets/delta_mark.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/ilsvrc_2012.py'
 outputting to 'datasets/ilsvrc_2012.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/ytfaces.py'
 outputting to 'datasets/ytfaces.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/base.py'
 outputting to 'datasets/base.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/widerface.py'
 outputting to 'datasets/widerface.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/pascalvoc_2012.py'
 outputting to 'datasets/pascalvoc_2012.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/cifar10.py'
 outputting to 'datasets/cifar10.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/mscoco.py'
 outputting to 'datasets/mscoco.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/open_images_v4.py'
 outputting to 'datasets/open_images_v4.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/pascalvoc_base.py'
 outputting to 'datasets/pascalvoc_base.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/cityscapes.py'
 outputting to 'datasets/cityscapes.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/cifar100.py'
 outputting to 'datasets/cifar100.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/dataset_iterator.py'
 outputting to 'datasets/dataset_iterator.py'
--------------------------------------------------------------------------------

215:19: INFO: Renamed 'tf.data.make_initializable_iterator' to 'tf.compat.v1.data.make_initializable_iterator'
220:29: INFO: Renamed 'tf.ConfigProto' to 'tf.compat.v1.ConfigProto'
221:28: INFO: Renamed 'tf.GPUOptions' to 'tf.compat.v1.GPUOptions'
227:29: INFO: Renamed 'tf.ConfigProto' to 'tf.compat.v1.ConfigProto'
228:23: INFO: Renamed 'tf.Session' to 'tf.compat.v1.Session'
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Processing file 'datasets/bdd100k.py'
 outputting to 'datasets/bdd100k.py'
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

bo-code-review-bot · 2020-03-04T08:56:13Z

This PR needs Approvals as follows.

Ownership Approval for / from iizukak, tkng, ruimashita
Readability Approval for Python from tkng, tsawada

Please choose reviewers and requet reviews!

Click to see how to approve each reviews

You can approve this PR by triggered comments as follows.

Approve all reviews requested to you (readability and ownership) and LGTM review
Approval, LGTM
Approve all ownership reviews
Ownership Approval or OA
Approve all readability reviews
Readability Approval or RA
Approve specified review targets
- Example of Ownership Reviewer of /: Ownership Approval for / or OA for /
- Example of Readability Reviewer of Python: Readability Approval for Python or RA for Python
Approve LGTM review
LGTM

See all trigger comments

Please replace [Target] to review target

Ownership Approval
- Ownership Approval for [Target]
- OA for [Target]
- Ownership Approval
- OA
- Approval
Readability Approval
- Readability Approval for [Target]
- RA for [Target]
- [Target] Readability Approval
- [Target] RA
- Readability Approval
- RA
- Approval
LGTM
- LGTM
- lgtm

iizukak

OA

tsawada

Readability Approval

iizukak · 2020-03-05T01:50:39Z

/ready

bo-mergebot · 2020-03-05T01:50:41Z

⏳Merge job is queued...

bo-mergebot · 2020-03-05T02:30:48Z

😥Status check failed. Please fix problems and send /ready again.

ytfksw · 2020-03-05T03:02:28Z

/ready

bo-mergebot · 2020-03-05T03:02:30Z

⏳Merge job is queued...

bo-mergebot · 2020-03-05T03:02:31Z

😥Status check failed. Please fix problems and send /ready again.

tfujiwar · 2020-03-05T03:34:39Z

/ready

bo-mergebot · 2020-03-05T03:34:41Z

⏳Merge job is queued...

ytfksw added 3 commits March 4, 2020 01:52

temporary change for convert

8c6db67

Apply tf_upgrade_v2

8e31b9c

Revert "temporary change for convert"

1eedf62

This reverts commit 8c6db67.

blueoil-butler bot added the CI: auto-run Run CI automatically label Mar 4, 2020

ytfksw changed the title ~~Convert tf2 dataset~~ tf_upgrade_v2 for dataset Mar 4, 2020

ytfksw marked this pull request as ready for review March 4, 2020 08:56

iizukak approved these changes Mar 5, 2020

View reviewed changes

iizukak requested a review from tsawada March 5, 2020 01:41

tsawada approved these changes Mar 5, 2020

View reviewed changes

Merge branch 'master' into convert_tf2_dataset

9b68255

primenumber added CI: test-blueoil Run blueoil test once CI: test-dlk Run dlk test once labels Mar 5, 2020

blueoil-butler bot removed CI: test-blueoil Run blueoil test once CI: test-dlk Run dlk test once labels Mar 5, 2020

bo-mergebot bot merged commit f1e07c8 into blue-oil:master Mar 5, 2020

ytfksw mentioned this pull request May 29, 2020

Consider upgrading to TensorFlow 2.0 #479

Open

ytfksw self-assigned this Jun 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tf_upgrade_v2 for dataset #894

tf_upgrade_v2 for dataset #894

ytfksw commented Mar 4, 2020 •

edited

Loading

ytfksw commented Mar 4, 2020

bo-code-review-bot bot commented Mar 4, 2020

iizukak left a comment

tsawada left a comment

iizukak commented Mar 5, 2020

bo-mergebot bot commented Mar 5, 2020

bo-mergebot bot commented Mar 5, 2020

ytfksw commented Mar 5, 2020

bo-mergebot bot commented Mar 5, 2020

bo-mergebot bot commented Mar 5, 2020

tfujiwar commented Mar 5, 2020

bo-mergebot bot commented Mar 5, 2020

tf_upgrade_v2 for dataset #894

tf_upgrade_v2 for dataset #894

Conversation

ytfksw commented Mar 4, 2020 • edited Loading

What this patch does to fix the issue.

Link to any relevant issues or pull requests.

ytfksw commented Mar 4, 2020

bo-code-review-bot bot commented Mar 4, 2020

iizukak left a comment

Choose a reason for hiding this comment

tsawada left a comment

Choose a reason for hiding this comment

iizukak commented Mar 5, 2020

bo-mergebot bot commented Mar 5, 2020

bo-mergebot bot commented Mar 5, 2020

ytfksw commented Mar 5, 2020

bo-mergebot bot commented Mar 5, 2020

bo-mergebot bot commented Mar 5, 2020

tfujiwar commented Mar 5, 2020

bo-mergebot bot commented Mar 5, 2020

ytfksw commented Mar 4, 2020 •

edited

Loading