forked from dmlc/gluon-cv
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* incorporate autodatasets (dmlc#1496) * Add torch clarification (dmlc#1495) * Add torch clarification * fix * Fix auto detectors (dmlc#1497) * fix yolo predictor * fix predict * fix config (dmlc#1498) * Added support for AWS Batch. Added support for docker (dmlc#1474) * Added support for AWS Batch. Added support for docker * Fixed style. Removed code in commet. Updated README to include boto3 usage * Renamed template file. Removed gluon aws id * fix readme * fix * fix imports (dmlc#1499) * fix imports * fix * fix image classification * fix * fix width height * fix * fix batch size * fix * fix * none to empty string (dmlc#1502) * [WIP] Tinycoco (dmlc#1501) * Add minicoco * update jenkins for minicoco * fix * renamed mini to tiny * fix * fix * fix, add VOCDetectionTiny * fix * fix env * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * test * test * test * clean up Co-authored-by: Joshua Z. Zhang <[email protected]> * Fix rcnn target generator (dmlc#1508) * fix not used rcnn target generator * fix lint * fix * fix * add get flops (dmlc#1509) * warmup scheduler for video torch (dmlc#1510) 1. refine warmup logic, now using cfg.CONFIG.TRAIN.USE_WARMUP to control open warmup or not. 2. fix bug in gluoncv/torch/utils/lr_policy.py 3. change training configs 4. change ddp_train_pytorch and ddp_train_shortonly_pytorch, This is tested on ec2 machines * update torchvideo model zoo (dmlc#1513) * add ir-csn-152 into torchvideo model zoo (dmlc#1515) * Revise danet.py (dmlc#1507) The dropout layer should be placed before the classification layer. * icnet missing background class (dmlc#1518) * Add CSN model to torch video model zoo (dmlc#1517) * add ircsn * update model zoo * fix lint * Improve auto tasks (dmlc#1523) * use in-memory pickle instead of disk file * add feature extractor for image classification * add tests * fix * fix lint * more unittests * fix * fix * Added github action and workflow for sanity check * Removed container and actions. * Added unit test * Added build docs * Fix * Fix * Fix * Fix * Test * test * Update unit test * fix * fix * fix * fix * fix * fix * fix * subclass coco * fix * fix * fix * fix * rebase conflict * fix rebase * fix * fix * add aws authentication * add aws authentication * test * test * test * test * test * fix log * test * test * test * test * test * test * fix * rebase * add tiny motorbike * fix * model zoo * test * fix docker * parallel jobs * parallel jobs * fix * add torch * add torch * fix * fix * fix * full test * full test * test build docs * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * test branch * test branch * fix * test * test * add comment Co-authored-by: Joshua Z. Zhang <[email protected]> Co-authored-by: Yi Zhu <[email protected]> Co-authored-by: Xinyu Li <[email protected]> Co-authored-by: Chunhui Liu <[email protected]> Co-authored-by: YANYI ZHANG <[email protected]> Co-authored-by: BebDong <[email protected]> Co-authored-by: Kuang Haofei <[email protected]>
- Loading branch information
8 people
authored
Nov 18, 2020
1 parent
3fa9613
commit 4e67616
Showing
10 changed files
with
272 additions
and
13 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| BRANCH=$(basename $1) | ||
| PR_NUMBER=$2 | ||
| COMMIT_SHA=$3 | ||
|
|
||
| EFS=/mnt/efs | ||
|
|
||
| mkdir -p ~/.mxnet/datasets | ||
| for f in $EFS/.mxnet/datasets/*; do | ||
| if [ -d "$f" ]; then | ||
| # Will not run if no directories are available | ||
| ln -s $f ~/.mxnet/datasets/$(basename "$f") | ||
| fi | ||
| done | ||
|
|
||
| python3 -m pip install sphinx>=1.5.5 sphinx-gallery sphinx_rtd_theme matplotlib Image recommonmark scipy mxtheme | ||
|
|
||
| export MXNET_CUDNN_AUTOTUNE_DEFAULT=0 | ||
| cd docs | ||
| make html | ||
| COMMAND_EXIT_CODE=$? | ||
| sed -i.bak 's/33\\,150\\,243/23\\,141\\,201/g' build/html/_static/material-design-lite-1.3.0/material.blue-deep_orange.min.css; | ||
| sed -i.bak 's/2196f3/178dc9/g' build/html/_static/sphinx_materialdesign_theme.css; | ||
| sed -i.bak 's/pre{padding:1rem;margin:1.5rem\\s0;overflow:auto;overflow-y:hidden}/pre{padding:1rem;margin:1.5rem 0;overflow:auto;overflow-y:scroll}/g' build/html/_static/sphinx_materialdesign_theme.css | ||
|
|
||
| if [[ $BRANCH == master ]]; then | ||
| # aws s3 cp s3://gluon-cv.mxnet.io/coverage.svg build/html/coverage.svg | ||
| aws s3 sync --delete build/html/ s3://gluoncv-ci/build_docs/master/ --acl public-read --cache-control max-age=7200 | ||
| # aws s3 cp build/html/coverage.svg s3://gluon-cv.mxnet.io/coverage.svg --acl public-read --cache-control max-age=300 | ||
| # echo "Uploaded doc to http://gluon-cv.mxnet.io" | ||
| echo master | ||
| else | ||
| # aws s3 cp s3://gluoncv-ci/build_docs/$PR_NUMBER/$COMMIT_SHA/coverage.svg build/html/coverage.svg | ||
| aws s3 sync --delete build/html/ s3://gluoncv-ci/build_docs/$PR_NUMBER/$COMMIT_SHA/ --acl public-read | ||
| # echo "Uploaded doc to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/${env.BRANCH_NAME}/${env.BUILD_NUMBER}/index.html" | ||
| echo $BRANCH | ||
| fi; | ||
| exit $COMMAND_EXIT_CODE |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,186 @@ | ||
| name: CI workflow | ||
|
|
||
| on: [push, pull_request_target] | ||
|
|
||
| jobs: | ||
| # unittests: | ||
| # runs-on: ${{ matrix.os }} | ||
| # strategy: | ||
| # matrix: | ||
| # os: [macos-latest, windows-latest, ubuntu-latest] | ||
| # steps: | ||
| # - name: checkout | ||
| # uses: actions/checkout@v2 | ||
| # - name: Setup Miniconda | ||
| # uses: conda-incubator/[email protected] | ||
| # with: | ||
| # auto-update-conda: true | ||
| # python-version: 3.7 | ||
| # - name: sanity-check | ||
| # shell: bash -l {0} | ||
| # run: | | ||
| # conda env create -n gluon_cv_lint -f ./tests/pylint.yml | ||
| # conda env update -n gluon-cv-lint -f ./tests/pylint.yml --prune | ||
| # conda activate gluon-cv-lint | ||
| # conda list | ||
| # make clean | ||
| # make pylint | ||
| # - name: unit-test | ||
| # shell: bash -l {0} | ||
| # run: | | ||
| # conda env create -n gluon_cv_py3_test -f tests/py3_mxnet_ci.yml | ||
| # conda env update -n gluon_cv_py3_test -f tests/py3_mxnet_ci.yml --prune | ||
| # conda activate gluon_cv_py3_test | ||
| # conda list | ||
| # export CUDA_VISIBLE_DEVICES=0 | ||
| # export KMP_DUPLICATE_LIB_OK=TRUE | ||
| # make clean | ||
| # pip install --upgrade --force-reinstall --no-deps . | ||
| # env | ||
| # export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64 | ||
| # export MPLBACKEND=Agg | ||
| # export MXNET_CUDNN_AUTOTUNE_DEFAULT=0 | ||
| # export TINY_COCO=~/.mxnet/datasets/tiny_coco | ||
| # export TINY_MOTORBIKE=~/.mxnet/datasets/tiny_motorbike | ||
| # mkdir -p $TINY_COCO/annotations | ||
| # curl -s https://gluoncv-ci.s3-us-west-2.amazonaws.com/mini_coco/sub_val.zip --output sub_val.zip | ||
| # unzip -q sub_val.zip -d $TINY_COCO | ||
| # mv $TINY_COCO/sub_val $TINY_COCO/val2017 | ||
| # curl -s https://gluoncv-ci.s3-us-west-2.amazonaws.com/mini_coco/instances_val2017_tiny.json --output instances_val2017_tiny.json | ||
| # mv instances_val2017_tiny.json $TINY_COCO/annotations | ||
| # curl -s https://gluoncv-ci.s3-us-west-2.amazonaws.com/tiny_motorbike.zip --output tiny_motorbike.zip | ||
| # unzip -q tiny_motorbike.zip -d $TINY_MOTORBIKE | ||
| # nosetests --with-timer --timer-ok 5 --timer-warning 20 -x --with-coverage --cover-package gluoncv -v tests/unittests | ||
| # model_zoo_mxnet: | ||
| # needs: unittests | ||
| # runs-on: ubuntu-latest | ||
| # steps: | ||
| # - name: checkout | ||
| # uses: actions/checkout@v2 | ||
| # - name: Configure AWS Credentials | ||
| # uses: aws-actions/configure-aws-credentials@v1 | ||
| # with: | ||
| # aws-access-key-id: ${{ secrets.GLUONCV_DEV_ACCESS_ID }} | ||
| # aws-secret-access-key: ${{ secrets.GLUONCV_DEV_SECRET_ACCESS_KEY }} | ||
| # aws-region: us-east-1 | ||
| # - name: Install dependencies | ||
| # run: | | ||
| # pip install --upgrade --force-reinstall --no-deps . | ||
| # pip install boto3 | ||
| # - name: Test model_zoo_mxnet on AWS Batch(For push) | ||
| # shell: bash -l {0} | ||
| # if: ${{ github.event_name == 'push' }} | ||
| # run: | | ||
| # echo "Start submitting job" | ||
| # python ./tools/batch/submit-job.py --region us-east-1 \ | ||
| # --job-type g4dn.4x \ | ||
| # --name GluonCV-GPU-Test-${{ github.ref }} \ | ||
| # --source-ref ${{ github.ref }} \ | ||
| # --work-dir tests \ | ||
| # --remote https://github.com/${{ github.repository }} \ | ||
| # --command "nosetests --with-timer --timer-ok 5 --timer-warning 20 -x --with-coverage --cover-package gluoncv -v model_zoo" \ | ||
| # --wait | ||
| # - name: Test model_zoo_mxnet on AWS Batch(For pull request) | ||
| # if: ${{ github.event_name == 'pull_request' || github.event_name == 'pull_request_target' }} | ||
| # run: | | ||
| # echo "Start submitting job" | ||
| # python ./tools/batch/submit-job.py --region us-east-1 \ | ||
| # --job-type g4dn.4x \ | ||
| # --name GluonCV-GPU-Test-PR#${{ github.event.number }} \ | ||
| # --source-ref ${{ github.event.pull_request.head.ref }} \ | ||
| # --work-dir tests \ | ||
| # --remote https://github.com/${{ github.event.pull_request.head.repo.full_name }} \ | ||
| # --command "nosetests --with-timer --timer-ok 5 --timer-warning 20 -x --with-coverage --cover-package gluoncv -v model_zoo" \ | ||
| # --wait | ||
| # model_zoo_torch: | ||
| # needs: unittests | ||
| # runs-on: ubuntu-latest | ||
| # steps: | ||
| # - name: checkout | ||
| # uses: actions/checkout@v2 | ||
| # - name: Configure AWS Credentials | ||
| # uses: aws-actions/configure-aws-credentials@v1 | ||
| # with: | ||
| # aws-access-key-id: ${{ secrets.GLUONCV_DEV_ACCESS_ID }} | ||
| # aws-secret-access-key: ${{ secrets.GLUONCV_DEV_SECRET_ACCESS_KEY }} | ||
| # aws-region: us-east-1 | ||
| # - name: Install dependencies | ||
| # run: | | ||
| # pip install --upgrade --force-reinstall --no-deps . | ||
| # pip install boto3 | ||
| # - name: Test model_zoo_torch on AWS Batch(For push) | ||
| # shell: bash -l {0} | ||
| # if: ${{ github.event_name == 'push' }} | ||
| # run: | | ||
| # echo "Start submitting job" | ||
| # python ./tools/batch/submit-job.py --region us-east-1 \ | ||
| # --job-type g4dn.4x \ | ||
| # --name GluonCV-GPU-Test-${{ github.ref }} \ | ||
| # --source-ref ${{ github.ref }} \ | ||
| # --work-dir tests \ | ||
| # --remote https://github.com/${{ github.repository }} \ | ||
| # --command "nosetests --with-timer --timer-ok 5 --timer-warning 20 -x --with-coverage --cover-package gluoncv/torch -v model_zoo_torch" \ | ||
| # --wait | ||
| # - name: Test model_zoo_torch on AWS Batch(For pull request) | ||
| # if: ${{ github.event_name == 'pull_request' || github.event_name == 'pull_request_target' }} | ||
| # run: | | ||
| # echo "Start submitting job" | ||
| # python ./tools/batch/submit-job.py --region us-east-1 \ | ||
| # --job-type g4dn.4x \ | ||
| # --name GluonCV-GPU-Test-PR#${{ github.event.number }} \ | ||
| # --source-ref ${{ github.event.pull_request.head.ref }} \ | ||
| # --work-dir tests \ | ||
| # --remote https://github.com/${{ github.event.pull_request.head.repo.full_name }} \ | ||
| # --command "nosetests --with-timer --timer-ok 5 --timer-warning 20 -x --with-coverage --cover-package gluoncv/torch -v model_zoo_torch" \ | ||
| # --wait | ||
| build-docs: | ||
| # needs: unittests, model_zoo_mxnet, model_zoo_torch | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: checkout | ||
| uses: actions/checkout@v2 | ||
| - name: Configure AWS Credentials | ||
| uses: aws-actions/configure-aws-credentials@v1 | ||
| with: | ||
| aws-access-key-id: ${{ secrets.GLUONCV_DEV_ACCESS_ID }} | ||
| aws-secret-access-key: ${{ secrets.GLUONCV_DEV_SECRET_ACCESS_KEY }} | ||
| aws-region: us-east-1 | ||
| - name: Install dependencies | ||
| run: | | ||
| pip install --upgrade --force-reinstall --no-deps . | ||
| pip install boto3 | ||
| - name: Build docs on AWS Batch(For push) | ||
| shell: bash -l {0} | ||
| if: ${{ github.event_name == 'push' }} | ||
| run: | | ||
| echo "Start submitting job" | ||
| python ./tools/batch/submit-job.py --region us-east-1 \ | ||
| --job-type g4dn.4x \ | ||
| --name GluonCV-GPU-BuildDocs-${{ github.ref }} \ | ||
| --source-ref ${{ github.ref }} \ | ||
| --work-dir . \ | ||
| --remote https://github.com/${{ github.repository }} \ | ||
| --command "chmod +x ./.github/workflows/build_docs.sh && ./.github/workflows/build_docs.sh ${{ github.ref }} ${{ github.event.number }} ${{ github.sha }}" \ | ||
| --wait | ||
| - name: Build docs on AWS Batch(For pull request) | ||
| if: ${{ github.event_name == 'pull_request' || github.event_name == 'pull_request_target' }} | ||
| run: | | ||
| echo "Start submitting job" | ||
| python ./tools/batch/submit-job.py --region us-east-1 \ | ||
| --job-type g4dn.4x \ | ||
| --name GluonCV-GPU-BuildDocs-PR#${{ github.event.number }} \ | ||
| --source-ref ${{ github.event.pull_request.head.ref }} \ | ||
| --work-dir . \ | ||
| --remote https://github.com/${{ github.event.pull_request.head.repo.full_name }} \ | ||
| --command "chmod +x ./.github/workflows/build_docs.sh && ./.github/workflows/build_docs.sh ${{ github.ref }} ${{ github.event.number }} ${{ github.sha }}" \ | ||
| --wait | ||
| - name: Comment on PR | ||
| if: ${{ github.event_name == 'pull_request' || github.event_name == 'pull_request_target' }} | ||
| uses: peter-evans/[email protected] | ||
| with: | ||
| issue-number: ${{ github.event.number }} | ||
| body: | | ||
| Job ${{ github.event.number }}-${{ github.sha }} is done. | ||
| Docs are uploaded to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/${{ github.event.number }}/${{ github.sha }}/index.html | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| name: gluon_cv_py3_mxnet | ||
| channels: | ||
| - conda-forge | ||
| - defaults | ||
| dependencies: | ||
| - python=3.6 | ||
| - perl | ||
| - sphinx=1.7.2 | ||
| - nose | ||
| - coverage=4.5.4 | ||
| - scipy | ||
| - cython | ||
| - pip=20.2.4 | ||
| - requests | ||
| - matplotlib | ||
| - tqdm | ||
| - pillow | ||
| - pip: | ||
| - mxnet | ||
| - coverage-badge | ||
| - awscli | ||
| - nose-timer | ||
| - opencv-python | ||
| - git+https://github.com/zhanghang1989/detail-api.git#subdirectory=PythonAPI | ||
| - portalocker | ||
| - autocfg | ||
| - boto3 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters