Skip to content

Latest commit

 

History

History
179 lines (126 loc) · 8.19 KB

mmedit.md

File metadata and controls

179 lines (126 loc) · 8.19 KB

MMEditing Deployment

MMEditing aka mmedit is an open-source image and video editing toolbox based on PyTorch. It is a part of the OpenMMLab project.

Installation

Install mmedit

Please follow the installation guide to install mmedit. If you have already done that, please move on to the next section.

Install mmdeploy

There are several methods to install mmdeploy, among which you can choose an appropriate one according to your target platform and device.

Method I: Install precompiled package

TODO. MMDeploy hasn't released based on dev-1.x branch.

Method II: Build using scripts

If your target platform is Ubuntu 18.04 or later version, we encourage you to run scripts. For example, the following commands install mmdeploy as well as inference engine - ONNX Runtime.

git clone --recursive -b dev-1.x https://github.com/open-mmlab/mmdeploy.git
cd mmdeploy
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH

Method III: Build from source

If neither I nor II meets your requirements, building mmdeploy from source is the last option.

Convert model

You can use tools/deploy.py to convert mmedit models to the specified backend models. Its detailed usage can be learned from here.

When using tools/deploy.py, it is crucial to specify the correct deployment config. We've already provided builtin deployment config files of all supported backends for mmedit, under which the config file path follows the pattern:

{task}/{task}_{backend}-{precision}_{static | dynamic}_{shape}.py

  • {task}: task in mmedit.

    MMDeploy supports models of one task in mmedit, i.e., super resolution. Please refer to chapter supported models for task-model organization.

    DO remember to use the corresponding deployment config file when trying to convert models of different tasks.

  • {backend}: inference backend, such as onnxruntime, tensorrt, pplnn, ncnn, openvino, coreml etc.

  • {precision}: fp16, int8. When it's empty, it means fp32

  • {static | dynamic}: static shape or dynamic shape

  • {shape}: input shape or shape range of a model

Convert super resolution model

The command below shows an example about converting ESRGAN model to onnx model that can be inferred by ONNX Runtime.

cd mmdeploy
# download esrgan model from mmedit model zoo
mim download mmedit --config esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k --dest .
# convert esrgan model to onnxruntime model with dynamic shape
python tools/deploy.py \
  configs/mmedit/super-resolution/super-resolution_onnxruntime_dynamic.py \
  esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py \
  esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth \
  demo/resources/face.png \
  --work-dir mmdeploy_models/mmedit/ort \
  --device cpu \
  --show

You can also convert the above model to other backend models by changing the deployment config file *_onnxruntime_dynamic.py to others, e.g., converting to tensorrt model by super-resolution/super-resolution_tensorrt-_dynamic-32x32-512x512.py.

When converting mmocr models to tensorrt models, --device should be set to "cuda"

Model specification

Before moving on to model inference chapter, let's know more about the converted model structure which is very important for model inference.

The converted model locates in the working directory like mmdeploy_models/mmedit/ort in the previous example. It includes:

mmdeploy_models/mmedit/ort
├── deploy.json
├── detail.json
├── end2end.onnx
└── pipeline.json

in which,

  • end2end.onnx: backend model which can be inferred by ONNX Runtime
  • deploy.json: meta information about backend model
  • pipeline.json: inference pipeline of mmdeploy SDK
  • detail.json: conversion parameters

The whole package mmdeploy_models/mmedit/ort is defined as mmdeploy SDK model, i.e., mmdeploy SDK model includes both backend model and inference meta information.

Model inference

Backend model inference

MMDeploy provides a unified API named as inference_model to inference model, making all inference backends API transparent to users.

Take the previous converted end2end.onnx model as an example, you can use the following code to inference the model and visualize the results.

from mmdeploy.apis.utils import build_task_processor
from mmdeploy.utils import get_input_shape, load_config
import torch

deploy_cfg = 'configs/mmedit/super-resolution/super-resolution_onnxruntime_dynamic.py'
model_cfg = 'esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py'
device = 'cpu'
backend_model = ['./mmdeploy_models/mmedit/ort/end2end.onnx']
image = './demo/resources/face.png'

# read deploy_cfg and model_cfg
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)

# build task and backend model
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
model = task_processor.build_backend_model(backend_model)

# process input image
input_shape = get_input_shape(deploy_cfg)
model_inputs, _ = task_processor.create_input(image, input_shape)

# do model inference
with torch.no_grad():
    result = model.test_step(model_inputs)

# visualize results
task_processor.visualize(
    image=image,
    model=model,
    result=result[0],
    window_name='visualize',
    output_file='output_restorer.bmp')

SDK model inference

TODO

Supported models

Model Task ONNX Runtime TensorRT ncnn PPLNN OpenVINO
SRCNN super-resolution Y Y Y Y Y
ESRGAN super-resolution Y Y Y Y Y
ESRGAN-PSNR super-resolution Y Y Y Y Y
SRGAN super-resolution Y Y Y Y Y
SRResNet super-resolution Y Y Y Y Y
Real-ESRGAN super-resolution Y Y Y Y Y
EDSR super-resolution Y Y Y N Y
RDN super-resolution Y Y Y Y Y