Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[microTVM] Enable micro tvmc tutorial testing in CI #10555

Merged
merged 5 commits into from
Mar 16, 2022
Merged

[microTVM] Enable micro tvmc tutorial testing in CI #10555

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0090877
Add script convertor
mehrdadh Mar 9, 2022
85dd77a
Address comments: added test
mehrdadh Mar 15, 2022
c83fe69
address comments
mehrdadh Mar 15, 2022
b986cd3
Merge branch 'main' into add_tvmc_to_ci
mehrdadh Mar 15, 2022
8325c59
Merge branch 'main' into add_tvmc_to_ci
mehrdadh Mar 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions docs/script_convert.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

import argparse
import pathlib

BASH = "# bash"
BASH_IGNORE = "# bash-ignore"
BASH_MULTILINE_COMMENT_START = ": '"
BASH_MULTILINE_COMMENT_END = "'"


def bash_to_python(src_path: pathlib.Path, dest_path: pathlib.Path):
"""Convert a bash script file to a Python format compatible with Sphinx doc."""
with open(src_path, "r") as src_f:
with open(dest_path, "w") as dest_f:
line = src_f.readline()
mehrdadh marked this conversation as resolved.
Show resolved Hide resolved
bash_block = []
bash_detected = False
bash_ignore_detected = False
new_line_required = False
while line:
line = line.strip("\n").strip("\r")
if bash_detected:
if line == BASH:
# write the bash block to destination
if new_line_required:
dest_f.write("\n")
python_code = "# .. code-block:: bash\n#\n"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we run the bash script from this file and include the output? we could probably do that by injecting output markers into the script e.g.

echo -e "\nSTART OF COMMAND 1 OUTPUT --->"
tvmc ...
echo -e "\n<--- START OF COMMAND 1 OUTPUT"

then we could take the bash-ignore commented areas, extract them into a common prelude, and also include stuff like set -eux so that people write proper bash scripts. we could maybe do this by putting that into a docs/bash_gallery_prelude.sh and erroring in this script if we don't see a line like:

source docs/bash_gallery_prelude.sh || exit 2

might need to think about the path a little bit

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the idea to add the output of each command to generated python file and finally have it as part of generated HTML file?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah exactly

for bash_line in bash_block:
python_code += f"#\t {bash_line}\n"
python_code += "#"
dest_f.write(python_code)

bash_detected = False
bash_block = []
new_line_required = True
else:
# add new bash command line
bash_block.append(line)
elif bash_ignore_detected:
if line == BASH_IGNORE:
bash_ignore_detected = False
new_line_required = True
else:
new_line_required = False
pass
else:
if line == BASH:
bash_detected = True
elif line == BASH_IGNORE:
bash_ignore_detected = True
elif line in [BASH_MULTILINE_COMMENT_START, BASH_MULTILINE_COMMENT_END]:
if new_line_required:
dest_f.write("\n")
dest_f.write('"""')
new_line_required = True
else:
if new_line_required:
dest_f.write("\n")
dest_f.write(f"{line}")
new_line_required = True

line = src_f.readline()
if new_line_required:
dest_f.write("\n")


def main():
parser = argparse.ArgumentParser(description="Convert tutorial script to Python.")
parser.add_argument("script", type=str, help="Path to script file.")
args = parser.parse_args()

src_path = pathlib.Path(args.script)
dest_path = src_path.parent / f"{src_path.stem}.py"
bash_to_python(src_path, dest_path)


if __name__ == "__main__":
main()
110 changes: 54 additions & 56 deletions ...y/how_to/work_with_microtvm/micro_tvmc.py → ...y/how_to/work_with_microtvm/micro_tvmc.sh
100644 → 100755
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# specific language governing permissions and limitations
# under the License.

"""
: '
.. _tutorial-micro-tvmc:

Executing a Tiny Model with TVMC Micro
Expand All @@ -25,7 +25,7 @@
This tutorial explains how to compile a tiny model for a micro device,
build a program on Zephyr platform to execute this model, flash the program
and run the model all using `tvmc micro` command.
"""
'

######################################################################
# .. note::
Expand All @@ -42,6 +42,10 @@
# ./docker/bash.sh tlcpack/ci-qemu
#

# bash-ignore
shopt -s expand_aliases
alias tvmc="python3 -m tvm.driver.tvmc"
# bash-ignore

############################################################
# Using TVMC Micro
Expand All @@ -55,16 +59,15 @@
#
# To check if you have TVMC command installed on your machine, you can run:
#
# .. code-block:: bash
#
# tvmc --help
#
# bash
tvmc --help
# bash
# To compile a model for microtvm we use ``tvmc compile`` subcommand. The output of this command
# is used in next steps with ``tvmc micro`` subcommands. You can check the availability of TVMC Micro using:
#
# .. code-block:: bash
#
# tvmc micro --help
# bash
tvmc micro --help
# bash
#
# The main tasks that you can perform using ``tvmc micro`` are ``create``, ``build`` and ``flash``.
# To read about specific options under a givern subcommand, use
Expand All @@ -80,10 +83,9 @@
#
# For this tutorial we will be using the model in tflite format.
#
# .. code-block:: bash
#
# wget https://github.com/tensorflow/tflite-micro/raw/main/tensorflow/lite/micro/examples/magic_wand/magic_wand.tflite
#
# bash
wget https://github.com/tensorflow/tflite-micro/raw/main/tensorflow/lite/micro/examples/magic_wand/magic_wand.tflite
# bash

############################################################
# Compiling a TFLite model to a Model Library Format
Expand All @@ -95,19 +97,18 @@
#
# Here, we generate a MLF file for ``qemu_x86`` Zephyr board. To generate MLF output for the ``magic_wand`` tflite model:
#
# .. code-block:: bash
#
# tvmc compile magic_wand.tflite \
# --target='c -keys=cpu -link-params=0 -model=host' \
# --runtime=crt \
# --runtime-crt-system-lib 1 \
# --executor='graph' \
# --executor-graph-link-params 0 \
# --output model.tar \
# --output-format mlf \
# --pass-config tir.disable_vectorize=1 \
# --disabled-pass=AlterOpLayout
#
# bash
tvmc compile magic_wand.tflite \
--target='c -keys=cpu -link-params=0 -model=host' \
--runtime=crt \
--runtime-crt-system-lib 1 \
--executor='graph' \
--executor-graph-link-params 0 \
--output model.tar \
--output-format mlf \
--pass-config tir.disable_vectorize=1 \
--disabled-pass=AlterOpLayout
# bash
# This will generate a ``model.tar`` file which contains TVM compiler output files. To run this command for
# a different Zephyr device, you need to update ``target``. For instance, for ``nrf5340dk_nrf5340_cpuapp`` board
# the target is ``--target='c -keys=cpu -link-params=0 -model=nrf5340dk'``.
Expand All @@ -122,14 +123,13 @@
# for the project to ``create`` subcommand along with project options. Project options for each
# platform (Zephyr/Arduino) are defined in their Project API server file. To generate Zephyr project, run:
#
# .. code-block:: bash
#
# tvmc micro create \
# project \
# model.tar \
# zephyr \
# --project-option project_type=host_driven zephyr_board=qemu_x86
#
# bash
tvmc micro create \
project \
model.tar \
zephyr \
--project-option project_type=host_driven zephyr_board=qemu_x86
# bash
# This will generate a ``Host-Driven`` Zephyr project for ``qemu_x86`` Zephyr board. In Host-Driven template project,
# the Graph Executor will run on host and perform the model execution on Zephyr device by issuing commands to the
# device using an RPC mechanism. Read more about `Host-Driven Execution <https://tvm.apache.org/docs/arch/microtvm_design.html#host-driven-execution>`_.
Expand All @@ -148,26 +148,24 @@
# Next step is to build the Zephyr project which includes TVM generated code for running the tiny model, Zephyr
# template code to run a model in Host-Driven mode and TVM runtime source/header files. To build the project:
#
# .. code-block:: bash
#
# tvmc micro build \
# project \
# zephyr \
# --project-option zephyr_board=qemu_x86
#
# bash
tvmc micro build \
project \
zephyr \
--project-option zephyr_board=qemu_x86
# bash
# This will build the project in ``project`` directory and generates binary files under ``project/build``. To build
# Zephyr project for a different Zephyr board, change ``zephyr_board`` project option.
#
# Next, we flash the Zephyr binary file to Zephyr device. For ``qemu_x86`` Zephyr board this step does not
# actually perform any action since QEMU will be used, however you need this step for physical hardware.
#
# .. code-block:: bash
#
# tvmc micro flash \
# project \
# zephyr \
# --project-option zephyr_board=qemu_x86
#
# bash
tvmc micro flash \
project \
zephyr \
--project-option zephyr_board=qemu_x86
# bash

############################################################
# Run Tiny Model on Micro Target
Expand All @@ -179,14 +177,14 @@
# and pass ``--device micro`` to specify the device type. This command will open a communication channel, set input
# values using ``Graph Executor`` on host and run full model on the device. Then it gets output from the device.
#
# .. code-block:: bash
#
# tvmc run \
# --device micro \
# project \
# --project-option zephyr_board=qemu_x86 \
# --fill-mode ones
# --print-top 4
# bash
tvmc run \
--device micro \
project \
--project-option zephyr_board=qemu_x86 \
--fill-mode ones \
--print-top 4
# bash
# # Output:
# #
# # INFO:__main__:b'[100%] [QEMU] CPU: qemu32,+nx,+pae\n'
Expand Down
5 changes: 1 addition & 4 deletions tests/python/unittest/test_ci.py → tests/python/ci/test_ci.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,16 +15,13 @@
# specific language governing permissions and limitations
# under the License.

import pathlib
import subprocess
import sys
import json
import textwrap
import tempfile

import pytest

REPO_ROOT = pathlib.Path(__file__).resolve().parent.parent.parent.parent
from test_utils import REPO_ROOT


class TempGit:
Expand Down
Loading