-
Notifications
You must be signed in to change notification settings - Fork 46
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #130 from goatmiemie/dpak_change
add the script for automatically deploying the OVS environment
- Loading branch information
Showing
11 changed files
with
348 additions
and
15 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| # OVS卸载自动化部署指南 | ||
|
|
||
| #### 介绍 | ||
| SP680智能网卡，适用于虚拟化、协议解析场景。本文档为SP680配套OVS软件的自动haul安装指南。此文档仅支持单节点部署，自动部署脚本仅供参考。 | ||
| #### 环境要求 | ||
|
|
||
| 1. 物理机要安装SP680智能网卡，需要在BIOS中开启IOMMU、配置内存大页并安装智能网卡驱动。 | ||
| 2. 节点需要关闭SELinux和防火墙，并能正常访问外部网络。 | ||
| 3. 仅支持openEuler 20.03 LTS SP1 操作系统。 | ||
|
|
||
| #### 使用说明 | ||
|
|
||
| 1. 安装网卡驱动： | ||
| 安装前请先获取相关驱动包和Hinic3_flash.bin 文件，放到'driver'目录中。 | ||
|
|
||
| | 驱动 | | ||
| | ---- | | ||
| | hinic3 | | ||
| | hinicadm3 | | ||
| | hiovs3 | | ||
| | hiovs3-lib | | ||
| | hisdk3 | | ||
|
|
||
| ``` | ||
| cd ovs_studio | ||
| bash install_driver.sh | ||
| ``` | ||
| 驱动安装完后重启服务器。 | ||
|
|
||
| 2. 使能virtio-net设备： | ||
|
|
||
| ``` | ||
| cd ovs_studio | ||
| bash virtio_envable.sh | ||
| ``` | ||
| 重新启动服务器后，virtio-net设备处于未使能状态，需要对其进行使能。可以执行上面命令使能virtio-net设备。 | ||
| 如果需要开机自动执行此脚本，可以进行如下操作。 | ||
|
|
||
| 1).为脚本添加一个服务，参考的virtio-enable.service文件如下: | ||
| [Unit] | ||
| Description=DPAK virtio enable script | ||
|
|
||
| [Service] | ||
| Type=simple | ||
| ExecStart=sh <path>/virtio_envable.sh | ||
|
|
||
| [Install] | ||
| WantedBy=multi-user.target | ||
|
|
||
| 其中，<path>替换为脚本所在的绝对路径。 | ||
|
|
||
| 2).将virtio-enable.service文件放入“/usr/lib/systemd/system” 目录，设置开机自启动。 | ||
| systemctl enable virtio-enable | ||
|
|
||
| 3. 部署ovs： | ||
| 安装前请先获取DPU-solution-dpak-runtime-host-repo_1.0.0_aarch64.zip 软件包，解压并放到package目录中。 | ||
| ``` | ||
| cd ovs_studio | ||
| bash install_env.sh | ||
| ``` |
56 changes: 56 additions & 0 deletions
56
Development_framework/dpak-sdk/build/ovs_build/install_driver.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| #!/usr/bin/bash | ||
|
|
||
| # Copyright 2022 Huawei Technologies Co., Ltd | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| current_dir=$(cd $(dirname $0);pwd) | ||
| driver_dir=${current_dir}/../driver | ||
|
|
||
| exit_or_not(){ | ||
| if [ $(echo $?) -ne 0 ];then | ||
| echo "--------------$1 build failed-------------------" | ||
| exit -1 | ||
| else | ||
| echo "--------------$1 build success-------------------" | ||
| fi | ||
| } | ||
|
|
||
| rpm -ivh ${driver_dir}/hinicadm*.rpm --force | ||
|
|
||
| rpm -e $(rpm -qa | grep hinic3) --nodeps | ||
| rpm -e $(rpm -qa | grep hiovs) --nodeps | ||
| rpm -e $(rpm -qa | grep hisdk) --nodeps | ||
|
|
||
|
|
||
| rpm -ivh ${driver_dir}/hisdk*.rpm --force | ||
| exit_or_not "sdk" | ||
|
|
||
| modprobe hiudk3 | ||
| exit_or_not "modprobe udk" | ||
|
|
||
| modprobe hisdk3 | ||
| exit_or_not "modprobe sdk" | ||
|
|
||
|
|
||
| rpm -ivh ${driver_dir}/hinic*.rpm --force | ||
| exit_or_not "nic" | ||
|
|
||
| rpm -ivh ${driver_dir}/hiovs*.rpm --force | ||
| exit_or_not "ovs" | ||
|
|
||
| [[ -f "/etc/modprobe.d/blacklist.conf" ]] && rm -f "/etc/modprobe.d/blacklist.conf" | ||
| echo "softdep hinic3 pre:virtio" > /etc/modprobe.d/sp680.conf | ||
|
|
||
| cd ${driver_dir} | ||
| hinicadm3 updatefw -i hinic0 -f ./Hinic3_flash.bin -x 0 | ||
| exit_or_not "firmaware" |
101 changes: 101 additions & 0 deletions
101
Development_framework/dpak-sdk/build/ovs_build/install_env.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| #!/usr/bin/bash | ||
|
|
||
| # Copyright 2022 Huawei Technologies Co., Ltd | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| current_dir=$(cd $(dirname $0);pwd) | ||
| package_dir=${current_dir}/../package | ||
|
|
||
| exit_or_not(){ | ||
| if [ $(echo $?) -ne 0 ];then | ||
| echo "--------------$1 build failed-------------------" | ||
| exit 1 | ||
| else | ||
| echo "--------------$1 build success-------------------" | ||
| fi | ||
| } | ||
|
|
||
| # install dependencies | ||
| yum install -y cmake rpcgen glib2-devel gnutls-devel libudev-devel libpciaccess-devel libxml2-devel libtirpc-devel \ | ||
| yajl-devel librbd-devel pixman-devel python3-docutils meson openssl openssl-devel autoconf automake libtool \ | ||
| python3-pyelftools libmlx5 libatomic unbound lubunwind pkgconfig rdma-core-devel libcap* python3-pkgconfig | ||
|
|
||
| # 1 install dpak | ||
| rpm -e `rpm -qa | grep in220-sdk` --nodeps | ||
| rpm -e `rpm -qa | grep dpak | grep ovs` --nodeps | ||
| rpm -ivh ${package_dir}/dpak*.rpm | ||
|
|
||
| exit_or_not 'install dpak' | ||
|
|
||
| #2 install dpdk | ||
| rpm -e --nodeps dpdk-21.11-1.aarch64.rpm | ||
| rpm -e --nodeps dpdk-devel-21.11-1.aarch64.rpm | ||
|
|
||
| rpm -ivh ${package_dir}/dpdk-21.11-1.aarch64.rpm | ||
| rpm -ivh ${package_dir}/dpdk-devel-21.11-1.aarch64.rpm | ||
| ldconfig | ||
|
|
||
| # 3 install openvswitch(ovs) | ||
| rpm -e --nodeps openvswitch-2.12.0.13.oe1.arrch64 | ||
| rpm -e --nodeps openvswitch-devel-2.12.0.13.oe1.arrch64 | ||
|
|
||
| rpm -ivh ${package_dir}/openvswitch-2.14.2-1.aarch64.rpm | ||
| rpm -ivh ${package_dir}/openvswitch-devel-2.14.2-1.arrch64.rpm | ||
|
|
||
| systemctl restart rsyslog | ||
| # create ovs log directory | ||
| mkdir -p /var/log/openvswitch | ||
| ln -s /usr/lib64/libevent-2.1.so.6 /usr/lib64/libevent-2.0.so.5 | ||
| ln -s /usr/lib64/libcrypto.so.1.1 /usr/lib64/libcrypto.so.10 | ||
|
|
||
| # configure OVS startup items | ||
| hwoff-pf-pci=$(hinicadm3 info | grep NIC | cut -c 19-30 | awk 'NR==1') | ||
| service openvswitch start | ||
|
|
||
| ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true | ||
| ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-extra="--iova-mode=pa" | ||
| ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0x6 | ||
| ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x6 | ||
| ovs-vsctl --no-wait set Open_vSwitch . other_config:offload-cpu-mask=0x18 | ||
| ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="8192" | ||
| ovs-vsctl --no-wait set Open_vSwitch . other_config:hwoff-pf-pci=${hwoff-pf-pci} | ||
| ovs-vsctl --no-wait set Open_vSwitch . other_config:hw-offload=true | ||
| service openvswitch restart | ||
|
|
||
| virtio_1=$(hinicadm3 info | grep VirtIO | awk -f "(" '{print $1}' | awk -f "--------" '{print $2}' sed -n '1p') | ||
| virtio_2=$(hinicadm3 info | grep VirtIO | awk -f "(" '{print $1}' | awk -f "--------" '{print $2}' sed -n '2p') | ||
|
|
||
| echo 2 > /sys/bus/pci/devices/${virtio_1}/sriov_numvfs | ||
| echo 2 > /sys/bus/pci/devices/${virtio_2}/sriov_numvfs | ||
|
|
||
| service openvswitch restart | ||
| sleep 5s | ||
|
|
||
| # resolve MAC address conflicts | ||
|
|
||
| vfs=$(lspci | grep Virtio | awk '{print $1}' |sed -n '3,6p') | ||
| vf_pci_part=$(echo ${virtio_1} | cut -c 1-8) | ||
| ovs-vsctl del-br br0 | ||
| for i in {1..4};do ovs-vsctl del-port vf$i;done | ||
|
|
||
| ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev | ||
|
|
||
| for i in {1..4};do | ||
| random_1=$(head -20 /dev/urandom | cksum | cut -c 1) | ||
| random_2=$(head -20 /dev/urandom | cksum | cut -c 1) | ||
| random_3=$(head -20 /dev/urandom | cksum | cut -c 1) | ||
| vf_pci=$(echo ${vfs} | awk "{print \$${i}}") | ||
| ovs-vsctl add-port br0 vf${i} --set interface vf${i} type=dpdk \ | ||
| options:dpdk-devargs=0000:${vf_pci},vf_mac=02:05:0${random_1}:0${random_2}:0${random_3}:0c | ||
| done |
96 changes: 96 additions & 0 deletions
96
Development_framework/dpak-sdk/build/ovs_build/virtio_envable.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| #!/usr/bin/bash | ||
|
|
||
| # Copyright 2022 Huawei Technologies Co., Ltd | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
|
|
||
| VIRTIO_DEV_BDF1="" | ||
| VIRTIO_DEV_BDF2="" | ||
| UP_PORT="" | ||
|
|
||
| get_bdf(){ | ||
| a=$(hinicadm3 info | grep VirtIO) | ||
| b1=${a%%(*} | ||
| VIRTIO_DEV_BDF1=${b1##*-} | ||
|
|
||
| b2=${a%(*} | ||
| VIRTIO_DEV_BDF2=${b2##*-} | ||
| if [ "${VIRTIO_DEV_BDF1:4:1}" != ":" ] || [ "${VIRTIO_DEV_BDF2:4:1}" != ":" ] || \ | ||
| [ "${VIRTIO_DEV_BDF1:7:1}" != ":" ]|| [ "${VIRTIO_DEV_BDF2:7:1}" != ":" ] || \ | ||
| [ "${VIRTIO_DEV_BDF1:10:1}" != "." ] || [ "${VIRTIO_DEV_BDF2:10:1}" != "." ];then | ||
| echo "[$(date +%Y-%m-%d\ %H:%M:%S)][ERROR] Get bdf failed. Check pci dev or hinicadm3 tool." | ||
| exit -1 | ||
| fi | ||
| } | ||
|
|
||
| get_up_port(){ | ||
| UP_PORT=$(find /sys -name remove | grep ${VIRTIO_DEV_BDF1} | awk -F '/' '{print $(NF-2)}') | ||
| } | ||
|
|
||
| delete_dev_spec(){ | ||
| echo 1 >/sys/bus/pci/devices/$1/remove | ||
| if [ $? -ne 0 ];then | ||
| echo "[$(date +%Y-%m-%d\ %H:%M:%S)][ERROR] Device [$1] remove err!" | ||
| exit -1 | ||
| fi | ||
| } | ||
|
|
||
| delete_dev(){ | ||
| delete_dev_spec ${VIRTIO_DEV_BDF1} | ||
| delete_dev_spec ${VIRTIO_DEV_BDF2} | ||
| } | ||
|
|
||
| check_dev(){ | ||
| ls /sys/bus/pci/devices/$1 >> /dev/null | ||
| if [ $? -ne 0 ];then | ||
| echo "[$(date +%Y-%m-%d\ %H:%M:%S)][ERROR] Rescan [${UP_PORT}] err, $1 is not detected." | ||
| exit -1 | ||
| fi | ||
| } | ||
|
|
||
| rescan_dev(){ | ||
| echo 1 > /sys/bus/pci/devices/${UP_PORT}/rescan | ||
| if [ $? -ne 0 ];then | ||
| echo "[$(date +%Y-%m-%d\ %H:%M:%S)][ERROR] Rescan [${UP_PORT}] err!" | ||
| exit -1 | ||
| fi | ||
| check_dev ${VIRTIO_DEV_BDF1} | ||
| check_dev ${VIRTIO_DEV_BDF2} | ||
| } | ||
|
|
||
| run_cmd(){ | ||
| $1 >> /dev/null | ||
| if [ $? -ne 0 ];then | ||
| echo "[$(date +%Y-%m-%d\ %H:%M:%S)][ERROR] Run cmd [$1] err!" | ||
| exit -1 | ||
| else | ||
| echo "[$(date +%Y-%m-%d\ %H:%M:%S)][INFO] $1." | ||
| fi | ||
| } | ||
|
|
||
| reprobe_driver(){ | ||
| run_cmd 'modprobe -r virtio-net' | ||
| run_cmd 'modprobe -r virtio-pci' | ||
| run_cmd 'modprobe virtio-pci' | ||
| run_cmd 'modprobe virtio-net' | ||
| } | ||
|
|
||
| hinicadm3 virtio_dev -i hinic0 -d net -s 0 -f 0 | ||
| get_bdf | ||
| get_up_port | ||
| delete_dev | ||
| rescan_dev | ||
| reprobe_driver | ||
|
|
||
| echo "[$(date +%Y-%m-%d\ %H:%M:%S)][INFO] Finished." |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.