Skip to content

Commit

Permalink
Merging changes from Dev branch into Device client integration (#372)
Browse files Browse the repository at this point in the history
* Introducing IoT Hub dataplane RBAC support + various improvements (#341)

* Introducing IoT Hub dataplane RBAC support.
* IoT Hub rbac support for most commands.
* Removed deprecated nested IoT edge artifacts.
* Removed deprecated show-connection-string artifacts.
* --auth-type supports configurable defaults.
* Improved IoT Hub test infrastructure.
* Significant refactor of IoT Hub tests.
* Various additional improvements.
* All warnings from tests are now shown.
* Uamqp integration with jwt auth.
* Modify HISTORY.rst

* Iotc command versioning (#340)

* add support for v1 and preview routes
* update history file
* address review comments
* lint fixes

* Add warning for qos deprecation and update contributing guide (#342)

* Add warning for qos deprecation and update contributing guide
* removing version from deprecate_info
* Integration tests command update in Contributing guide

* Integrate TQDM to show progress bar when simulator sends d2c messages

* remove unused loop varable

* Update progress bar descriptionb

* Iotc command ga (#348)

* remove preview tags
* history updates

* Use enum value instead of literal str. (#349)

* Managed identity support for device-identity import and export  (#344)

* Fix for new identity parameter format
* test updates
* better differentiation of storage vars
* updated to remove *all* user-identitites after storage tests until CLI core is patched

* Update azext_metadata.json (#351)

* Increment version to v0.10.13

* Update README.md

* Update HISTORY.rst

* update twin reported properties during simulation

* update unit tests

* Add dataplane reset (#352)

* pr comments
* update dt reset to only support deleting both
* Remove unused import

* styling updates

* C2D messaging improvements. (#354)

* Remove hiding of warnings from sample pytest.ini

* Digital Twin wait commands (#345)

* update to use wait
* fix error status code
* int testing
* update help

* Add Identity Storage Account ID param to sentinel values (#355)

* Using SDK Listener for Twin properties update

* Module identity renew key (#356)

* initial changes
* update params, help
* add missing module to test
* Word change

* Update README.md

* merge from remote

* Pipeline updates (#359)

* Pipeline updates

Added nightly build pipeline

Added template to run tests against minimum supported AZ CLI

* Pipeline parameter refactoring

* Moved deprecated ubuntu images to `ubuntu-latest`

Co-authored-by: Paymaun <[email protected]>

* Structured mqtt formatting and eliminate dependency on six

* remove indent property not needed any more

* Check for conditionals before running test jobs (#363)

* Update d2c and simulate commands to return errors for non SaS devices - update help to reflect the change (#346)

* Add warning for qos deprecation and update contributing guide

* removing version from deprecate_info

* Integration tests command update in Contributing guide

* Update help message for d2c command and mqtt simulation to indicate only sas auth is supported

* Checks and error messages when non SAS devices are used for MQTT operations

* Indentation update in help file

* fix styling

* Adding enum and updating test

* Addressed comments

* Support C2D Message decoding and Add TQDM for HTTP simulation

* Update MQTT operations to run on web sockets

* Remove duplicate test already in dev branch

* Device connection is automatic now

* Styling update

Co-authored-by: Paymaun <[email protected]>
Co-authored-by: valluriraj <[email protected]>
Co-authored-by: Ryan K <[email protected]>
Co-authored-by: Paymaun Heidari <[email protected]>
Co-authored-by: vilit1 <[email protected]>
  • Loading branch information
6 people authored Jun 10, 2021
1 parent 6a3d477 commit 0cbe549
Show file tree
Hide file tree
Showing 100 changed files with 6,895 additions and 4,491 deletions.
51 changes: 37 additions & 14 deletions .azure-devops/create-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,28 @@ pr: none

trigger: none

variables:
pythonVersion: '3.6.x'
architecture: 'x64'
parameters:
- name: pythonVersion
type: string
default: '3.6.x'
values:
- 3.6.x
- 3.9.x
- name: architecture
type: string
default: 'x64'
- name: 'testCentral'
type: boolean
default: true
- name: 'testADT'
type: boolean
default: true
- name: 'testDPS'
type: boolean
default: true
- name: 'testHub'
type: boolean
default: true

stages:
- stage: 'build'
Expand All @@ -13,27 +32,27 @@ stages:

- job: 'Build_Publish_Azure_IoT_CLI_Extension'
pool:
vmImage: 'Ubuntu-16.04'
vmImage: 'ubuntu-latest'

steps:
- task: UsePythonVersion@0
inputs:
versionSpec: $(pythonVersion)
architecture: $(architecture)
versionSpec: ${{ parameters.pythonVersion }}
architecture: ${{ parameters.architecture }}

- template: templates/setup-ci-machine.yml

- template: templates/build-publish-azure-iot-cli-extension.yml

- job: 'Build_Publish_Azure_CLI_Test_SDK'
pool:
vmImage: 'Ubuntu-16.04'
vmImage: 'ubuntu-latest'

steps:
- task: UsePythonVersion@0
inputs:
versionSpec: $(pythonVersion)
architecture: $(architecture)
versionSpec: ${{ parameters.pythonVersion }}
architecture: ${{ parameters.architecture }}

- template: templates/setup-ci-machine.yml

Expand All @@ -45,19 +64,20 @@ stages:
steps:
- template: templates/setup-dev-test-env.yml
parameters:
pythonVersion: $(pythonVersion)
architecture: $(architecture)
pythonVersion: ${{ parameters.pythonVersion }}
architecture: ${{ parameters.architecture }}

- template: templates/install-and-record-version.yml

- stage: 'test'
displayName: 'Run tests'
pool:
vmImage: 'Ubuntu-16.04'
vmImage: 'ubuntu-latest'
dependsOn: build
jobs:
- job: 'testCentral'
displayName: 'Test IoT Central'
condition: eq('${{ parameters.testCentral }}', true)
steps:
- template: templates/run-tests.yml
parameters:
Expand All @@ -66,6 +86,7 @@ stages:

- job: 'testADT'
displayName: 'Test Azure DigitalTwins'
condition: eq('${{ parameters.testADT }}', true)
steps:
- template: templates/run-tests.yml
parameters:
Expand All @@ -74,6 +95,7 @@ stages:

- job: 'testDPS'
displayName: 'Test DPS'
condition: eq('${{ parameters.testDPS }}', true)
steps:
- template: templates/run-tests.yml
parameters:
Expand All @@ -82,6 +104,7 @@ stages:

- job: 'testHub'
displayName: 'Test IoT Hub'
condition: eq('${{ parameters.testHub }}', true)
steps:
- template: templates/run-tests.yml
parameters:
Expand All @@ -105,8 +128,8 @@ stages:
steps:
- template: templates/calculate-code-coverage.yml
parameters:
pythonVersion: $(pythonVersion)
architecture: $(architecture)
pythonVersion: ${{ parameters.pythonVersion }}
architecture: ${{ parameters.architecture }}

- stage: 'release'
displayName: 'Stage GitHub release'
Expand Down
22 changes: 11 additions & 11 deletions .azure-devops/merge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:

- job: 'build_and_publish_azure_iot_cli_ext'
pool:
vmImage: 'Ubuntu-16.04'
vmImage: 'ubuntu-latest'

steps:
- task: UsePythonVersion@0
Expand All @@ -31,7 +31,7 @@ jobs:

- job: 'build_and_publish_azure_cli_test_sdk'
pool:
vmImage: 'Ubuntu-16.04'
vmImage: 'ubuntu-latest'

steps:
- task: UsePythonVersion@0
Expand All @@ -45,7 +45,7 @@ jobs:
- job: 'run_unit_tests_ubuntu'
dependsOn: [ 'build_and_publish_azure_iot_cli_ext', 'build_and_publish_azure_cli_test_sdk']
pool:
vmImage: 'Ubuntu-16.04'
vmImage: 'ubuntu-latest'
strategy:
matrix:
Python36:
Expand All @@ -62,8 +62,8 @@ jobs:
- template: templates/run-tests.yml
parameters:
pythonVersion: '$(python.version)'
runUnitTests: 'true'
runIntTests: 'false'
runUnitTests: true
runIntTests: false

- job: 'run_unit_tests_macOs'
dependsOn: ['build_and_publish_azure_iot_cli_ext', 'build_and_publish_azure_cli_test_sdk']
Expand All @@ -74,8 +74,8 @@ jobs:
- template: templates/run-tests.yml
parameters:
pythonVersion: '3.8.x'
runUnitTests: 'true'
runIntTests: 'false'
runUnitTests: true
runIntTests: false

- template: templates/calculate-code-coverage.yml

Expand All @@ -93,13 +93,13 @@ jobs:
- template: templates/run-tests.yml
parameters:
pythonVersion: '3.8.x'
runUnitTests: 'true'
runIntTests: 'false'
runUnitTests: true
runIntTests: false

- job: 'run_style_check'
dependsOn: ['build_and_publish_azure_iot_cli_ext', 'build_and_publish_azure_cli_test_sdk']
pool:
vmImage: 'Ubuntu-16.04'
vmImage: 'ubuntu-latest'

steps:
- task: UsePythonVersion@0
Expand All @@ -124,7 +124,7 @@ jobs:
dependsOn: ['build_and_publish_azure_iot_cli_ext']
displayName: 'Evaluate IoT extension command table'
pool:
vmImage: 'Ubuntu-16.04'
vmImage: 'ubuntu-latest'

steps:
- task: UsePythonVersion@0
Expand Down
88 changes: 88 additions & 0 deletions .azure-devops/nightly.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Run nightly at midnight.
schedules:
- cron: "0 0 * * *"
displayName: Nightly Integration Build
branches:
include:
- dev

variables:
pythonVersion: '3.6.x'
architecture: 'x64'

stages:
- stage: 'build'
displayName: 'Build and Publish Artifacts'
jobs:

- job: 'Build_Publish_Azure_IoT_CLI_Extension'
pool:
vmImage: 'ubuntu-latest'

steps:
- task: UsePythonVersion@0
inputs:
versionSpec: $(pythonVersion)
architecture: $(architecture)

- template: templates/setup-ci-machine.yml

- template: templates/build-publish-azure-iot-cli-extension.yml

- job: 'Build_Publish_Azure_CLI_Test_SDK'
pool:
vmImage: 'ubuntu-latest'

steps:
- task: UsePythonVersion@0
inputs:
versionSpec: $(pythonVersion)
architecture: $(architecture)

- template: templates/setup-ci-machine.yml

- template: templates/build-publish-azure-cli-test-sdk.yml

- job: 'recordVersion'
displayName: 'Install and verify version'
dependsOn: [Build_Publish_Azure_IoT_CLI_Extension, Build_Publish_Azure_CLI_Test_SDK]
steps:
- template: templates/setup-dev-test-env.yml
parameters:
pythonVersion: $(pythonVersion)
architecture: $(architecture)

- template: templates/install-and-record-version.yml

- stage: 'test'
displayName: 'Run all tests'
pool:
vmImage: 'ubuntu-latest'
dependsOn: build
jobs:
- job: 'azEdge'
displayName: 'Test against edge AZ CLI'
steps:
- template: templates/nightly-tests.yml
parameters:
azureCLIVersion: 'edge'
- job: 'azMin'
dependsOn: 'azEdge'
displayName: 'Test against minimum supported AZ CLI'
steps:
- template: templates/nightly-tests.yml
parameters:
azureCLIVersion: 'min'

- stage: 'kpi'
displayName: 'Build KPIs'
dependsOn: [build, test]
jobs:
- job: 'calculateCodeCoverage'
displayName: 'Calculate distributed code coverage'
steps:
- template: templates/calculate-code-coverage.yml
parameters:
pythonVersion: $(pythonVersion)
architecture: $(architecture)

14 changes: 14 additions & 0 deletions .azure-devops/templates/install-azure-cli-min.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
steps:
- task: PythonScript@0
displayName: 'Check minimum supported version of Azure CLI'
inputs:
scriptSource: 'inline'
script: |
import json
with open("$(System.DefaultWorkingDirectory)/azext_iot/azext_metadata.json") as f:
metadata = json.load(f)
version = metadata['azext.minCliCoreVersion']
print('##vso[task.setvariable variable=min_cli_version]{}'.format(version))
- bash: |
pip install azure-cli==$(min_cli_version)
displayName: "Install minimum supported CLI version"
55 changes: 55 additions & 0 deletions .azure-devops/templates/nightly-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
parameters:
- name: pythonVersion
type: string
default: '3.6.x'
- name: architecture
type: string
default: 'x64'
- name: azureCLIVersion
type: string
default: released
values:
- min
- released
- edge

steps:
- template: setup-dev-test-env.yml
parameters:
architecture: ${{ parameters.architecture }}
pythonVersion: ${{ parameters.pythonVersion }}
azureCLIVersion: ${{ parameters.azureCLIVersion }}

- template: set-testenv-sentinel.yml

- script: |
pytest -vv azext_iot/tests -k "_unit" --cov=azext_iot --cov-config .coveragerc --junitxml=junit/test-iotext-unit.xml
displayName: 'All unit tests'
env:
COVERAGE_FILE: .coverage.all
- task: AzureCLI@2
continueOnError: true
displayName: 'All integration tests'
inputs:
azureSubscription: az-cli-nightly
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
export COVERAGE_FILE=.coverage.all
pytest -vv azext_iot/tests -k "_int" --cov=azext_iot --cov-config .coveragerc --junitxml=junit/test-iotext-int.xml
- task: PublishBuildArtifacts@1
inputs:
pathToPublish: .coverage.all
publishLocation: 'Container'
artifactName: 'coverage'

- task: PublishTestResults@2
condition: succeededOrFailed()
displayName: 'Publish Test Results'
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/test-*.xml'
testRunTitle: 'Publish test results for Python ${{ parameters.pythonVersion }} on OS $(Agent.OS)'
searchFolder: '$(System.DefaultWorkingDirectory)'
Loading

0 comments on commit 0cbe549

Please sign in to comment.