Enable integration tests to run in parallel (#388)

* Introducing IoT Hub dataplane RBAC support + various improvements (#341)

* Introducing IoT Hub dataplane RBAC support.
* IoT Hub rbac support for most commands.
* Removed deprecated nested IoT edge artifacts.
* Removed deprecated show-connection-string artifacts.
* --auth-type supports configurable defaults.
* Improved IoT Hub test infrastructure.
* Significant refactor of IoT Hub tests.
* Various additional improvements.
* All warnings from tests are now shown.
* Uamqp integration with jwt auth.
* Modify HISTORY.rst

* Iotc command versioning (#340)

* add support for v1 and preview routes
* update history file
* address review comments
* lint fixes

* Add warning for qos deprecation and update contributing guide (#342)

* Add warning for qos deprecation and update contributing guide
* removing version from deprecate_info
* Integration tests command update in Contributing guide

* Iotc command ga (#348)

* remove preview tags
* history updates

* Use enum value instead of literal str. (#349)

* Managed identity support for device-identity import and export  (#344)

* Fix for new identity parameter format
* test updates
* better differentiation of storage vars
* updated to remove *all* user-identitites after storage tests until CLI core is patched

* Update azext_metadata.json (#351)

* Increment version to v0.10.13

* Update README.md

* Update HISTORY.rst

* Add dataplane reset (#352)

* pr comments
* update dt reset to only support deleting both
* Remove unused import

* C2D messaging improvements. (#354)

* Remove hiding of warnings from sample pytest.ini

* Digital Twin wait commands (#345)

* update to use wait
* fix error status code
* int testing
* update help

* Add Identity Storage Account ID param to sentinel values (#355)

* Module identity renew key (#356)

* initial changes
* update params, help
* add missing module to test
* Word change

* Update README.md

* Pipeline updates (#359)

* Pipeline updates

Added nightly build pipeline

Added template to run tests against minimum supported AZ CLI

* Pipeline parameter refactoring

* Moved deprecated ubuntu images to `ubuntu-latest`

Co-authored-by: Paymaun <[email protected]>

* Check for conditionals before running test jobs (#363)

* Update d2c and simulate commands to return errors for non SaS devices - update help to reflect the change (#346)

* Add warning for qos deprecation and update contributing guide

* removing version from deprecate_info

* Integration tests command update in Contributing guide

* Update help message for d2c command and mqtt simulation to indicate only sas auth is supported

* Checks and error messages when non SAS devices are used for MQTT operations

* Indentation update in help file

* fix styling

* Adding enum and updating test

* Addressed comments

* Update simulate message (#371)

IoTHub is not a MQTT Broker

* Make gitHubConnection a variable. (#373)

* Iotc updates (#368)

* initial commit

* add integration tests.

* history file update. lint fixes

* history updates

* Update HISTORY.rst

* address review comments

Co-authored-by: Ryan K <[email protected]>

* Edge deployment validation changes. (#382)

* Only system module content is validated by default.
* The target schema used is with respect to "schemaVersion" in
  "properties.desired".
* Any schema defintion processing issues will skip validation.

* Couple linter fixes. (#384)

* Extend generate token (#375)

* generate sas token with all parsers
* style fixes
* update param, parser enum
* updates with respect to pr
* updated error message

* Roll pipeline ubuntu versions back to Ubuntu-16.04 (#386)

* Support for upcoming changes in Track 2 SDKs (#383)

* Initial DPS track 2 breaking change fixes
* Test update to patch azure cli core Profile.get_subscription_id
* revert change of unit test subscription id

* Remove six (#387)

Co-authored-by: Paymaun <[email protected]>

* Separate IoT Hub per test suite to enable parallel test execution

* Addressed comments

* minor update

* test environment pipeline updates

* Update help for --login and --hub-name (#389)

* Update mock import. (#391)

* Refactoring and addressing comments

* Update setup.py

* fstrings fix

* Formatting updates to tests

* Create one hub per test suite and provide Data contributor role to user

* Account refresh and test class parallel run

* remove commented parts of test

* lint updates

* Merge with dev

* Azure devops merge fix

* Run integration tests in parallel

* Update to tests

Co-authored-by: Paymaun <[email protected]>
Co-authored-by: valluriraj <[email protected]>
Co-authored-by: Ryan K <[email protected]>
Co-authored-by: Paymaun Heidari <[email protected]>
Co-authored-by: vilit1 <[email protected]>
Co-authored-by: Ricardo Minguez (Rido) <[email protected]>

.azure-devops/templates/set-testenv-sentinel.yml

@@ -14,8 +14,9 @@ steps:
             "azext_iot_testrg": os.environ.get("AZEXT_IOT_TESTRG", sentinel_value),
             "azext_iot_testhub": os.environ.get("AZEXT_IOT_TESTHUB", sentinel_value),
             "azext_iot_testdps": os.environ.get("AZEXT_IOT_TESTDPS", sentinel_value),
-            "azext_iot_teststorageuri": os.environ.get("AZEXT_IOT_TESTSTORAGEURI", sentinel_value),
-            "azext_iot_identity_teststorageid": os.environ.get("AZEXT_IOT_IDENTITY_TESTSTORAGEID", sentinel_value),
+            "azext_iot_identity_teststorageid": os.environ.get("AZEXT_IOT_IDENTITY_TESTSTORAGEID", sentinel_value),           
+            "azext_iot_teststorageaccount": os.environ.get("AZEXT_IOT_TESTSTORAGEACCOUNT", sentinel_value),
+            "azext_iot_teststoragecontainer": os.environ.get("AZEXT_IOT_TESTSTORAGECONTAINER", sentinel_value),
             "azext_iot_central_app_id": os.environ.get("AZEXT_IOT_CENTRAL_APP_ID", sentinel_value),
             "azext_dt_ep_eventgrid_topic": os.environ.get("AZEXT_DT_EP_EVENTGRID_TOPIC", sentinel_value),
             "azext_dt_ep_servicebus_namespace": os.environ.get("AZEXT_DT_EP_SERVICEBUS_NAMESPACE", sentinel_value),
@@ -39,8 +40,9 @@ steps:
       AZEXT_IOT_TESTRG: $(azext_iot_testrg)
       AZEXT_IOT_TESTHUB: $(azext_iot_testhub)
       AZEXT_IOT_TESTDPS: $(azext_iot_testdps)
-      AZEXT_IOT_TESTSTORAGEURI: $(azext_iot_teststorageuri)
       AZEXT_IOT_IDENTITY_TESTSTORAGEID: $(azext_iot_identity_teststorageid)
+      AZEXT_IOT_TESTSTORAGEACCOUNT: $(azext_iot_teststorageaccount)
+      AZEXT_IOT_TESTSTORAGECONTAINER: $(azext_iot_teststoragecontainer)
       AZEXT_IOT_CENTRAL_APP_ID: $(azext_iot_central_app_id)
       AZEXT_DT_EP_EVENTGRID_TOPIC: $(azext_dt_ep_eventgrid_topic)
       AZEXT_DT_EP_SERVICEBUS_NAMESPACE: $(azext_dt_ep_servicebus_namespace)

CONTRIBUTING.md

@@ -128,12 +128,11 @@ You can either manually set the environment variables or use the `pytest.ini.exa
     azext_iot_testrg="Resource Group that contains your IoT Hub"
     azext_iot_testhub="IoT Hub Name"
     azext_iot_testdps="IoT Hub DPS Name"
-    azext_iot_teststorageuri="Blob Container SAS Uri"
     azext_iot_identity_teststorageid="Storage Account ID"
+    azext_iot_teststorageaccount="Storage Account Name"
+    azext_iot_teststoragecontainer="Blob container name"
 ```
 
-`azext_iot_teststorageuri` is optional and only required when you want to test device export and file upload functionality. You can generate a SAS Uri for your Blob container using the [Azure Storage Explorer](https://azure.microsoft.com/en-us/features/storage-explorer/).  You must also configure your IoT Hub's File Upload storage container via the Azure Portal for this test to pass.
-
 `azext_iot_identity_teststorageid` is optional and only required when you want to test Identity-Based device export and file upload functionality. During this test, your hub will be assigned a System-Assigned AAD identity, and will be granted the role of "Storage Blob Data Contributor" on the storage account you provide. Both the hub's identity and the RBAC role will be removed once the test completes.
 
 ##### IoT Hub

azext_iot/operations/_mqtt.py

@@ -109,18 +109,11 @@ def twin_patch_handler(self, patch):
             self.device_client.patch_twin_reported_properties(modified_properties)
 
     def execute(self, data, properties={}, publish_delay=2, msg_count=100):
-        from azext_iot.operations.hub import _iot_device_twin_update
         from tqdm import tqdm
 
         try:
             if self.init_reported_properties:
-                twin_properties = {
-                    "properties": {
-                        "desired": self.init_reported_properties
-                    }
-                }
-
-                _iot_device_twin_update(self.target, self.device_id, twin_properties)
+                self.device_client.patch_twin_reported_properties(self.init_reported_properties)
 
             for _ in tqdm(range(msg_count), desc='Device simulation in progress'):
                 self.send_d2c_message(message_text=data.generate(True), properties=properties)

azext_iot/tests/__init__.py

@@ -7,18 +7,35 @@
 import sys
 import io
 import os
+import pytest
+import time
 
 from azext_iot.tests.iothub import DATAPLANE_AUTH_TYPES
 from azure.cli.testsdk import LiveScenarioTest
 from contextlib import contextmanager
 from typing import List
+from azext_iot.tests.settings import DynamoSettings, ENV_SET_TEST_IOTHUB_REQUIRED, ENV_SET_TEST_IOTHUB_OPTIONAL
+from azext_iot.tests.generators import generate_generic_id
+from azure.cli.core._profile import Profile
+from azure.cli.core.mock import DummyCli
 
 PREFIX_DEVICE = "test-device-"
 PREFIX_EDGE_DEVICE = "test-edge-device-"
 PREFIX_DEVICE_MODULE = "test-module-"
 PREFIX_CONFIG = "test-config-"
 PREFIX_EDGE_CONFIG = "test-edgedeploy-"
 PREFIX_JOB = "test-job-"
+USER_ROLE = "IoT Hub Data Contributor"
+DEFAULT_CONTAINER = "devices"
+
+settings = DynamoSettings(req_env_set=ENV_SET_TEST_IOTHUB_REQUIRED, opt_env_set=ENV_SET_TEST_IOTHUB_OPTIONAL)
+
+ENTITY_RG = settings.env.azext_iot_testrg
+ENTITY_NAME = settings.env.azext_iot_testhub if settings.env.azext_iot_testhub else "test-hub-" + generate_generic_id()
+STORAGE_CONTAINER = (
+    settings.env.azext_iot_teststoragecontainer if settings.env.azext_iot_teststoragecontainer else DEFAULT_CONTAINER
+)
+ROLE_ASSIGNMENT_REFRESH_TIME = 30
 
 
 @contextmanager
@@ -71,16 +88,59 @@ def command_execute_assert(self, command, asserts):
 
 
 class IoTLiveScenarioTest(CaptureOutputLiveScenarioTest):
-    def __init__(self, test_scenario, entity_name, entity_rg):
+    def __init__(self, test_scenario):
         assert test_scenario
-        assert entity_name
-        assert entity_rg
-
-        self.entity_name = entity_name
-        self.entity_rg = entity_rg
-
+        self.entity_rg = ENTITY_RG
+        self.entity_name = ENTITY_NAME
         super(IoTLiveScenarioTest, self).__init__(test_scenario)
 
+        if not settings.env.azext_iot_testhub:
+            hubs_list = self.cmd(
+                '''iot hub list -g "{}"'''.format(self.entity_rg)
+            ).get_output_in_json()
+
+            target_hub = None
+            for hub in hubs_list:
+                if hub["name"] == self.entity_name:
+                    target_hub = hub
+                    break
+
+            if not target_hub:
+                if settings.env.azext_iot_teststorageaccount:
+                    storage_account_connenction = self.cmd(
+                        "storage account show-connection-string --name {}".format(
+                            settings.env.azext_iot_teststorageaccount
+                        )
+                    ).get_output_in_json()
+
+                    self.cmd(
+                        "iot hub create --name {} --resource-group {} --fc {} --fcs {} --sku S1 ".format(
+                            self.entity_name, self.entity_rg,
+                            STORAGE_CONTAINER, storage_account_connenction["connectionString"]
+                        )
+                    )
+                else:
+                    self.cmd(
+                        "iot hub create --name {} --resource-group {} --sku S1 ".format(
+                            self.entity_name, self.entity_rg
+                        )
+                    )
+
+                profile = Profile(cli_ctx=DummyCli())
+                user_id = profile.get_current_account_user()
+                new_hub = self.cmd(
+                    "iot hub show -n {} -g {}".format(self.entity_name, self.entity_rg)
+                ).get_output_in_json()
+
+                # assign IoT Hub Data Contributor role to current user
+                self.cmd(
+                    '''role assignment create --assignee {} --role "{}" --scope "{}"'''.format(
+                        user_id, USER_ROLE, new_hub["id"]
+                    )
+                )
+                profile.refresh_accounts()
+                time.sleep(ROLE_ASSIGNMENT_REFRESH_TIME)
+
         self.region = self.get_region()
         self.connection_string = self.get_hub_cstring()
 
@@ -191,6 +251,16 @@ def set_cmd_auth_type(self, command: str, auth_type: str) -> str:
 
         return f"{command} --auth-type {auth_type}"
 
+    @pytest.fixture(scope='class', autouse=True)
+    def tearDownSuite(self):
+        yield None
+        if not settings.env.azext_iot_testhub:
+            self.cmd(
+                "iot hub delete --name {} --resource-group {}".format(
+                    ENTITY_NAME, ENTITY_RG
+                )
+            )
+
 
 def disable_telemetry(test_function):
     def wrapper(*args, **kwargs):

azext_iot/tests/dps/test_iot_dps_int.py

@@ -12,11 +12,12 @@
 from azext_iot.common.utility import generate_key
 from azext_iot.iothub.providers.discovery import IotHubDiscovery
 from azext_iot.tests.settings import Setting
+from azext_iot.tests.generators import generate_generic_id
 
 # Set these to the proper IoT Hub DPS, IoT Hub and Resource Group for Integration Tests.
 dps = os.environ.get("azext_iot_testdps")
 rg = os.environ.get("azext_iot_testrg")
-hub = os.environ.get("azext_iot_testhub")
+hub = os.environ.get("azext_iot_testhub") if os.environ.get("azext_iot_testhub") else "test-hub-" + generate_generic_id()
 
 if not all([dps, rg, hub]):
     raise ValueError(
@@ -101,6 +102,13 @@ def __init__(self, test_method):
         self.cmd_shell = Setting()
         setattr(self.cmd_shell, "cli_ctx", self.cli_ctx)
 
+        if hub != os.environ.get("azext_iot_testhub"):
+            self.cmd(
+                "iot hub create --name {} --resource-group {} --sku S1".format(
+                    hub, rg
+                )
+            )
+
         _ensure_dps_hub_link(self, dps, rg, hub)
 
         output_dir = os.getcwd()
@@ -125,6 +133,13 @@ def __del__(self):
         if os.path.exists(cert_path):
             os.remove(cert_path)
 
+        if hub != os.environ.get("azext_iot_testhub"):
+            self.cmd(
+                "iot hub delete --name {} --resource-group {}".format(
+                    hub, rg
+                )
+            )
+
     def test_dps_compute_device_key(self):
         device_key = self.cmd(
             'az iot dps compute-device-key --key "{}" '