Merging changes from Dev branch into Device client integration (Azure…

…#372)

* Introducing IoT Hub dataplane RBAC support + various improvements (Azure#341)

* Introducing IoT Hub dataplane RBAC support.
* IoT Hub rbac support for most commands.
* Removed deprecated nested IoT edge artifacts.
* Removed deprecated show-connection-string artifacts.
* --auth-type supports configurable defaults.
* Improved IoT Hub test infrastructure.
* Significant refactor of IoT Hub tests.
* Various additional improvements.
* All warnings from tests are now shown.
* Uamqp integration with jwt auth.
* Modify HISTORY.rst

* Iotc command versioning (Azure#340)

* add support for v1 and preview routes
* update history file
* address review comments
* lint fixes

* Add warning for qos deprecation and update contributing guide (Azure#342)

* Add warning for qos deprecation and update contributing guide
* removing version from deprecate_info
* Integration tests command update in Contributing guide

* Integrate TQDM to show progress bar when simulator sends d2c messages

* remove unused loop varable

* Update progress bar descriptionb

* Iotc command ga (Azure#348)

* remove preview tags
* history updates

* Use enum value instead of literal str. (Azure#349)

* Managed identity support for device-identity import and export  (Azure#344)

* Fix for new identity parameter format
* test updates
* better differentiation of storage vars
* updated to remove *all* user-identitites after storage tests until CLI core is patched

* Update azext_metadata.json (Azure#351)

* Increment version to v0.10.13

* Update README.md

* Update HISTORY.rst

* update twin reported properties during simulation

* update unit tests

* Add dataplane reset (Azure#352)

* pr comments
* update dt reset to only support deleting both
* Remove unused import

* styling updates

* C2D messaging improvements. (Azure#354)

* Remove hiding of warnings from sample pytest.ini

* Digital Twin wait commands (Azure#345)

* update to use wait
* fix error status code
* int testing
* update help

* Add Identity Storage Account ID param to sentinel values (Azure#355)

* Using SDK Listener for Twin properties update

* Module identity renew key (Azure#356)

* initial changes
* update params, help
* add missing module to test
* Word change

* Update README.md

* merge from remote

* Pipeline updates (Azure#359)

* Pipeline updates

Added nightly build pipeline

Added template to run tests against minimum supported AZ CLI

* Pipeline parameter refactoring

* Moved deprecated ubuntu images to `ubuntu-latest`

Co-authored-by: Paymaun <[email protected]>

* Structured mqtt formatting and eliminate dependency on six

* remove indent property not needed any more

* Check for conditionals before running test jobs (Azure#363)

* Update d2c and simulate commands to return errors for non SaS devices - update help to reflect the change (Azure#346)

* Add warning for qos deprecation and update contributing guide

* removing version from deprecate_info

* Integration tests command update in Contributing guide

* Update help message for d2c command and mqtt simulation to indicate only sas auth is supported

* Checks and error messages when non SAS devices are used for MQTT operations

* Indentation update in help file

* fix styling

* Adding enum and updating test

* Addressed comments

* Support C2D Message decoding and Add TQDM for HTTP simulation

* Update MQTT operations to run on web sockets

* Remove duplicate test already in dev branch

* Device connection is automatic now

* Styling update

Co-authored-by: Paymaun <[email protected]>
Co-authored-by: valluriraj <[email protected]>
Co-authored-by: Ryan K <[email protected]>
Co-authored-by: Paymaun Heidari <[email protected]>
Co-authored-by: vilit1 <[email protected]>

.azure-devops/create-release.yml

@@ -32,7 +32,7 @@ stages:
 
     - job: 'Build_Publish_Azure_IoT_CLI_Extension'
       pool:
-        vmImage: 'Ubuntu-16.04'
+        vmImage: 'ubuntu-latest'
 
       steps:
       - task: UsePythonVersion@0
@@ -46,7 +46,7 @@ stages:
 
     - job: 'Build_Publish_Azure_CLI_Test_SDK'
       pool:
-        vmImage: 'Ubuntu-16.04'
+        vmImage: 'ubuntu-latest'
 
       steps:
       - task: UsePythonVersion@0
@@ -72,7 +72,7 @@ stages:
   - stage: 'test'
     displayName: 'Run tests'
     pool:
-      vmImage: 'Ubuntu-16.04'
+      vmImage: 'ubuntu-latest'
     dependsOn: build
     jobs:
     - job: 'testCentral'

.azure-devops/merge.yml

@@ -22,7 +22,7 @@ jobs:
 
 - job: 'build_and_publish_azure_iot_cli_ext'
   pool:
-    vmImage: 'Ubuntu-16.04'
+    vmImage: 'ubuntu-latest'
 
   steps:
   - task: UsePythonVersion@0
@@ -35,7 +35,7 @@ jobs:
 
 - job: 'build_and_publish_azure_cli_test_sdk'
   pool:
-    vmImage: 'Ubuntu-16.04'
+    vmImage: 'ubuntu-latest'
 
   steps:
   - task: UsePythonVersion@0
@@ -49,7 +49,7 @@ jobs:
 - job: 'run_unit_tests_ubuntu'
   dependsOn: [ 'build_and_publish_azure_iot_cli_ext', 'build_and_publish_azure_cli_test_sdk']
   pool:
-    vmImage: 'Ubuntu-16.04'
+    vmImage: 'ubuntu-latest'
   strategy:
     matrix:
       Python36:
@@ -103,7 +103,7 @@ jobs:
 - job: 'run_style_check'
   dependsOn: ['build_and_publish_azure_iot_cli_ext', 'build_and_publish_azure_cli_test_sdk']
   pool:
-    vmImage: 'Ubuntu-16.04'
+    vmImage: 'ubuntu-latest'
 
   steps:
   - task: UsePythonVersion@0
@@ -128,7 +128,7 @@ jobs:
   dependsOn: ['build_and_publish_azure_iot_cli_ext']
   displayName: 'Evaluate IoT extension command table'
   pool:
-    vmImage: 'Ubuntu-16.04'
+    vmImage: 'ubuntu-latest'
 
   steps:
   - task: UsePythonVersion@0

.azure-devops/nightly.yml

@@ -17,7 +17,7 @@ stages:
 
     - job: 'Build_Publish_Azure_IoT_CLI_Extension'
       pool:
-        vmImage: 'Ubuntu-16.04'
+        vmImage: 'ubuntu-latest'
 
       steps:
       - task: UsePythonVersion@0
@@ -31,7 +31,7 @@ stages:
 
     - job: 'Build_Publish_Azure_CLI_Test_SDK'
       pool:
-        vmImage: 'Ubuntu-16.04'
+        vmImage: 'ubuntu-latest'
 
       steps:
       - task: UsePythonVersion@0
@@ -57,7 +57,7 @@ stages:
   - stage: 'test'
     displayName: 'Run all tests'
     pool:
-      vmImage: 'Ubuntu-16.04'
+      vmImage: 'ubuntu-latest'
     dependsOn: build
     jobs:
     - job: 'azEdge'

CONTRIBUTING.md

@@ -140,13 +140,13 @@ You can either manually set the environment variables or use the `pytest.ini.exa
 
 Execute the following command to run the IoT Hub integration tests:
 
-`pytest azext_iot/tests/iothub/test_iot_ext_int.py`
+`pytest azext_iot/tests/iothub/ -k "_int"`
 
 ##### Device Provisioning Service
 
 Execute the following command to run the IoT Hub DPS integration tests:
 
-`pytest azext_iot/tests/dps/test_iot_dps_int.py`
+`pytest azext_iot/tests/dps/ -k "_int"`
 
 #### Unit and Integration Tests Single Command
 

HISTORY.rst

@@ -3,41 +3,20 @@
 Release History
 ===============
 
-0.10.15
-+++++++++++++++
-
-**IoT Central updates**
-
-* Adds support for listing device groups
-* Adds support for listing roles and get role by id
-
 0.10.14
 +++++++++++++++
 
-**IoT Central updates**
-
-* Adds support to run root/interface level device commands.
-* Adds support to get command history for root/interface level device commands.
-* The --interface-id parameter for commands "device command run" , "device command history" changed to optional.
-
 **IoT Hub updates**
 
 * Fix for "az iot hub c2d-message receive" - the command will use the "ContentEncoding" header value (which indicates the message body encoding)
   or fallback to utf-8 to decode the received message body.
 
-* Addition for "az iot hub generate-sas-token" - the command will allow offline generation of a SAS Token using a connection string.
-
-* Changes to Edge validation for set-modules and edge deployment creation:
-
-  By default only properties of system modules $edgeAgent and $edgeHub are validated against schemas installed with the IoT extension.
-  This can be disabled by using the --no-validation switch.
-
 **Azure Digital Twins updates**
 
 * Addition of the following commands
 
   * az dt reset - Preview command which deletes all data entities from the target instance (models, twins, twin relationships).
-
+  
 
 0.10.13
 +++++++++++++++
@@ -55,7 +34,7 @@ Release History
 * Public API GA update
 
   * Remove preview tag for  api-token, device, device-template, user routes. Default routes use central GA API's.
-  * Add support for preview and 1.0 routes.
+  * Add support for preview and 1.0 routes. 
   * Addition of the optional '--av' argument to specify the version of API for the requested operation.
 
 **IoT Hub updates**

azext_iot/_help.py

@@ -878,7 +878,8 @@
                    While the device simulation is running, the device will automatically receive
                    and acknowledge cloud-to-device (c2d) messages. For mqtt simulation, all c2d messages will
                    be acknowledged with completion. For http simulation c2d acknowledgement is based on user
-                   selection which can be complete, reject or abandon. The mqtt simulation also supports direct
+                   selection which can be complete, reject or abandon. Additionally, mqtt simulation is only
+                   supported for symmetric key auth (SAS) based devices. The mqtt simulation also supports direct
                    method invocation which can be acknowledged by a response status code and response payload
 
                    Note: The command by default will set content-type to application/json and content-encoding

azext_iot/central/command_map.py

@@ -111,17 +111,6 @@ def load_central_commands(self, _):
         cmd_group.command("create", "create_device_template")
         cmd_group.command("delete", "delete_device_template")
 
-    with self.command_group(
-        "iot central device-group", command_type=central_device_groups_ops, is_preview=True
-    ) as cmd_group:
-        cmd_group.command("list", "list_device_groups")
-
-    with self.command_group(
-        "iot central role", command_type=central_roles_ops, is_preview=True
-    ) as cmd_group:
-        cmd_group.show_command("show", "get_role")
-        cmd_group.command("list", "list_roles")
-
     with self.command_group(
         "iot central device twin", command_type=central_device_twin_ops,
     ) as cmd_group:

azext_iot/central/commands_device.py

@@ -122,7 +122,7 @@ def run_command(
     else:
         provider = CentralDeviceProviderV1(cmd=cmd, app_id=app_id, token=token)
 
-    return provider.run_command(
+    return provider.run_component_command(
         device_id=device_id,
         interface_id=interface_id,
         command_name=command_name,
@@ -174,7 +174,7 @@ def get_command_history(
     else:
         provider = CentralDeviceProviderV1(cmd=cmd, app_id=app_id, token=token)
 
-    return provider.get_command_history(
+    return provider.get_component_command_history(
         device_id=device_id,
         interface_id=interface_id,
         command_name=command_name,

azext_iot/central/models/__init__.py

@@ -8,16 +8,12 @@
 from azext_iot.central.models.devicetwin import DeviceTwin
 from azext_iot.central.models.templatepreview import TemplatePreview
 from azext_iot.central.models.templatev1 import TemplateV1
-from azext_iot.central.models.deviceGroupPreview import DeviceGroupPreview
-from azext_iot.central.models.rolePreview import RolePreview
 
 
 __all__ = [
     "DevicePreview",
-    "DeviceGroupPreview",
     "DeviceV1",
     "DeviceTwin",
     "TemplatePreview",
     "TemplateV1",
-    "RolePreview"
 ]

azext_iot/central/params.py

@@ -215,10 +215,3 @@ def load_central_arguments(self, _):
             options_list=["--module-id", "-m"],
             help="Provide IoT Edge Module ID if the device type is IoT Edge.",
         )
-
-    with self.argument_context("iot central role") as context:
-        context.argument(
-            "role_id",
-            options_list=["--role-id", "-r"],
-            help="Provide a unique identifier for the role"
-        )

azext_iot/central/providers/preview/__init__.py

@@ -17,19 +17,10 @@
 from azext_iot.central.providers.preview.api_token_provider_preview import (
     CentralApiTokenProviderPreview,
 )
-from azext_iot.central.providers.preview.device_group_provider_preview import (
-    CentralDeviceGroupProviderPreview
-)
-from azext_iot.central.providers.preview.role_provider_preview import (
-    CentralRoleProviderPreview
-)
 
 __all__ = [
     "CentralDeviceProviderPreview",
     "CentralDeviceTemplateProviderPreview",
     "CentralUserProviderPreview",
     "CentralApiTokenProviderPreview",
-    "CentralDeviceGroupProviderPreview",
-    "CentralRoleProviderPreview"
-
 ]

azext_iot/central/providers/preview/device_provider_preview.py

@@ -126,64 +126,39 @@ def delete_device(self, device_id, central_dns_suffix=CENTRAL_ENDPOINT,) -> dict
 
         return result
 
-    def run_command(
+    def run_component_command(
         self,
         device_id: str,
         interface_id: str,
         command_name: str,
         payload: dict,
         central_dns_suffix=CENTRAL_ENDPOINT,
     ) -> dict:
-
-        if interface_id:
-            return central_services.device.run_component_command(
-                cmd=self._cmd,
-                app_id=self._app_id,
-                token=self._token,
-                device_id=device_id,
-                interface_id=interface_id,
-                command_name=command_name,
-                payload=payload,
-                central_dns_suffix=central_dns_suffix,
-                api_version=ApiVersion.preview.value,
-            )
-
-        return central_services.device.run_command(
+        return central_services.device.run_component_command(
             cmd=self._cmd,
             app_id=self._app_id,
             token=self._token,
             device_id=device_id,
+            interface_id=interface_id,
             command_name=command_name,
             payload=payload,
             central_dns_suffix=central_dns_suffix,
             api_version=ApiVersion.preview.value,
         )
 
-    def get_command_history(
+    def get_component_command_history(
         self,
         device_id: str,
         interface_id: str,
         command_name: str,
         central_dns_suffix=CENTRAL_ENDPOINT,
     ) -> dict:
-
-        if interface_id:
-            return central_services.device.get_component_command_history(
-                cmd=self._cmd,
-                app_id=self._app_id,
-                token=self._token,
-                device_id=device_id,
-                interface_id=interface_id,
-                command_name=command_name,
-                central_dns_suffix=central_dns_suffix,
-                api_version=ApiVersion.preview.value,
-            )
-
-        return central_services.device.get_command_history(
+        return central_services.device.get_component_command_history(
             cmd=self._cmd,
             app_id=self._app_id,
             token=self._token,
             device_id=device_id,
+            interface_id=interface_id,
             command_name=command_name,
             central_dns_suffix=central_dns_suffix,
             api_version=ApiVersion.preview.value,

azext_iot/central/services/device.py

@@ -303,53 +303,6 @@ def get_device_credentials(
     return _utility.try_extract_result(response)
 
 
-def run_command(
-    cmd,
-    app_id: str,
-    token: str,
-    device_id: str,
-    command_name: str,
-    payload: dict,
-    central_dns_suffix=CENTRAL_ENDPOINT,
-    api_version=ApiVersion.v1.value,
-):
-    """
-    Execute a direct method on a device
-
-    Args:
-        cmd: command passed into az
-        app_id: name of app (used for forming request URL)
-        device_id: unique case-sensitive device id
-        command_name: name of command to execute
-        payload: params for command
-        token: (OPTIONAL) authorization token to fetch device details from IoTC.
-            MUST INCLUDE type (e.g. 'SharedAccessToken ...', 'Bearer ...')
-        central_dns_suffix: {centralDnsSuffixInPath} as found in docs
-
-    Returns:
-        result (currently a 201)
-    """
-    url = "https://{}.{}/{}/{}/commands/{}".format(
-        app_id, central_dns_suffix, BASE_PATH, device_id, command_name
-    )
-    headers = _utility.get_headers(token, cmd)
-
-    # Construct parameters
-    query_parameters = {}
-    query_parameters["api-version"] = api_version
-
-    response = requests.post(
-        url, headers=headers, json=payload, params=query_parameters
-    )
-
-    # execute command response has caveats in it due to Async/Sync device methods
-    # return the response if we get 201, otherwise try to apply generic logic
-    if response.status_code == 201:
-        return response.json()
-
-    return _utility.try_extract_result(response)
-
-
 def run_component_command(
     cmd,
     app_id: str,

azext_iot/constants.py

@@ -7,7 +7,7 @@
 
 import os
 
-VERSION = "0.10.15"
+VERSION = "0.10.14"
 EXTENSION_NAME = "azure-iot"
 EXTENSION_ROOT = os.path.dirname(os.path.abspath(__file__))
 EXTENSION_CONFIG_ROOT_KEY = "iotext"