[BUG] Error in the AzureML tests with error client is not permitted to update status for run. Status for this run is system managed

[miguelgfierro]

Description

In all AzureML tests, we get a similar error (see https://github.com/microsoft/recommenders/actions/runs/4049332367/jobs/6965640900)

Streaming user_logs/std_log.txt
===============================

INFO:submit_groupwise_azureml_pytest.py:Tests to be executed
INFO:submit_groupwise_azureml_pytest.py:['tests/smoke/examples/test_notebooks_python.py::test_baseline_deep_dive_smoke', 'tests/integration/examples/test_notebooks_python.py::test_baseline_deep_dive_integration', 'tests/smoke/examples/test_notebooks_python.py::test_surprise_svd_smoke', 'tests/integration/examples/test_notebooks_python.py::test_surprise_svd_integration', 'tests/integration/examples/test_notebooks_python.py::test_geoimc_integration', 'tests/integration/examples/test_notebooks_python.py::test_benchmark_movielens_cpu']
INFO:azureml._restclient.clientbase:Created a worker pool for first use
INFO:submit_groupwise_azureml_pytest.py:Python version 
INFO:submit_groupwise_azureml_pytest.py:3.7.13 (default, Mar 29 2022, 02:18:16) 
[GCC 7.5.0]
INFO:submit_groupwise_azureml_pytest.py:Executing tests now...
============================= test session starts ==============================
platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0
rootdir: /mnt/azureml/cr/j/82c5a50f0b2d4c9881f6deb1628b75cc/exe/wd, configfile: pyproject.toml
plugins: anyio-3.6.2, cov-4.0.0, hypothesis-6.60.0, rerunfailures-10.3, mock-3.10.0
collected 6 items

tests/smoke/examples/test_notebooks_python.py .                          [ 16%]
tests/integration/examples/test_notebooks_python.py 
tests/smoke/examples/test_notebooks_python.py 
tests/integration/examples/test_notebooks_python.py 

=============================== warnings summary ===============================
../../../../../../../azureml-envs/azureml_362c648550ddf3444cbaf086714b5547/lib/python3.7/site-packages/ansiwrap/core.py:6
  /azureml-envs/azureml_362c648550ddf3444cbaf086714b5547/lib/python3.7/site-packages/ansiwrap/core.py:6: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

tests/smoke/examples/test_notebooks_python.py::test_baseline_deep_dive_smoke
tests/integration/examples/test_notebooks_python.py::test_baseline_deep_dive_integration[1m-expected_values0]
  /azureml-envs/azureml_362c648550ddf3444cbaf086714b5547/lib/python3.7/site-packages/traitlets/config/configurable.py:85: DeprecationWarning: Passing unrecognized arguments to super(PapermillNotebookClient).__init__(input_path='/mnt/azureml/cr/j/82c5a50f0b2d4c9881f6deb1628b75cc/exe/wd/examples/02_model_collaborative_filtering/baseline_deep_dive.ipynb').
  object.__init__() takes exactly one argument (the instance to initialize)
  This is deprecated in traitlets 4.2.This error will be raised in a future release of traitlets.
    super().__init__(**kwargs)

tests/smoke/examples/test_notebooks_python.py::test_surprise_svd_smoke
tests/integration/examples/test_notebooks_python.py::test_surprise_svd_integration[1m-expected_values0]
  /azureml-envs/azureml_362c648550ddf3444cbaf086714b5547/lib/python3.7/site-packages/traitlets/config/configurable.py:85: DeprecationWarning: Passing unrecognized arguments to super(PapermillNotebookClient).__init__(input_path='/mnt/azureml/cr/j/82c5a50f0b2d4c9881f6deb1628b75cc/exe/wd/examples/02_model_collaborative_filtering/surprise_svd_deep_dive.ipynb').
  object.__init__() takes exactly one argument (the instance to initialize)
  This is deprecated in traitlets 4.2.This error will be raised in a future release of traitlets.
    super().__init__(**kwargs)

tests/integration/examples/test_notebooks_python.py::test_geoimc_integration[expected_values0]
  /azureml-envs/azureml_362c648550ddf3444cbaf086714b5547/lib/python3.7/site-packages/traitlets/config/configurable.py:85: DeprecationWarning: Passing unrecognized arguments to super(PapermillNotebookClient).__init__(input_path='/mnt/azureml/cr/j/82c5a50f0b2d4c9881f6deb1628b75cc/exe/wd/examples/00_quick_start/geoimc_movielens.ipynb').
  object.__init__() takes exactly one argument (the instance to initialize)
  This is deprecated in traitlets 4.2.This error will be raised in a future release of traitlets.
    super().__init__(**kwargs)

tests/integration/examples/test_notebooks_python.py::test_benchmark_movielens_cpu[size0-algos0-expected_values_ndcg0]
  /azureml-envs/azureml_362c648550ddf3444cbaf086714b5547/lib/python3.7/site-packages/traitlets/config/configurable.py:85: DeprecationWarning: Passing unrecognized arguments to super(PapermillNotebookClient).__init__(input_path='/mnt/azureml/cr/j/82c5a50f0b2d4c9881f6deb1628b75cc/exe/wd/examples/06_benchmarks/movielens.ipynb').
  object.__init__() takes exactly one argument (the instance to initialize)
  This is deprecated in traitlets 4.2.This error will be raised in a future release of traitlets.
    super().__init__(**kwargs)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================== 6 passed, 7 warnings in 1552.75s (0:25:52) ==================
INFO:submit_groupwise_azureml_pytest.py:Test execution completed!
Cleaning up all outstanding Run operations, waiting 300.0 seconds
1 items cleaning up...
Cleanup took 0.5428390502929688 seconds

Execution Summary
=================
RunId: nightly_tests_group_cpu_002_1675123876_8a2b565c
Web View: https://ml.azure.com/runs/nightly_tests_group_cpu_002_1675123876_8a2b565c?wsid=/subscriptions/***/resourcegroups/recommenders_project_resources/workspaces/azureml-test-workspace&tid=***

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 589, in _execute_with_arguments
    return self._call_api(func, *args_list, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 245, in _call_api
    return self._execute_with_base_arguments(func, *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 333, in _execute_with_base_arguments
    return ClientBase._execute_func_internal(
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 367, in _execute_func_internal
    left_retry = cls._handle_retry(back_off, left_retry, total_retry, error, logger, func)
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 426, in _handle_retry
    raise error
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 358, in _execute_func_internal
    response = func(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/operations/events_operations.py", line 95, in post
    raise models.ErrorResponseException(self._deserialize, response)
azureml._restclient.models.error_response.ErrorResponseException: (ValidationError) Run nightly_tests_group_cpu_002_1675123876_8a2b565c status cannot be updated by python/3.8.16 (Linux-5.15.0-1031-azure-x86_64-with-glibc2.34) msrest/0.7.1  azureml._restclient/1.5.0 azureml-sdk-core/1.48.0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py", line 499, in <module>
    run.complete()
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/core/run.py", line [1411](https://github.com/microsoft/recommenders/actions/runs/4049332367/jobs/6965640900#step:3:1420), in complete
    self._client.complete(_set_status=_set_status)
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_run_impl/run_history_facade.py", line 621, in complete
    self.set_completed_status(_set_status=_set_status)
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_run_impl/run_history_facade.py", line 625, in set_completed_status
    self.run.post_event_completed(caller=self.identity)
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/run_client.py", line 384, in post_event_completed
    return self.post_event(event, is_async=is_async, caller=caller, custom_headers=custom_headers)
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/run_client.py", line 316, in post_event
    return self._execute_with_run_arguments(self._client.events.post, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/run_client.py", line 564, in _execute_with_run_arguments
    return self._execute_with_arguments(func, copy.deepcopy(self._run_arguments), *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 591, in _execute_with_arguments
    raise ServiceException(e)
azureml._restclient.exceptions.ServiceException: ServiceException:
	Code: 400
	Message: (ValidationError) Run nightly_tests_group_cpu_002_1675123876_8a2b565c status cannot be updated by python/3.8.16 (Linux-5.15.0-1031-azure-x86_64-with-glibc2.34) msrest/0.7.1  azureml._restclient/1.5.0 azureml-sdk-core/1.48.0
	Details:
		Run nightly_tests_group_cpu_002_1675123876_8a2b565c status cannot be updated by python/3.8.16 (Linux-5.15.0-1031-azure-x86_64-with-glibc2.34) msrest/0.7.1  azureml._restclient/1.5.0 azureml-sdk-core/1.48.0

	Headers: ***
	    "Date": "Tue, 31 Jan 2023 00:41:13 GMT",
	    "Content-Type": "application/json; charset=utf-8",
	    "Content-Length": "1330",
	    "Connection": "keep-alive",
	    "Request-Context": "appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d",
	    "x-ms-response-type": "error",
	    "Strict-Transport-Security": "max-age=15724800; includeSubDomains; preload",
	    "X-Content-Type-Options": "nosniff",
	    "x-aml-cluster": "vienna-eastus-02",
	    "x-request-time": "0.052"
	***
	InnerException: ***
    "additional_properties": ***,
    "error": ***
        "additional_properties": ***
            "debugInfo": null
        ***,
        "code": "ValidationError",
        "severity": null,
        "message": "Run nightly_tests_group_cpu_002_1675123876_8a2b565c status cannot be updated by python/3.8.16 (Linux-5.15.0-1031-azure-x86_64-with-glibc2.34) msrest/0.7.1  azureml._restclient/1.5.0 azureml-sdk-core/1.48.0",
        "message_format": null,
        "message_parameters": null,
        "reference_code": null,
        "details_uri": null,
        "target": "Status",
        "details": [
            ***
                "additional_properties": ***
                    "debugInfo": null
                ***,
                "code": "Immutable",
                "severity": null,
                "message": "Run nightly_tests_group_cpu_002_1675123876_8a2b565c status cannot be updated by python/3.8.16 (Linux-5.15.0-1031-azure-x86_64-with-glibc2.34) msrest/0.7.1  azureml._restclient/1.5.0 azureml-sdk-core/1.48.0",
                "message_format": null,
                "message_parameters": ***,
                "reference_code": null,
                "details_uri": null,
                "target": "Status",
                "details": [],
                "inner_error": null,
                "additional_info": null
            ***
        ],
        "inner_error": null,
        "additional_info": null
    ***,
    "correlation": ***
        "operation": "f7b112a337eb8db68a8f472148b4d221",
        "request": "5600226ee0f6b4f9"
    ***,
    "environment": "eastus",
    "location": "eastus",
    "time": ***,
    "component_name": "run-history"
***
Error: Process completed with exit code 1.

In which platform does it happen?

AzureML tests

How do we replicate the issue?

Expected behavior (i.e. solution)

Other Comments

[miguelgfierro]

@pradnyeshjoshi do you know what the problem could be?