⚗️ Changed periodic_task_result internals in the attempt of solving issue

[GitHK]

What do these changes do?

Somehow after the periodic_task_result is done polling for the status it recovers the result and then raises an error error relative to the status retrieval done in the coroutine awaited by await_for.

This implementation is equivalent and tries to avoid the issue by making the situation impossible to occur.

"2025-01-13T10:15:07.608Z","osparc-srv03","log_level=INFO | log_timestamp=2025-01-13 10:15:07,608 | log_source=httpx:_send_single_request(1773) | log_uid=None | log_oec=None| log_trace_id=0 | log_span_id=0 | log_resource.service.name= | log_trace_sampled=False] | log_msg=HTTP Request: GET http://dy-sidecar_20a82fa9-0614-4a3b-84e0-48dd5935003c:8000/task/simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567 ""HTTP/1.1 200 OK"""
"2025-01-13T10:15:07.822Z","osparc-srv03","log_level=INFO | log_timestamp=2025-01-13 10:15:07,822 | log_source=httpx:_send_single_request(1773) | log_uid=None | log_oec=None| log_trace_id=b13b450573dfec1a316d59a11a490299 | log_span_id=fa2efb5f572b4a82 | log_resource.service.name= | log_trace_sampled=True] | log_msg=HTTP Request: GET http://dy-sidecar_20a82fa9-0614-4a3b-84e0-48dd5935003c:8000/task/simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567 ""HTTP/1.1 200 OK"""
"2025-01-13T10:15:07.833Z","osparc-srv03","log_level=INFO | log_timestamp=2025-01-13 10:15:07,833 | log_source=httpx:_send_single_request(1773) | log_uid=None | log_oec=None| log_trace_id=b13b450573dfec1a316d59a11a490299 | log_span_id=fa2efb5f572b4a82 | log_resource.service.name= | log_trace_sampled=True] | log_msg=HTTP Request: GET http://dy-sidecar_20a82fa9-0614-4a3b-84e0-48dd5935003c:8000/task/simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567/result ""HTTP/1.1 200 OK"""
"2025-01-13T10:15:08.629Z","osparc-srv03","log_level=INFO | log_timestamp=2025-01-13 10:15:08,629 | log_source=httpx:_send_single_request(1773) | log_uid=None | log_oec=None| log_trace_id=0 | log_span_id=0 | log_resource.service.name= | log_trace_sampled=False] | log_msg=HTTP Request: GET http://dy-sidecar_20a82fa9-0614-4a3b-84e0-48dd5935003c:8000/task/simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567 ""HTTP/1.1 404 Not Found"""
"2025-01-13T10:15:08.641Z","osparc-srv03","log_level=ERROR | log_timestamp=2025-01-13 10:15:08,634 | log_source=simcore_service_director_v2.modules.dynamic_sidecar.scheduler._core._observer:observing_single_service(155) | log_uid=19937 | log_oec=OEC:140568355058400| log_trace_id=0 | log_span_id=0 | log_resource.service.name= | log_trace_sampled=False] | log_msg=This service (dy-sidecar_20a82fa9-0614-4a3b-84e0-48dd5935003c) unexpectedly failed. Our team has recorded the issue and is working to resolve it as quickly as possible. Thank you for your patience..
{'context': ""{'action': 'getting_status',
""
            "" 'body': ""
            '\'{""code"":""long_running_task.task_not_found"",""message"":""No task '
            ""with '
""
            '         '
            ""'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567 ""
            ""'
""
            '         \'found""}\',
'
            "" 'code': 'long_running_task.client.generic_error',
""
            ' \'message\': ""Unexpected error while \'getting_status\' for ""
'
            '            '
            '""\'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567\': '
            '""
'
            ""            'status=404 '
""
            '            '
            '\'body={""code"":""long_running_task.task_not_found"",""message"":""No '
            ""'
""
            ""            'task with '
""
            '            '
            ""'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567 ""
            ""'
""
            '            \'found""}\',
'
            "" 'service_name': ""
            ""'dy-sidecar_20a82fa9-0614-4a3b-84e0-48dd5935003c',
""
            "" 'status': 404,
""
            "" 'task_id': ""
            ""'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567',
""
            "" 'user_id': 19937}"",
 'error_code': 'OEC:140568355058400',
 'exception_details': ""Unexpected error while 'getting_status' for ""
                      ""'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567': ""
                      'status=404 '
                      'body={""code"":""long_running_task.task_not_found"",""message"":""No '
                      'task with '
                      'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567 '
                      'found""}',
 'exception_type': '<class '
                   ""'servicelib.long_running_tasks._errors.GenericClientError'>"",
 'tip': 'Observation of '
        ""service_name='dy-sidecar_20a82fa9-0614-4a3b-84e0-48dd5935003c' ""
        'unexpectedly failed'}
Traceback (most recent call last):
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_observer.py"", line 139, in observing_single_service
    await _apply_observation_cycle(scheduler, scheduler_data)
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_observer.py"", line 69, in _apply_observation_cycle
    await dynamic_scheduler_event.action(app, scheduler_data)
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_events.py"", line 205, in action
    await create_user_services(app, scheduler_data)
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_events_user_services.py"", line 233, in create_user_services
    await sidecars_client.pull_service_input_ports(dynamic_sidecar_endpoint)
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/api_client/_public.py"", line 411, in pull_service_input_ports
    transferred_bytes = await self._await_for_result(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/api_client/_public.py"", line 306, in _await_for_result
    async with periodic_task_result(
  File ""/usr/local/lib/python3.11/contextlib.py"", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/fastapi/long_running_tasks/_context_manager.py"", line 114, in periodic_task_result
    await asyncio.wait_for(_wait_task_completion(), timeout=task_timeout)
  File ""/usr/local/lib/python3.11/asyncio/tasks.py"", line 489, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/fastapi/long_running_tasks/_context_manager.py"", line 111, in _wait_task_completion
    task_status = await _status_update()
                  ^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/fastapi/long_running_tasks/_context_manager.py"", line 98, in _status_update
    task_status: TaskStatus = await client.get_task_status(task_id)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/fastapi/long_running_tasks/_client.py"", line 97, in request_wrapper
    async for attempt in AsyncRetrying(
  File ""/home/scu/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py"", line 166, in __anext__
    do = await self.iter(retry_state=self._retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py"", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/tenacity/_utils.py"", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/tenacity/__init__.py"", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/concurrent/futures/_base.py"", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/concurrent/futures/_base.py"", line 401, in __get_result
    raise self._exception
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/fastapi/long_running_tasks/_client.py"", line 106, in request_wrapper
    return await request_func(zelf, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/fastapi/long_running_tasks/_client.py"", line 150, in get_task_status
    raise GenericClientError(
servicelib.long_running_tasks._errors.GenericClientError: Unexpected error while 'getting_status' for 'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567': status=404 body={""code"":""long_running_task.task_not_found"",""message"":""No task with simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_inputs_pull.8d6c2a62-4df6-43db-8310-0e5f6e6d4567 found""}"

Related issue/s

How to test

Dev-ops checklist

• No ENV changes or I properly updated ENV (read the instruction)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.76%. Comparing base (4d732df) to head (57f6c32).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7032      +/-   ##
==========================================
- Coverage   87.56%   86.76%   -0.80%     
==========================================
  Files        1629     1299     -330     
  Lines       63454    55099    -8355     
  Branches     2047     1010    -1037     
==========================================
- Hits        55566    47809    -7757     
+ Misses       7552     7096     -456     
+ Partials      336      194     -142     

Flag	Coverage Δ
integrationtests	64.48% <ø> (+0.71%)	⬆️
unittests	84.51% <100.00%> (-1.45%)	⬇️

Components	Coverage Δ
api	∅ <ø> (∅)
pkg_aws_library	∅ <ø> (∅)
pkg_dask_task_models_library	∅ <ø> (∅)
pkg_models_library	∅ <ø> (∅)
pkg_notifications_library	∅ <ø> (∅)
pkg_postgres_database	∅ <ø> (∅)
pkg_service_integration	∅ <ø> (∅)
pkg_service_library	73.89% <100.00%> (ø)
pkg_settings_library	∅ <ø> (∅)
pkg_simcore_sdk	85.38% <ø> (ø)
agent	96.45% <ø> (ø)
api_server	90.54% <ø> (ø)
autoscaling	96.09% <ø> (ø)
catalog	90.66% <ø> (ø)
clusters_keeper	99.24% <ø> (ø)
dask_sidecar	91.26% <ø> (ø)
datcore_adapter	93.18% <ø> (ø)
director	76.51% <ø> (+0.08%)	⬆️
director_v2	91.30% <ø> (ø)
dynamic_scheduler	97.21% <ø> (ø)
dynamic_sidecar	89.75% <ø> (ø)
efs_guardian	90.46% <ø> (ø)
invitations	93.44% <ø> (ø)
osparc_gateway_server	∅ <ø> (∅)
payments	92.66% <ø> (ø)
resource_usage_tracker	89.52% <ø> (+0.11%)	⬆️
storage	89.57% <ø> (ø)
webclient	∅ <ø> (∅)
webserver	84.10% <ø> (-1.84%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d732df...57f6c32. Read the comment docs.

[sanderegg]

ok

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud