SST: Copying remote data source failed

[Evadzi]

Expected behavior

*I expect cate to ingest all SST data from ODP *

Actual behavior

cate is unable to ingest all SST data from ODP except L4 data: esacci.SST.day.L4.SSTdepth.multi-sensor.multi-platform.OSTIA.1-1.r1

Steps to reproduce the problem

1. *filter data for sst in the find data source filter panel *
2. *download any of the sst data except the last data-esacci.SST.day.L4.SSTdepth.multi-sensor.multi-platform.OSTIA.1-1.r1 *

Error Message

e.g.
Traceback (most recent call last): File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\ds\esa_cci_odp.py", line 937, in _make_local local_ds.add_dataset(os.path.join(local_id, filename), (coverage_from, coverage_to)) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\ds\local.py", line 364, in add_dataset self._extend_temporal_coverage(TimeRangeLike.convert(time_coverage)) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\types.py", line 698, in convert cls.raise_validation_error(value) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\types.py", line 154, in raise_validation_error .format(value=text_value, type=cls.name())) cate.core.types.ValidationError: Value (datetime.datetime(1995,... cannot be converted into a 'TimeRangeLike'. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\util\web\jsonrpchandler.py", line 209, in send_service_method_result result = future.result() File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\concurrent\futures\_base.py", line 425, in result return self.__get_result() File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\concurrent\futures\_base.py", line 384, in __get_result raise self._exception File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\concurrent\futures\thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\util\web\jsonrpchandler.py", line 306, in call_service_method result = method(*method_params, monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\webapi\websocket.py", line 292, in set_workspace_resource monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\wsmanag.py", line 320, in set_workspace_resource workspace.execute_workflow(res_name=res_name, monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\workspace.py", line 662, in execute_workflow self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\workflow.py", line 627, in invoke_steps steps[0].invoke(context=context, monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\workflow.py", line 318, in invoke self._invoke_impl(_new_context(context, step=self), monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\workflow.py", line 980, in _invoke_impl return_value = self._op(monitor=monitor, **input_values) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\op.py", line 216, in __call__ return_value = self._wrapped_op(**input_values) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\ops\io.py", line 83, in open_dataset monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\ds.py", line 588, in open_dataset monitor=monitor.child(80)) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\ds\esa_cci_odp.py", line 1007, in make_local raise e File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\ds\esa_cci_odp.py", line 1000, in make_local self._make_local(local_ds, time_range, region, var_names, monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\ds\esa_cci_odp.py", line 946, in _make_local raise ValidationError("Copying remote data source failed: {}".format(e)) from e cate.core.types.ValidationError: Copying remote data source failed: Value (datetime.datetime(1995,... cannot be converted into a 'TimeRangeLike.

Specifications

cate 2.0.0. dev-20, Windows 10

[HelenClifton]

Hi @Evadzi @forman For info I'm able to download that dataset using cate-2.0.0-dev.20 when following constraints are applied [2006-01-01, 2007-12-31] , [Lon. from: -175 to -115, Lat. from: -10 to 10], [vars: analysed_sst, analysis_error]. This is part of use case 6 happy path (I was using Windows 7)

@Evadzi Which temporal / regional / variable constraints did you use (if any) when you saw the error ?

[Evadzi]

@HelenClifton Thank you for the feedback.
Yes, I tried the subset for the last SST data (esacci.SST.day.L4.SSTdepth.multi-sensor.multi-platform.OSTIA.1-1.r1) which I excluded from this issue.

I noticed the remaining sst datasets are limited to one year, and the time range is also not displayed in the product summary when the product is selected. The temporal coverage is displayed in the product summary only and only if you click on the open data icon.(the later is general for all data in the ODP)

It is important to note that this issue which is about Copying remote data source failed: Value (datetime.datetime(2000,... cannot be converted into a 'TimeRangeLike'. is also realized with other datasets e.g landcover, and must open with or without constraints..

Below is the related error message (Ingestion of landcover data from ODP) below

Traceback (most recent call last): File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\ds\esa_cci_odp.py", line 937, in _make_local local_ds.add_dataset(os.path.join(local_id, filename), (coverage_from, coverage_to)) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\ds\local.py", line 364, in add_dataset self._extend_temporal_coverage(TimeRangeLike.convert(time_coverage)) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\types.py", line 698, in convert cls.raise_validation_error(value) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\types.py", line 154, in raise_validation_error .format(value=text_value, type=cls.name())) cate.core.types.ValidationError: Value (datetime.datetime(2000,... cannot be converted into a 'TimeRangeLike'. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\util\web\jsonrpchandler.py", line 209, in send_service_method_result result = future.result() File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\concurrent\futures\_base.py", line 425, in result return self.__get_result() File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\concurrent\futures\_base.py", line 384, in __get_result raise self._exception File "C:\Users\prosper\Miniconda3\envs\cate-env\lib\concurrent\futures\thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\util\web\jsonrpchandler.py", line 306, in call_service_method result = method(*method_params, monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\webapi\websocket.py", line 292, in set_workspace_resource monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\wsmanag.py", line 320, in set_workspace_resource workspace.execute_workflow(res_name=res_name, monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\workspace.py", line 662, in execute_workflow self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\workflow.py", line 627, in invoke_steps steps[0].invoke(context=context, monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\workflow.py", line 318, in invoke self._invoke_impl(_new_context(context, step=self), monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\workflow.py", line 980, in _invoke_impl return_value = self._op(monitor=monitor, **input_values) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\op.py", line 216, in __call__ return_value = self._wrapped_op(**input_values) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\ops\io.py", line 83, in open_dataset monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\core\ds.py", line 588, in open_dataset monitor=monitor.child(80)) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\ds\esa_cci_odp.py", line 1007, in make_local raise e File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\ds\esa_cci_odp.py", line 1000, in make_local self._make_local(local_ds, time_range, region, var_names, monitor=monitor) File "c:\users\prosper\miniconda3\envs\cate-env\lib\site-packages\cate-2.0.0.dev21-py3.6.egg\cate\ds\esa_cci_odp.py", line 946, in _make_local raise ValidationError("Copying remote data source failed: {}".format(e)) from e cate.core.types.ValidationError: Copying remote data source failed: Value (datetime.datetime(2000,... cannot be converted into a 'TimeRangeLike'.

[Evadzi]

##Update
@HelenClifton @forman
I did not get any lead while analysing the Stack trace but I noticed the error is related to the generation of ID for all data to be downloaded from the ODP.
The download works if the download button is not checked or a name provided
I hope this helps….

[forman]

Turned this into blocker, because we currently cannot use any CCI SST data.

[papesci]

Hi all, most of the SST dataset has been split in small files with the same data coverage times (coverage_from == coverage_to) thus the exception because the TimeRange Type expect to handle two different date time. The fix is simple, just changed the check operator from < to <= to accept also timerange of duration 0. Branch https://github.com/CCI-Tools/cate/tree/798-pp-SST-copying-error