Skip to content
This repository has been archived by the owner on Aug 29, 2023. It is now read-only.

Happy Path UC02 #327

Closed
HerzogStephan opened this issue Aug 16, 2017 · 7 comments · Fixed by #336
Closed

Happy Path UC02 #327

HerzogStephan opened this issue Aug 16, 2017 · 7 comments · Fixed by #336
Assignees
@JanisGailis
Labels
cu_review ops uc02
Milestone
1.0

Comments

@HerzogStephan
Copy link
Contributor

HerzogStephan commented Aug 16, 2017 •
edited
Loading

Comment

Relates to: #146
I conducted a preliminary happy path for Use Case 2 (see appended sh-script). When invoking it I run into some problems (see 'Steps to reproduce'-section).

Steps to reproduce the problem

  1. Invoke Cate-CLI

cate ds copy esacci.CLOUD.mon.L3C.CLD_PRODUCTS.multi-sensor.multi-platform.ATSR2-AATSR.2-0.r1 --name CLOUD_2006_2009_region_7_48_10_52 --time '2006-01-01,2009-12-31' --region '7,48,10,52'
3.
cate ws new
4.
cate res open cloud local.CLOUD_2006_2009_region_7_48_10_52
5.
cate res set cloud_lta long_term_average ds=@cloud var="cfc"
6.
cate res set station read_csv file="/data/sherzog/cate-0.9.0.dev4/UC02/produkt_klima_Tageswerte_20050715_20151231_07341_output2.txt" delimiter=";" index_col="time"

(Link to testdata: https://github.com/CCI-Tools/cate-e2e/blob/master/testdata/produkt_klima_Tageswerte_20050715_20151231_07341_vers002.txt)
7.
cate res set station_ds from_dataframe df=@station
8.
cate res set station_outl detect_outliers ds=@station_ds var=" precipitation" threshold_low=0.00 threshold_high=100.00 quantiles=False
9.
cate res set station_lta long_term_average ds=@station_outl var="precipitation"
10.
cate res set res_8 plot ds=@station_ds var="precipitation" file="/data/sherzog/cate-0.9.0.dev4/UC02/station_ds_precipitation.jpg"

Comment Problems:

Step 9. returns the following message:
''cate res: error: Long term average operation expects a dataset with the time coordinate of type datetime64[ns], but received object. Running the harmonization operation on this dataset may help''

Step 10. returns the following:
''Executing 3 workflow step(s): done
cate res: error: Plotting requires coordinates to be numeric or dates.''

Idea:

I think the errors are due to the fact, that cate does not recognize the time-variable from the testdata well. Or the format in the testdata is wrong.

Appendix:

uc02.txt

Specifications

cate-0.9.0.dev4, on linux
@mzuehlke
Copy link
Collaborator

Some more information about the data set:

marcoz ~/Scratch/327 $ cate res print station_ds
<xarray.Dataset>
Dimensions:        (time: 3821)
Coordinates:
  * time           (time) object '2005-07-15' '2005-07-16' '2005-07-17' ...
Data variables:
    Unnamed: 0     (time) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 ...
    temperature    (time) float64 -999.0 -999.0 -999.0 22.4 19.8 17.9 17.5 ...
    windvelocity   (time) float64 -999.0 -999.0 -999.0 -999.0 -999.0 -999.0 ...
    precipitation  (time) float64 1.8 -999.0 0.0 0.7 3.3 1.5 0.1 1.2 0.0 5.4 ...

And here the complete stacktrace

Traceback (most recent call last):
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/webapi/rest.py", line 243, in post
    monitor=_new_monitor())
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/wsmanag.py", line 313, in set_workspace_resource
    workspace.execute_workflow(res_name=res_name, monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/workspace.py", line 630, in execute_workflow
    self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/workflow.py", line 629, in invoke_steps
    step.invoke(context=context, monitor=monitor.child(work=1))
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/workflow.py", line 315, in invoke
    self._invoke_impl(_new_context(context, step=self), monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/workflow.py", line 977, in _invoke_impl
    return_value = self._op(monitor=monitor, **input_values)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/op.py", line 211, in __call__
    return_value = self._wrapped_op(**input_values)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/ops/aggregate.py", line 70, in long_term_average
    ' dataset may help'.format(ds.time.dtype))
ValueError: Long term average operation expects a dataset with the time coordinate of type datetime64[ns], but received object. Running the harmonization operation on this dataset may help

The here mentioned harmonize operation does not exist. It is mentioned in other places, too:

marcoz ~/Projects/BC/PyBc/cci-tools/cate-core/cate/ops (master *%)$ grep harmonization *py
aggregate.py:                         ' {}. Running the harmonization operation on this'
aggregate.py:                         ' {}. Running the harmonization operation on this'
anomaly.py:                             ' dtype datetime64[ns]. Running the harmonization'
coregistration.py:                             ' harmonization operation'
coregistration.py:                             ' harmonization operations'
subset.py:                         ' {}. Running the harmonization operation on this'

@JanisGailis maybe the normalize operation is meant ? But it does not help here.

@JanisGailis
Copy link
Member

Yes, the error message has not been updated apparently, it should be 'normalize'.

I'll look into this when I get back to cate. Should be soon.

@JanisGailis JanisGailis mentioned this issue Aug 30, 2017
Merged
@mzuehlke mzuehlke added this to the IPM6 milestone Aug 31, 2017
@HerzogStephan
Copy link
Contributor Author

I just tried to execute the reworked happy path (link) and I run into two problems:

  1. One has to add the following line, before running the 'detect_outliers' operation:
    "cate res set station_ds from_dataframe df=@station"
  2. My previously reported Problem above ('Comment Problems'-Section, Step 9) persists

@JanisGailis
Copy link
Member

I guess the dev6 has not yet been released. The detect_outliers now takes both, a dataset and a dataframe, so from_dataframe should not be used anywhere in our current happy path scripts.

@JanisGailis JanisGailis reopened this Sep 6, 2017
@HerzogStephan
Copy link
Contributor Author

HerzogStephan commented Sep 21, 2017 •
edited
Loading

@JanisGailis : Did you change the weather-station-data?

When I execute your happy path (and also my version of it) I encounter the following within the temporal_aggregation step:

Traceback (most recent call last):
  File "/data/sherzog/cate-0.9.0.dev6/lib/python3.6/site-packages/cate/util/web/jsonrpchandler.py", line 190, in send_service_method_result
    result = future.result()
  File "/data/sherzog/cate-0.9.0.dev6/lib/python3.6/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/data/sherzog/cate-0.9.0.dev6/lib/python3.6/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/data/sherzog/cate-0.9.0.dev6/lib/python3.6/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/data/sherzog/cate-0.9.0.dev6/lib/python3.6/site-packages/cate/util/web/jsonrpchandler.py", line 269, in call_service_method
    result = method(**method_params, monitor=monitor)
  File "/data/sherzog/cate-0.9.0.dev6/lib/python3.6/site-packages/cate/webapi/websocket.py", line 303, in set_workspace_resource
    monitor=monitor)
  File "/data/sherzog/cate-0.9.0.dev6/lib/python3.6/site-packages/cate/core/wsmanag.py", line 313, in set_workspace_resource
    workspace.execute_workflow(res_name=res_name, monitor=monitor)
  File "/data/sherzog/cate-0.9.0.dev6/lib/python3.6/site-packages/cate/core/workspace.py", line 598, in execute_workflow
    self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor)
  File "/data/sherzog/cate-0.9.0.dev6/lib/python3.6/site-packages/cate/core/workflow.py", line 629, in invoke_steps
    step.invoke(context=context, monitor=monitor.child(work=1))
  File "/data/sherzog/cate-0.9.0.dev6/lib/python3.6/site-packages/cate/core/workflow.py", line 315, in invoke
    self._invoke_impl(_new_context(context, step=self), monitor=monitor)
  File "/data/sherzog/cate-0.9.0.dev6/lib/python3.6/site-packages/cate/core/workflow.py", line 977, in _invoke_impl
    return_value = self._op(monitor=monitor, **input_values)
  File "/data/sherzog/cate-0.9.0.dev6/lib/python3.6/site-packages/cate/core/op.py", line 211, in __call__
    return_value = self._wrapped_op(**input_values)
  File "/data/sherzog/cate-0.9.0.dev6/lib/python3.6/site-packages/cate/ops/aggregate.py", line 163, in temporal_aggregation
    retset = ds.resample(freq='MS', dim='time', keep_attrs=True, how=method)
  File "/data/sherzog/cate-0.9.0.dev6/lib/python3.6/site-packages/xarray/core/common.py", line 548, in resample
    gb = self._groupby_cls(self, group, grouper=time_grouper)
  File "/data/sherzog/cate-0.9.0.dev6/lib/python3.6/site-packages/xarray/core/groupby.py", line 243, in __init__
    raise ValueError('index must be monotonic for resampling')
ValueError: index must be monotonic for resampling

cate res: error: set_workspace_resource() call raised exception: "index must be monotonic for resampling"

@JanisGailis : I cannot find my mistake. But your happy path works for you?

@JanisGailis
Copy link
Member

Yes, you have to use the same data as used in the 'official' UC02 happy path. The last line was empty (,,,,) generating this error, I removed it.

@forman forman modified the milestones: IPM6, 1.0 Sep 21, 2017
@HerzogStephan
Copy link
Contributor Author

Works perfectly!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@JanisGailis JanisGailis

Labels
cu_review ops uc02
Projects
None yet
Milestone
1.0
Development

Successfully merging a pull request may close this issue.

145 uc02 api demo CCI-Tools/cate
4 participants
@mzuehlke @forman @JanisGailis @HerzogStephan