get_data_sources call raised exception

[dogarcar]

Dear all,
I'm unable to use cate desktop 1.0.0 after installing it in my Windows 10 pro.
When starting the programme I get the following error:

get_data_sources() call raised exception: "DataStore 'esa_cci_odp' returned error: Cannot download Open Data Portal ECV index"

That is the CCI catalogue is not downloaded from the data portal.

In case it helps: my computer is under my company's proxy

Thanks.
Cheers

[forman]

This seems to be related to #20. See also #627.

[forman]

@dogarcar Try setting HTTP_PROXY and HTTPS_PROXY environment variables to your proxy server before running Cate.

[papesci]

Hi @dogarcar it could be a missing certificate, can you try to open the following link from a browser ?
link

If it works the certificate will be automatically installed so you should be allowed to open cate and get the complete list of data sources.

[dogarcar]

@forman how can I set the environment variables HTTP_PROXY and HTTPS_PROXY under windows? The option is not available in the Cate Desktop settings, isn't?

@papesci I did it. I saw some code in the browser with the variables and their attributes. But it doesn't help. I still get a failed task with the following description:

Cate Desktop, version 1.0.0

get_data_sources() call raised exception: "DataStore 'esa_cci_odp' returned error: Cannot download Open Data Portal ECV index"

An error (code 20) occurred in Cate Core:

Traceback (most recent call last):
File "C:\cate\lib\urllib\request.py", line 1318, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "C:\cate\lib\http\client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\cate\lib\http\client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\cate\lib\http\client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\cate\lib\http\client.py", line 1026, in _send_output
self.send(msg)
File "C:\cate\lib\http\client.py", line 964, in send
self.connect()
File "C:\cate\lib\http\client.py", line 1392, in connect
super().connect()
File "C:\cate\lib\http\client.py", line 936, in connect
(self.host,self.port), self.timeout, self.source_address)
File "C:\cate\lib\socket.py", line 722, in create_connection
raise err
File "C:\cate\lib\socket.py", line 713, in create_connection
sock.connect(sa)
TimeoutError: [WinError 10060] Se produjo un error durante el intento de conexión ya que la parte conectada no respondió adecuadamente tras un periodo de tiempo, o bien se produjo un error en la conexión establecida ya que el host conectado no ha podido responder

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\cate\lib\site-packages\cate\ds\esa_cci_odp.py", line 156, in _fetch_solr_json
with urllib.request.urlopen(url, timeout=timeout) as response:
File "C:\cate\lib\urllib\request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "C:\cate\lib\urllib\request.py", line 526, in open
response = self._open(req, data)
File "C:\cate\lib\urllib\request.py", line 544, in _open
'_open', req)
File "C:\cate\lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "C:\cate\lib\urllib\request.py", line 1361, in https_open
context=self._context, check_hostname=self._check_hostname)
File "C:\cate\lib\urllib\request.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [WinError 10060] Se produjo un error durante el intento de conexión ya que la parte conectada no respondió adecuadamente tras un periodo de tiempo, o bien se produjo un error en la conexión establecida ya que el host conectado no ha podido responder>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\cate\lib\site-packages\cate\ds\esa_cci_odp.py", line 225, in _load_or_fetch_json
json_obj = fetch_json_function(*(fetch_json_args or []), **(fetch_json_kwargs or {}))
File "C:\cate\lib\site-packages\cate\ds\esa_cci_odp.py", line 172, in _fetch_solr_json
.format(error, base_url))
cate.core.ds.DataAccessError: Open Data Portal index download failed, <urlopen error [WinError 10060] Se produjo un error durante el intento de conexión ya que la parte conectada no respondió adecuadamente tras un periodo de tiempo, o bien se produjo un error en la conexión establecida ya que el host conectado no ha podido responder>
https://esgf-index1.ceda.ac.uk/esg-search/search/

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\cate\lib\site-packages\cate\ds\esa_cci_odp.py", line 383, in _load_index
cache_expiration_days=self._index_cache_expiration_days)
File "C:\cate\lib\site-packages\cate\ds\esa_cci_odp.py", line 241, in _load_or_fetch_json
raise DataAccessError(None, "Cannot fetch information from Open Data Portal server.")
cate.core.ds.DataAccessError: Cannot fetch information from Open Data Portal server.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\cate\lib\site-packages\cate\util\web\jsonrpchandler.py", line 192, in send_service_method_result
result = future.result()
File "C:\cate\lib\concurrent\futures_base.py", line 398, in result
return self.__get_result()
File "C:\cate\lib\concurrent\futures_base.py", line 357, in __get_result
raise self._exception
File "C:\cate\lib\concurrent\futures\thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\cate\lib\site-packages\cate\util\web\jsonrpchandler.py", line 269, in call_service_method
result = method(*method_params, monitor=monitor)
File "C:\cate\lib\site-packages\cate\webapi\websocket.py", line 123, in get_data_sources
data_sources = data_store.query(monitor=monitor)
File "C:\cate\lib\site-packages\cate\ds\esa_cci_odp.py", line 326, in query
self._init_data_sources()
File "C:\cate\lib\site-packages\cate\ds\esa_cci_odp.py", line 348, in _init_data_sources
self._load_index()
File "C:\cate\lib\site-packages\cate\ds\esa_cci_odp.py", line 394, in _load_index
raise DataAccessError(self, "Cannot download Open Data Portal ECV index")
cate.core.ds.DataAccessError: DataStore 'esa_cci_odp' returned error: Cannot download Open Data Portal ECV index

[papesci]

@dogarcar thanks for trying, i think the problem is the proxy configuration.
in your home directory (usually c:\Users<your username> there is a hided folder named .cate
inside it you will find a file conf.py, open it and please set the two commented variables
http_proxy = 'your http proxy configuration'
https_proxy = 'your https proxy config'

[dogarcar]

Thanks for your help. I found the file in .cate/1.0.0

But the proxy setting options were not present in the file. So I created the lines with the corresponding proxy info using WordPad. However I still get the same error.

[papesci]

@dogarcar sorry you are right the cate core release (1.0) you are using do not support that feature yet.
So please can you set manually the two required environment variables on you window system, i think you have to be admin.. open system->advanced system settings-> click on Environment variables button and set :
http_proxy
https_proxy

[papesci]

No feedback from the user, maybe he is using now cate 2.0. We close it, in case the issue reappear we can reopen it.