Natural Earth downloads timing out

[leecronce]

When trying to plot using Cartopy on a new installation, any Natural Earth download is timing out. In addition, if I try to download a copy of a zip file from the Natural Earth website, it times out as well. Is there an issue with the host? Here is the traceback from Python:

/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/cartopy/io/init.py:264: DownloadWarning: Downloading: http://naciscdn.org/naturalearth/110m/physical/ne_110m_coastline.zip
warnings.warn('Downloading: {}'.format(url), DownloadWarning)
Traceback (most recent call last):
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/matplotlib/backends/backend_qt5agg.py", line 197, in __draw_idle_agg
FigureCanvasAgg.draw(self)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 464, in draw
self.figure.draw(self.renderer)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/matplotlib/artist.py", line 63, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/matplotlib/figure.py", line 1144, in draw
renderer, self, dsu, self.suppressComposite)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/matplotlib/image.py", line 139, in _draw_list_compositing_images
a.draw(renderer)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/matplotlib/artist.py", line 63, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/cartopy/mpl/geoaxes.py", line 360, in draw
inframe=inframe)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/matplotlib/artist.py", line 63, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/matplotlib/axes/_base.py", line 2426, in draw
mimage._draw_list_compositing_images(renderer, self, dsu)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/matplotlib/image.py", line 139, in _draw_list_compositing_images
a.draw(renderer)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/matplotlib/artist.py", line 63, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/cartopy/mpl/feature_artist.py", line 134, in draw
geoms = self._feature.intersecting_geometries(extent)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/cartopy/feature.py", line 116, in intersecting_geometries
return (geom for geom in self.geometries() if
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/cartopy/feature.py", line 183, in geometries
name=self.name)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/cartopy/io/shapereader.py", line 264, in natural_earth
return ne_downloader.path(format_dict)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/cartopy/io/init.py", line 226, in path
result_path = self.acquire_resource(target_path, format_dict)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/cartopy/io/shapereader.py", line 319, in acquire_resource
shapefile_online = self._urlopen(url)
File "/home/leec/.conda/envs/pylee/lib/python2.7/site-packages/cartopy/io/init.py", line 265, in _urlopen
return urlopen(url)
File "/home/leec/.conda/envs/pylee/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/home/leec/.conda/envs/pylee/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/home/leec/.conda/envs/pylee/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/home/leec/.conda/envs/pylee/lib/python2.7/urllib2.py", line 467, in error
result = self._call_chain(*args)
File "/home/leec/.conda/envs/pylee/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/home/leec/.conda/envs/pylee/lib/python2.7/urllib2.py", line 654, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/home/leec/.conda/envs/pylee/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/home/leec/.conda/envs/pylee/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/home/leec/.conda/envs/pylee/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/home/leec/.conda/envs/pylee/lib/python2.7/urllib2.py", line 1241, in https_open
context=self._context)
File "/home/leec/.conda/envs/pylee/lib/python2.7/urllib2.py", line 1198, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 110] Connection timed out>

[akrherz]

See #1294 and https://downforeveryoneorjustme.com/naciscdn.org

[hafez-ahmad]

I am unable to download using cartopy

add land and coastline and 1000m bathymetry line

ax.add_feature(cfeature.LAND, facecolor='grey', zorder=1)
ax.add_feature(cfeature.COASTLINE, linewidth=0.25, zorder=1)
bathym = cfeature.NaturalEarthFeature(name='bathymetry_J_1000', scale='10m', category='physical')
bathym = cascaded_union(list(bathym.geometries()))

DownloadWarning: Downloading: http://naciscdn.org/naturalearth/10m/physical/ne_10m_bathymetry_J_1000.zip
warnings.warn('Downloading: {}'.format(url), DownloadWarning)
Traceback (most recent call last):

[dopplershift]

Please open a new issue rather than commenting on this closed one. Also, be sure to include the full error message/traceback (it's not included above).

[QuLogic]

It's not a Cartopy issue, but Natural Earth nvkelso/natural-earth-vector#370, so opening a new issue is not necessary.

[bjlittle]

@dopplershift and @QuLogic

The http://naciscdn.org/ is proving to be not very resilient as a service recently... it's also caused travis-ci to grind to a halt on scitools/iris... makes me think standing-up a mirror service that scrapes http://naciscdn.org/ for content when it is available, and hitting that instead (automatically on an naciscdn.org outage) might be something to consider going forwards.

What do you guys think?

Otherwise, I really do think that this is going to happen again, and again... which is a royal pain for the community.

[bjlittle]

The other alternative is to make the cartopy data available as a conda package, akin to https://github.com/conda-forge/iris-sample-data-feedstock or explore/extend https://github.com/conda-forge/cartopy_offlinedata-feedstock.

Just floating some ideas... 🤔

[greglucas]

Looks like another issue is looking into it with more recent updates: nvkelso/natural-earth-vector#372

Another thought is to write a thin wrapper that downloads all of the contents of the zip files from the main GitHub repo (they aren't zipped up there which would have enabled the mirror approach easily).
https://github.com/nvkelso/natural-earth-vector

Creating a conda package with the basic data in addition to a more robust mirror would also potentially be useful. See #1325 for interest in getting the data on a cluster without internet access.

[dopplershift]

@greglucas That jogged a memory--that package already exists: cartopy_offlinedata.

>conda search cartopy_offlinedata    
Loading channels: done
# Name                       Version           Build  Channel             
cartopy_offlinedata            0.2.1            py_0  conda-forge         
cartopy_offlinedata            0.2.1            py_1  conda-forge         
cartopy_offlinedata            0.2.1    pyh9f0ad1d_2  conda-forge         
cartopy_offlinedata            0.2.2    pyh9f0ad1d_0  conda-forge         
cartopy_offlinedata            0.2.3    pyh9f0ad1d_0  conda-forge         

Edit: I see @bjlittle said that now.

[aurelgriesser]

Hi dopplershift, thanks for the suggestion.

Unfortunately, even after installing the cartopy_offlinedata package my code gets stuck with a DownloadWarning, which eventually times out (urllib Errno 60 Operation timed out). I don't know why because the machine has internet access, and I don't know why it's even trying to download anything because the shapefiles of the cartopy_offlinedata package exist in the /Users/me/miniconda3/envs/geo/share/cartopy/shapefiles/... folders (where I believe they should be, as placed by cartopy_offlinedata).

Should they still have "ne_" prefixes? Presume so since that's how cartopy_offlinedata left them.

Thanks for any tips...

[dopplershift]

@aurelgriesser I can say that all of my downloaded files have the "ne_" prefix. Can you share the output of conda list for the environment you're using to run cartopy?

[akrherz]

I am reproducing this issue too now, but sadly only within Github Actions and not locally :(

cartopy-0.19.0.post1       py38hc9c980b_0      
cartopy_offlinedata-0.2.3       pyh9f0ad1d_0 

And I still get:

DownloadWarning: Downloading: https://naciscdn.org/naturalearth/50m/physical/ne_50m_land.zip

I'm trying to figure this mystery out too!

[greglucas]

I wonder if the Cartopy config locations need to be updated when using that package.
Perhaps try logging the paths in cartopy.config: https://scitools.org.uk/cartopy/docs/latest/cartopy.html

python -c "import cartopy; print(cartopy.config)"

and see if they are in the correct location on the test runners... My local install is not pointing to my virtual environment, but rather my $HOME location... $HOME/.local/share/cartopy, so that data is shared between all of my virtual environments.

[akrherz]

Thanks @greglucas, I am debugging with that idea now. I should clarify, that I am reproducing this within an apache/mod_wsgi github actions workflow. Doing things from pytest are working as expected on github actions...

Additionally, I have submitted a PR to conda-forge/cartopy-feedstock#116 to patch the cartopy available there to make folks CI's green again :)

[akrherz]

Ah, so the immediate difference is the setting of pre_existing_data_dir.

From the command line, I am doing the full conda activate env, which invokes the shell environment scripts that set the environment variable CARTOPY_OFFLINE_SHARED, which then gets used by cartopy_userconfig/__init__.py to set the config['pre_existing_data_dir']

For apache, I am being lazy and just setting $PATH, etc into the conda installation, so it does not have CARTOPY_OFFLINE_SHARED set.

[diegormsouza]

Hi everyone!

I use Google Colab for my classes and I'm having this same timout problem with cartopy. It worked fine until some weeks ago... (I noticed that it was unstable but it worked sometimes. Now it just doesn't work anymore).

Now I'm trying to solve the problem manually downloading the files before using the cartopy commands, without success.

This is what I'm doing in Google Colab:

1-) Install cartopy via pip

!pip install cartopy

!pip install shapely --no-binary shapely --force

2-) Check where the data dir is:

!python -c "import cartopy; print(cartopy.config)"

This is the output:

'data_dir': '/root/.local/share/cartopy'

3-) Manually download all the shapefiles and put then in the data_dir:

!wget -c https://naturalearth.s3.amazonaws.com/110m_cultural/110m_cultural.zip -P $HOME/.local/share/cartopy/shapefiles/natural_earth/physical/
!wget -c https://naturalearth.s3.amazonaws.com/110m_physical/ne_110m_coastline.zip -P $HOME/.local/share/cartopy/shapefiles/natural_earth/physical/
!wget -c https://naturalearth.s3.amazonaws.com/110m_cultural/ne_110m_admin_0_boundary_lines_land.zip -P $HOME/.local/share/cartopy/shapefiles/natural_earth/physical/

!unzip -o $HOME/.local/share/cartopy/shapefiles/natural_earth/physical/110m_cultural.zip
!unzip -o $HOME/.local/share/cartopy/shapefiles/natural_earth/physical/ne_110m_coastline.zip
!unzip -o $HOME/.local/share/cartopy/shapefiles/natural_earth/physical/ne_110m_admin_0_boundary_lines_land.zip

But when I try to run the coastlines or border commands, the script keep trying to donwload the data from the failing server.

Do you have any hints? How can I make this work when using Colab?

Thanks,
Diego

[rcomer]

Hi @diegormsouza, I just got a simple Google Colab notebook to work with

!wget https://raw.githubusercontent.com/SciTools/cartopy/master/tools/cartopy_feature_download.py
!python cartopy_feature_download.py physical

[diegormsouza]

Uow! Hi @rcomer, thanks for the hint!

Could add the coastlines and all other features following this procedure!

Huge thanks!

# Installing the Cartopy library
!pip install cartopy
!pip install shapely --no-binary shapely --force

# Downloading the Cartopy maps from AWS
!wget https://raw.githubusercontent.com/SciTools/cartopy/master/tools/cartopy_feature_download.py
!python cartopy_feature_download.py physical
!python cartopy_feature_download.py cultural

import matplotlib.pyplot as plt
import cartopy.feature as cfeature

fig = plt.figure(figsize=(10, 5))
ax = fig.add_subplot(1, 1, 1, projection=ccrs.Robinson())

# make the map global rather than have it zoom in to
# the extents of any plotted data
ax.set_global()

# add some features
ax.coastlines(resolution='110m', color='black', linewidth=0.8)
ax.add_feature(cfeature.LAND)
ax.add_feature(cfeature.OCEAN)
ax.add_feature(cfeature.COASTLINE)
ax.add_feature(cfeature.LAKES, alpha=0.5)
ax.add_feature(cfeature.RIVERS)
ax.add_feature(cfeature.BORDERS)

plt.show()

[ucyo]

Another option is to replace line 313 in .../site-packages/cartopy/io/shapereader.py

    NE_URL_TEMPLATE = ('https://naciscdn.org/naturalearth/{resolution}'
                       '/{category}/ne_{resolution}_{name}.zip')

to

    _NE_URL_TEMPLATE = ('https://naturalearth.s3.amazonaws.com/{resolution}_{category}/ne_{resolution}_{name}.zip')

[rcomer]

@ucyo that was done for Cartopy v0.20, which was released two days after @diegormsouza found his solution. The current advice would therefore be to upgrade to the latest Cartopy version.

[LanaaaG]

@diegormsouza Hi! Ⅰ tried your way but Ⅰ still get "

--2021-10-30 14:38:14-- https://raw.githubusercontent.com/SciTools/cartopy/master/tools/cartopy_feature_download.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 0.0.0.0
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|0.0.0.0|:443... failed: Connection refused.
"

Do you have any idea how to deal with this problem? Thanks in advance.

[diegormsouza]

@LanaaaG Hi! I just tried the following on Colab and it worked well!

# Installing the Cartopy library
!pip install cartopy
!pip install shapely --no-binary shapely --force

# Downloading the Cartopy default maps from AWS (this is necessary due to a recent problems with the NaturalEarth server)
!wget https://raw.githubusercontent.com/SciTools/cartopy/master/tools/cartopy_feature_download.py
!python cartopy_feature_download.py physical
!python cartopy_feature_download.py cultural
!python cartopy_feature_download.py cultural-extra

Are you trying exactly this?
Thanks

[LanaaaG]

@diegormsouza Hi! Ⅰ retried your code in Jupyter (with campus network) and it worked! Thank you! Ⅰ guess the problem is something wrong with the previous network firewall.