Move file hosting to S3 via AWS public dataset

[nvkelso]

The data has been hosted by FSU (huge thank you!) for the last 11 years.

NACIS (Natural Earth's formal project sponsor) is in discussions with AWS to host the data files in their public dataset program.

[nvkelso]

AWS Public Dataset approval has been granted. Need to setup new org and S3 bucket next (with CW logging enabled for analytics), post files, update the download manager, and document direct S3 download links.

[nvkelso]

The S3 bucket has been setup and initial data transfer of v4 released files has kicked off.

[barronh]

This is great news! What is the status of documentation? I'd suggest making a registry entry that documents these buckets. I think you can just fork the registry repo, add a yaml file with meta data, and make a PR. The biggest commitment is filling in the template.

https://registry.opendata.aws/

thanks for all you do!

[nvkelso]

The related PR (WIP) is awslabs/open-data-registry#853.

[nvkelso]

That PR is ready to merge now.

But looking at the path structure of the S3 bucket it has 10m_cultural instead of 10m/cultural so I need to have a closer look, they should match before switching over naciscdn.org to S3 instead of FREAC.

[nvkelso]

The AWS public data registry PR has been merged and the data is live. But it needs a little bit more automation for updates, and really versioned releases.

[nvkelso]

I'm adding the missing 10m and 50m rasters now.

[nvkelso]

Versioned will live in dirs like:

• https://naturalearth.s3.us-west-2.amazonaws.com/4.1.1 (done)
• https://naturalearth.s3.us-west-2.amazonaws.com/5.0.0 (stub'd)

And latest stable release will live at the root:

• https://naturalearth.s3.us-west-2.amazonaws.com/ (done)

The same will be true for the naciscdn links:

• https://www.naturalearthdata.com/https://naciscdn.org/naturalearth/4.1.1 (done)
• https://www.naturalearthdata.com/https://naciscdn.org/naturalearth/5.0.0 (stub'd)

And latest stable release will live at the root:

• https://www.naturalearthdata.com/https://naciscdn.org/naturalearth/ (done)