Skip to content

Commit

Permalink
Harvester arcgis (#8229)
Browse files Browse the repository at this point in the history
* Bump urllib3 from 1.26.2 to 1.26.3 (#6908)

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.2 to 1.26.3.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.3/CHANGES.rst)
- [Commits](urllib3/urllib3@1.26.2...1.26.3)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Toni <[email protected]>

* [Fixes #6880] Circle CI upload tests fail irregulary (#6881)

* [Fixes #6880] Circle CI upload tests fail irregulary

* CircleCI test fix: sometimes expires due to upload timeout in the test environment

* - Avoid infinite loop on upload testing

* Revert "CircleCI test fix: sometimes expires due to upload timeout in the test environment"

This reverts commit 66139fd.

Co-authored-by: Alessio Fabiani <[email protected]>
Co-authored-by: afabiani <[email protected]>

* [Fixes #6914] Remove "add to basket" tool for documents and maps (#6915)

* Added malnajdi as contributor

* [Fixes #6910] meaningful filename for document download (#6911)

* get meaningful document filenames on download

* - Strip extension from document title before slugify it (e.g.: image.jpg instead of imagejpg.jpg)

Co-authored-by: afabiani <[email protected]>
Co-authored-by: Alessio Fabiani <[email protected]>

* - CircleCI Upload Tests: trying to reduce more the risk of infinite loop on "wait_for_progress"

* [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION (#6913)

* [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION

* [Fixes #6916] unit test for GEOSERVER_LOCATION

* Bump django-cors-headers from 3.6.0 to 3.7.0 (#6901)

Bumps [django-cors-headers](https://github.com/adamchainz/django-cors-headers) from 3.6.0 to 3.7.0.
- [Release notes](https://github.com/adamchainz/django-cors-headers/releases)
- [Changelog](https://github.com/adamchainz/django-cors-headers/blob/master/HISTORY.rst)
- [Commits](adamchainz/django-cors-headers@3.6.0...3.7.0)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump amqp from 5.0.3 to 5.0.5 (#6905)

Bumps [amqp](https://github.com/celery/py-amqp) from 5.0.3 to 5.0.5.
- [Release notes](https://github.com/celery/py-amqp/releases)
- [Changelog](https://github.com/celery/py-amqp/blob/master/Changelog)
- [Commits](celery/py-amqp@v5.0.3...v5.0.5)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pip from 21.0 to 21.0.1 (#6900)

Bumps [pip](https://github.com/pypa/pip) from 21.0 to 21.0.1.
- [Release notes](https://github.com/pypa/pip/releases)
- [Changelog](https://github.com/pypa/pip/blob/master/NEWS.rst)
- [Commits](pypa/pip@21.0...21.0.1)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump coverage from 5.3.1 to 5.4 (#6903)

Bumps [coverage](https://github.com/nedbat/coveragepy) from 5.3.1 to 5.4.
- [Release notes](https://github.com/nedbat/coveragepy/releases)
- [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst)
- [Commits](nedbat/coveragepy@coverage-5.3.1...coverage-5.4)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pytest from 6.2.1 to 6.2.2 (#6907)

Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.1 to 6.2.2.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@6.2.1...6.2.2)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump djangorestframework-gis from 0.16 to 0.17 (#6902)

Bumps [djangorestframework-gis](https://github.com/openwisp/django-rest-framework-gis) from 0.16 to 0.17.
- [Release notes](https://github.com/openwisp/django-rest-framework-gis/releases)
- [Changelog](https://github.com/openwisp/django-rest-framework-gis/blob/master/CHANGES.rst)
- [Commits](openwisp/django-rest-framework-gis@v0.16.0...v0.17.0)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* - Algin setup.cfg to requirements.txt

* [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has… (#6923)

* [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has been uploaded

* - Add REST APIs test suite to CircleCI

* [Fixes #6918] Removal of QGIS support (#6919)

* [Cleanup and Refactor] Remove QGIS server backend dependencies

* [Cleanup and Refactor] Remove QGIS server backend dependencies

* - Fix LGTM issues

* allow Basic authenticated requests in LOCKDOWN mode

* fix to avoid circular import

* flake8 check fix

* added tests

* [Fixes #6880] Circle CI upload tests fail irregulary (#6881)

* [Fixes #6880] Circle CI upload tests fail irregulary

* CircleCI test fix: sometimes expires due to upload timeout in the test environment

* - Avoid infinite loop on upload testing

* Revert "CircleCI test fix: sometimes expires due to upload timeout in the test environment"

This reverts commit 66139fd.

Co-authored-by: Alessio Fabiani <[email protected]>
Co-authored-by: afabiani <[email protected]>

* [Fixes #6914] Remove "add to basket" tool for documents and maps (#6915)

* Added malnajdi as contributor

* Bump pip from 21.0 to 21.0.1 (#6900)

Bumps [pip](https://github.com/pypa/pip) from 21.0 to 21.0.1.
- [Release notes](https://github.com/pypa/pip/releases)
- [Changelog](https://github.com/pypa/pip/blob/master/NEWS.rst)
- [Commits](pypa/pip@21.0...21.0.1)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* - Algin setup.cfg to requirements.txt

* [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has… (#6923)

* [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has been uploaded

* - Add REST APIs test suite to CircleCI

* [Fixes #6918] Removal of QGIS support (#6919)

* [Cleanup and Refactor] Remove QGIS server backend dependencies

* [Cleanup and Refactor] Remove QGIS server backend dependencies

* - Fix LGTM issues

* allow Basic authenticated requests in LOCKDOWN mode

* fix to avoid circular import

* - Align to upstream master branch

* [Fixes #7945] Ingest harvested layer data to geonode

* Improve harvesting session and the admin

* fix migration files conflict

* Initial work for implementing stoppable harvesting sessions

* Implement aborting of harvesting celery tasks

Add the `AsynchronousHarvestingSession` model, which is used
to implement sessions for both refreshing of a harvester's
harvestable resources and for the harvesting of remote
resources.

Refactor the `admin`, `api` and `tasks` to use this new model.

Moved some functions out of `harvesting.utils` module in order
to avoid circular imports

* fix tests

* fix conflicts

* Add a couple more tests

* Uncomment `settings.py` line that designates the GeoNode test runner as the one to be used

* Add harvesting scheduler task

* Improving support for the harvesting scheduler task

* Continue implementation of the harvesting scheduler task

* Remove the dependency on `django-celery-beat`, as the harvesting does not need it anymore

* Revert "Remove the dependency on `django-celery-beat`, as the harvesting does not need it anymore"

This reverts commit 8d50cf2.

* Remove reference to the django-celery-beat app in the harvesting models and signals

* Implement different strategy for the harvesting scheduler

- remove django-celery-beat as a dependency
- implement a simple harvesting scheduler as a celery task
- implement an action for resetting a harvester's status
- add (and fix) tests

* Add harvesting scheduler task

* Improving support for the harvesting scheduler task

* Continue implementation of the harvesting scheduler task

* Remove the dependency on `django-celery-beat`, as the harvesting does not need it anymore

* [Fixes #7945] Ingest harvested layer data to geonode

* Improve harvesting session and the admin

* fix migration files conflict

* Implement aborting of harvesting celery tasks

Add the `AsynchronousHarvestingSession` model, which is used
to implement sessions for both refreshing of a harvester's
harvestable resources and for the harvesting of remote
resources.

Refactor the `admin`, `api` and `tasks` to use this new model.

Moved some functions out of `harvesting.utils` module in order
to avoid circular imports

* fix tests

* fix conflicts

* Uncomment `settings.py` line that designates the GeoNode test runner as the one to be used

* Add harvesting scheduler task

* Improving support for the harvesting scheduler task

* Revert "Remove the dependency on `django-celery-beat`, as the harvesting does not need it anymore"

This reverts commit 8d50cf2.

* Remove reference to the django-celery-beat app in the harvesting models and signals

* Implement different strategy for the harvesting scheduler

- remove django-celery-beat as a dependency
- implement a simple harvesting scheduler as a celery task
- implement an action for resetting a harvester's status
- add (and fix) tests

* Improving support for the harvesting scheduler task

* Continue implementation of the harvesting scheduler task

* Begin implementation of a harvester for current GeoNode version

* Continue implementation of a modern harvester for remote GeoNode deployments

* Rebase and fix migration conflicts

* Fix invalid tiemporal computation after creating a new harvester

* fixing pagination (wip)

* Fix bugs with remote harvested layers

* Try out both geonode harvester worker classes and iron out inconsistencies

* Remove commented line

* Fix tests

* Harvester API serializer no longer tries to update harvestable resources upon creation.

This makes it behave in a similar way as when a harvester is created through the django admin

* Made GeonodeUnifiedHarvesterWorker the default harvester for geonode remotes

* Fix bits that were incorrectly merged when rebasing

* fix flake8 errors

* [Fixes #7945] Ingest harvested layer data to geonode

* Improve harvesting session and the admin

* fix migration files conflict

* Initial work for implementing stoppable harvesting sessions

* Implement aborting of harvesting celery tasks

Add the `AsynchronousHarvestingSession` model, which is used
to implement sessions for both refreshing of a harvester's
harvestable resources and for the harvesting of remote
resources.

Refactor the `admin`, `api` and `tasks` to use this new model.

Moved some functions out of `harvesting.utils` module in order
to avoid circular imports

* fix tests

* fix conflicts

* Remove accidental duplication of code that crept in during conflict resolution

* Add a couple more tests

* Uncomment `settings.py` line that designates the GeoNode test runner as the one to be used

* Add harvesting scheduler task

* Improving support for the harvesting scheduler task

* Continue implementation of the harvesting scheduler task

* Remove the dependency on `django-celery-beat`, as the harvesting does not need it anymore

* Revert "Remove the dependency on `django-celery-beat`, as the harvesting does not need it anymore"

This reverts commit 8d50cf2.

* Remove reference to the django-celery-beat app in the harvesting models and signals

* Implement different strategy for the harvesting scheduler

- remove django-celery-beat as a dependency
- implement a simple harvesting scheduler as a celery task
- implement an action for resetting a harvester's status
- add (and fix) tests

* Add harvesting scheduler task

* Continue implementation of the harvesting scheduler task

* Improve harvesting session and the admin

* Initial work for implementing stoppable harvesting sessions

* Implement aborting of harvesting celery tasks

Add the `AsynchronousHarvestingSession` model, which is used
to implement sessions for both refreshing of a harvester's
harvestable resources and for the harvesting of remote
resources.

Refactor the `admin`, `api` and `tasks` to use this new model.

Moved some functions out of `harvesting.utils` module in order
to avoid circular imports

* fix tests

* Update signature of get_resource method in WMS harvester worker

* Add harvesting scheduler task

* Improving support for the harvesting scheduler task

* Continue implementation of the harvesting scheduler task

* Revert "Remove the dependency on `django-celery-beat`, as the harvesting does not need it anymore"

This reverts commit 8d50cf2.

* Remove reference to the django-celery-beat app in the harvesting models and signals

* Implement different strategy for the harvesting scheduler

- remove django-celery-beat as a dependency
- implement a simple harvesting scheduler as a celery task
- implement an action for resetting a harvester's status
- add (and fix) tests

* Improving support for the harvesting scheduler task

* fix migration conflicts

* Begin implementation of ArcGIS-related harvester

* Continue implementation of ArcGIS-related harvester

* Continue implementation of arcgis harvester

* Continue implementation of arcgis harvester

* HAndling harvesting of MapServer layers

* Move gxp enumerations from the services package to the layers one

* Modify `dataset_detail()` view in order to configure gxp layer for layers that come from ArcGIS REST services

* Conclude implementation of ArcGIS harvester

* Add some tests

* Update code

* Implement resource_name_filter for ESRI harvester

* Minor changes to keep compatibility with Remote Services

* [ArcGIS Harvester] Make sure we can parse also nested services (e.g.: https://pro-ags2.dfs.un.org/arcgis/rest/services/UNMISS/UNMISS_Road_Rehabilitation/MapServer)

* [Remote Services] Re-enable ArcGIS Remote Services

* Improve WMS harvester based on recent develpments

* - Better management of the "ll_bbox" and "thumbnail" at the "ResourceManager" level

* Remove `name` from the default geonode parameters generated by the base harvester

This was a regression. Since GeoNode documents do not have a name, this field must only be added when relevant (e.g. when creating datasets)

* [CircleCI] Fix test cases

Co-authored-by: Giovanni Allegri <[email protected]>
Co-authored-by: allyoucanmap <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Toni <[email protected]>
Co-authored-by: Alessio Fabiani <[email protected]>
Co-authored-by: afabiani <[email protected]>
Co-authored-by: Florian Hoedt <[email protected]>
Co-authored-by: Mohammed Y. Alnajdi <[email protected]>
Co-authored-by: biegan <[email protected]>
Co-authored-by: meomancer <[email protected]>
  • Loading branch information
11 people authored Oct 25, 2021
1 parent 040abed commit b2f2ddc
Show file tree
Hide file tree
Showing 40 changed files with 1,707 additions and 294 deletions.
5 changes: 4 additions & 1 deletion geonode/base/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@
from geonode.utils import (
bbox_to_wkt,
find_by_attr,
bbox_to_projection,
is_monochromatic_image)
from geonode.groups.models import GroupProfile
from geonode.security.utils import get_visible_resources, get_geoapp_subtypes
Expand Down Expand Up @@ -1426,7 +1427,9 @@ def set_bbox_polygon(self, bbox, srid):
match = re.match(r'^(EPSG:)?(?P<srid>\d{4,6})$', str(srid))
bbox_polygon.srid = int(match.group('srid')) if match else 4326
try:
self.ll_bbox_polygon = bbox_polygon.transform(4326, clone=True)
# self.ll_bbox_polygon = bbox_polygon.transform(4326, clone=True)
self.ll_bbox_polygon = Polygon.from_bbox(
bbox_to_projection(list(bbox_polygon.extent) + [srid])[:-1])
except Exception as e:
logger.error(e)
self.ll_bbox_polygon = bbox_polygon
Expand Down
3 changes: 0 additions & 3 deletions geonode/harvesting/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -334,15 +334,13 @@ class HarvestableResourceAdmin(admin.ModelAdmin):
"last_harvested",
"unique_identifier",
"title",
"abstract",
"show_link_to_harvester",
"should_be_harvested",
"remote_resource_type",
)
readonly_fields = (
"unique_identifier",
"title",
"abstract",
"harvester",
"last_updated",
"last_refreshed",
Expand All @@ -360,7 +358,6 @@ class HarvestableResourceAdmin(admin.ModelAdmin):
)
search_fields = (
"title",
"abstract",
)
list_editable = (
"should_be_harvested",
Expand Down
8 changes: 1 addition & 7 deletions geonode/harvesting/api/serializers.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,13 +162,7 @@ def create(self, validated_data):
f"Either omit it or provide a "
f"value of {models.Harvester.STATUS_READY!r}"
)
harvester = super().create(validated_data)
available = harvester.update_availability()
if available:
harvester.status = harvester.STATUS_UPDATING_HARVESTABLE_RESOURCES
harvester.save()
tasks.update_harvestable_resources.apply_async(args=(harvester.pk,))
return harvester
return super().create(validated_data)

def update(self, instance: models.Harvester, validated_data):
"""Update harvester and perform any required business logic as a side-effect.
Expand Down
34 changes: 0 additions & 34 deletions geonode/harvesting/apps.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,43 +20,10 @@
from django.apps import AppConfig
from django.conf import settings
from django.conf.urls import url, include
from django.db.models.signals import post_migrate

from . import config


def run_setup_hooks(sender, **kwargs):
from django.utils import timezone

# Initialize periodic tasks
if 'django_celery_beat' in settings.INSTALLED_APPS and \
getattr(settings, 'CELERY_BEAT_SCHEDULER', None) == 'django_celery_beat.schedulers:DatabaseScheduler':
from django_celery_beat.models import (
IntervalSchedule,
PeriodicTask,
)

secs = config.get_setting("HARVESTER_SCHEDULER_FREQUENCY_MINUTES") * 60
check_intervals = IntervalSchedule.objects.filter(every=secs, period="seconds")
if not check_intervals.exists():
check_interval, _ = IntervalSchedule.objects.get_or_create(
every=secs,
period="seconds"
)
else:
check_interval = check_intervals.first()

PeriodicTask.objects.update_or_create(
name="harvesting-scheduler",
defaults=dict(
task="geonode.harvesting.tasks.harvesting_scheduler",
interval=check_interval,
args='',
start_time=timezone.now()
)
)


class HarvestingAppConfig(AppConfig):

name = "geonode.harvesting"
Expand All @@ -67,7 +34,6 @@ def ready(self):
urlpatterns += [
url(r'^api/v2/', include('geonode.harvesting.api.urls'))
]
post_migrate.connect(run_setup_hooks, sender=self)
settings.CELERY_BEAT_SCHEDULE['harvesting-scheduler'] = {
"task": "geonode.harvesting.tasks.harvesting_scheduler",
"schedule": config.get_setting("HARVESTER_SCHEDULER_FREQUENCY_MINUTES") * 60,
Expand Down
4 changes: 2 additions & 2 deletions geonode/harvesting/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,9 @@


_DEFAULT_HARVESTERS: typing.Final = [
"geonode.harvesting.harvesters.geonodeharvester.GeonodeLegacyHarvester",
"geonode.harvesting.harvesters.geonodeharvester.GeonodeUnifiedHarvesterWorker",
"geonode.harvesting.harvesters.wms.OgcWmsHarvester",
# "geonode.harvesting.harvesters.geonode.GeonodeCswHarvester",
"geonode.harvesting.harvesters.arcgis.ArcgisHarvesterWorker",
]


Expand Down
Loading

0 comments on commit b2f2ddc

Please sign in to comment.