[Fixes #7060] Refactoring of thumbnail creation (#7061)

* Bump urllib3 from 1.26.2 to 1.26.3 (#6908) Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.2 to 1.26.3. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/1.26.3/CHANGES.rst) - [Commits](urllib3/urllib3@1.26.2...1.26.3) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni <[email protected]> * [Fixes #6880] Circle CI upload tests fail irregulary (#6881) * [Fixes #6880] Circle CI upload tests fail irregulary * CircleCI test fix: sometimes expires due to upload timeout in the test environment * - Avoid infinite loop on upload testing * Revert "CircleCI test fix: sometimes expires due to upload timeout in the test environment" This reverts commit 66139fd. Co-authored-by: Alessio Fabiani <[email protected]> Co-authored-by: afabiani <[email protected]> * [Fixes #6914] Remove "add to basket" tool for documents and maps (#6915) * Added malnajdi as contributor * [Fixes #6910] meaningful filename for document download (#6911) * get meaningful document filenames on download * - Strip extension from document title before slugify it (e.g.: image.jpg instead of imagejpg.jpg) Co-authored-by: afabiani <[email protected]> Co-authored-by: Alessio Fabiani <[email protected]> * - CircleCI Upload Tests: trying to reduce more the risk of infinite loop on "wait_for_progress" * [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION (#6913) * [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION * [Fixes #6916] unit test for GEOSERVER_LOCATION * Bump django-cors-headers from 3.6.0 to 3.7.0 (#6901) Bumps [django-cors-headers](https://github.com/adamchainz/django-cors-headers) from 3.6.0 to 3.7.0. - [Release notes](https://github.com/adamchainz/django-cors-headers/releases) - [Changelog](https://github.com/adamchainz/django-cors-headers/blob/master/HISTORY.rst) - [Commits](adamchainz/django-cors-headers@3.6.0...3.7.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump amqp from 5.0.3 to 5.0.5 (#6905) Bumps [amqp](https://github.com/celery/py-amqp) from 5.0.3 to 5.0.5. - [Release notes](https://github.com/celery/py-amqp/releases) - [Changelog](https://github.com/celery/py-amqp/blob/master/Changelog) - [Commits](celery/py-amqp@v5.0.3...v5.0.5) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump pip from 21.0 to 21.0.1 (#6900) Bumps [pip](https://github.com/pypa/pip) from 21.0 to 21.0.1. - [Release notes](https://github.com/pypa/pip/releases) - [Changelog](https://github.com/pypa/pip/blob/master/NEWS.rst) - [Commits](pypa/pip@21.0...21.0.1) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump coverage from 5.3.1 to 5.4 (#6903) Bumps [coverage](https://github.com/nedbat/coveragepy) from 5.3.1 to 5.4. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](nedbat/coveragepy@coverage-5.3.1...coverage-5.4) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump pytest from 6.2.1 to 6.2.2 (#6907) Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.1 to 6.2.2. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst) - [Commits](pytest-dev/pytest@6.2.1...6.2.2) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump djangorestframework-gis from 0.16 to 0.17 (#6902) Bumps [djangorestframework-gis](https://github.com/openwisp/django-rest-framework-gis) from 0.16 to 0.17. - [Release notes](https://github.com/openwisp/django-rest-framework-gis/releases) - [Changelog](https://github.com/openwisp/django-rest-framework-gis/blob/master/CHANGES.rst) - [Commits](openwisp/django-rest-framework-gis@v0.16.0...v0.17.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * - Algin setup.cfg to requirements.txt * [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has… (#6923) * [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has been uploaded * - Add REST APIs test suite to CircleCI * [Fixes #6918] Removal of QGIS support (#6919) * [Cleanup and Refactor] Remove QGIS server backend dependencies * [Cleanup and Refactor] Remove QGIS server backend dependencies * - Fix LGTM issues * allow Basic authenticated requests in LOCKDOWN mode * fix to avoid circular import * flake8 check fix * added tests * first draft * remove service integration * bbox structure fix * layers and maps endpoints integration * formatting * flake formatting * fix for longitude exceeding world width * description * fix for longitude exceeding world width 2 * fixes requested by PR review * exceeding latitude fix * immutability fix * thumbs background configs * poc of refactored wizard layer thumbnail * missing EPSG prefix * wizard map thumbnail * correct layers order in a map * security issue fix * fix for small thumbnails * comments to wizard wiget refactoring * preserve thumbnail's ratio * preserve thumbnail's ratio fix * fix thumbnail URL for layer's wizard panels * typo * bbox method fix * tests ongoing * tests ongoing * preparing the thumbnail.js for the refactoring in MapStore * tests and formatting * remove legacy thumbnail generation * test fix * minimal documentation of background generator options * remove outdated tests * add base.tests.ThumbnailTests cleanup * formatting * failing CI tests fix * - Add thumbs tests * - Fix test cases * - Added few more consistency checks in order to make sure we catch 'zero-area' cases * - Fix test cases * - Fix test cases * - Fix test cases * fixes to thumbnail set frm detail pages * - Fix test cases * - Fix test cases * - Fix test cases * - Test fixes * - Test fixes * - Test fixes Co-authored-by: allyoucanmap <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni <[email protected]> Co-authored-by: Alessio Fabiani <[email protected]> Co-authored-by: afabiani <[email protected]> Co-authored-by: Florian Hoedt <[email protected]> Co-authored-by: Mohammed Y. Alnajdi <[email protected]> Co-authored-by: biegan <[email protected]> (cherry picked from commit c6c6ec4) # Conflicts: # geonode/layers/views.py
GeoNode · Mar 17, 2021 · 47aaf73 · 47aaf73
1 parent 729fdb3
commit 47aaf73
Show file tree

Hide file tree

Showing 78 changed files with 3,234 additions and 975 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -123,19 +123,19 @@ workflows:
           name: geonode_test_suite_smoke
           load_docker_cache: true
           save_docker_cache: true
-          test_suite: coverage run --branch --source=geonode manage.py test -v 3 --keepdb geonode.tests.smoke geonode.tests.test_message_notifications geonode.tests.test_rest_api geonode.tests.test_search geonode.tests.test_utils
+          test_suite: ./test.sh geonode.tests.smoke geonode.tests.test_message_notifications geonode.tests.test_rest_api geonode.tests.test_search geonode.tests.test_utils
       - build:
           name: geonode_test_suite
           load_docker_cache: true
           save_docker_cache: false
-          test_suite: coverage run --branch --source=geonode manage.py test -v 3 --keepdb $(python -c "import sys;from geonode import settings;sys.stdout.write('\'' '\''.join([a+'\''.tests'\'' for a in settings.GEONODE_APPS]))")
+          test_suite: ./test.sh $(python -c "import sys;from geonode import settings;sys.stdout.write('\'' '\''.join([a+'\''.tests'\'' for a in settings.GEONODE_APPS]))") geonode.thumbs.tests
           requires:
             - geonode_test_suite_smoke
       - build:
           name: geonode_test_rest_apis
           load_docker_cache: true
           save_docker_cache: false
-          test_suite: coverage run --branch --source=geonode manage.py test -v 3 --keepdb geonode.base.api.tests geonode.layers.api.tests geonode.maps.api.tests geonode.documents.api.tests geonode.geoapps.api.tests
+          test_suite: ./test.sh geonode.base.api.tests geonode.layers.api.tests geonode.maps.api.tests geonode.documents.api.tests geonode.geoapps.api.tests
           requires:
             - geonode_test_suite_smoke
       - build:
@@ -156,7 +156,7 @@ workflows:
           name: geonode_test_integration_monitoring
           load_docker_cache: true
           save_docker_cache: false
-          test_suite: coverage run --branch --source=geonode manage.py test -v 3 --keepdb geonode.tests.smoke geonode.monitoring.tests.integration
+          test_suite: ./test.sh geonode.tests.smoke geonode.monitoring.tests.integration
           requires:
             - geonode_test_suite_smoke
 

diff --git a/.env_dev b/.env_dev
@@ -20,11 +20,11 @@ POSTGRES_PASSWORD=postgres
 GEONODE_DATABASE=geonode
 GEONODE_DATABASE_PASSWORD=geonode
 GEONODE_GEODATABASE=geonode_data
-GEONODE_GEODATABASE_PASSWORD=geonode_data
+GEONODE_GEODATABASE_PASSWORD=geonode
 GEONODE_DATABASE_SCHEMA=public
 GEONODE_GEODATABASE_SCHEMA=public
 DATABASE_URL=postgis://geonode:geonode@localhost:5432/geonode
-GEODATABASE_URL=postgis://geonode_data:geonode@localhost:5432/geonode_data
+GEODATABASE_URL=postgis://geonode:geonode@localhost:5432/geonode_data
 GEONODE_DB_CONN_MAX_AGE=0
 GEONODE_DB_CONN_TOUT=5
 DEFAULT_BACKEND_DATASTORE=datastore
@@ -39,9 +39,9 @@ SITEURL=http://localhost:8000/
 
 ALLOWED_HOSTS="['django', '*']"
 
-# DEFAULT_BACKEND_UPLOADER=geonode.importer
-# TIME_ENABLED=True
-# MOSAIC_ENABLED=False
+DEFAULT_BACKEND_UPLOADER=geonode.importer
+TIME_ENABLED=True
+MOSAIC_ENABLED=False
 HAYSTACK_SEARCH=False
 HAYSTACK_ENGINE_URL=http://elasticsearch:9200/
 HAYSTACK_ENGINE_INDEX_NAME=haystack

diff --git a/.env_test b/.env_test
@@ -12,9 +12,8 @@ DEBUG=False
 
 DJANGO_SETTINGS_MODULE=geonode.settings
 GEONODE_INSTANCE_NAME=geonode
-GEONODE_LB_HOST_IP
-GEONODE_LB_PORT
-
+GEONODE_LB_HOST_IP=
+GEONODE_LB_PORT=
 
 POSTGRES_USER=postgres
 POSTGRES_PASSWORD=postgres
@@ -32,14 +31,13 @@ DEFAULT_BACKEND_DATASTORE=datastore
 BROKER_URL=amqp://guest:guest@rabbitmq:5672/
 ASYNC_SIGNALS=True
 
-
 SITEURL=http://localhost:8001/
 
 STATIC_ROOT=/mnt/volumes/statics/static/
 MEDIA_ROOT=/mnt/volumes/statics/uploaded/
 GEOIP_PATH=/mnt/volumes/statics/geoip.db
 
-ALLOWED_HOSTS=['django', '*']
+ALLOWED_HOSTS="['django', '*']"
 
 DEFAULT_BACKEND_UPLOADER=geonode.importer
 TIME_ENABLED=True
@@ -144,7 +142,7 @@ DEFAULT_FROM_EMAIL='GeoNode <[email protected]>'
 # Session/Access Control
 LOCKDOWN_GEONODE=False
 CORS_ORIGIN_ALLOW_ALL=True
-X_FRAME_OPTIONS=ALLOW-FROM ALL
+X_FRAME_OPTIONS="ALLOW-FROM ALL"
 SESSION_EXPIRED_CONTROL_ENABLED=True
 DEFAULT_ANONYMOUS_VIEW_PERMISSION=True
 DEFAULT_ANONYMOUS_DOWNLOAD_PERMISSION=True

diff --git a/.gitignore b/.gitignore
@@ -7,7 +7,6 @@ pip-log.txt
 MANIFEST
 dist
 .idea/
-.vscode/
 .pytest_cache/
 pip-wheel-metadata/
 # supporting stuff pulled in by paver script

diff --git a/.vscode/launch.json b/.vscode/launch.json
@@ -0,0 +1,20 @@
+{
+    // Use IntelliSense to learn about possible attributes.
+    // Hover to view descriptions of existing attributes.
+    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "Python: Django",
+            "type": "python",
+            "request": "launch",
+            "program": "${workspaceFolder}/manage.py",
+            "envFile": "${workspaceFolder}/.env_dev",
+            "args": [
+                "runserver",
+                "0.0.0.0:8000"
+            ],
+            "django": true
+        }
+    ]
+}
diff --git a/codecov.yml b/codecov.yml
@@ -1,26 +1,26 @@
-codecov:
-  notify:
-    require_ci_to_pass: yes
-
-coverage:
-  precision: 2
-  round: down
-  range: "60...100"
-
-  status:
-    project: yes
-    patch: yes
-    changes: no
-
-parsers:
-  gcov:
-    branch_detection:
-      conditional: yes
-      loop: yes
-      method: no
-      macro: no
-
-comment:
-  layout: "header, diff"
-  behavior: default
-  require_changes: no
+codecov:
+  notify:
+    require_ci_to_pass: yes
+
+coverage:
+  precision: 2
+  round: down
+  range: "60...100"
+
+  status:
+    project: yes
+    patch: yes
+    changes: no
+
+parsers:
+  gcov:
+    branch_detection:
+      conditional: yes
+      loop: yes
+      method: no
+      macro: no
+
+comment:
+  layout: "header, diff"
+  behavior: default
+  require_changes: no
diff --git a/geonode/base/models.py b/geonode/base/models.py
@@ -1507,7 +1507,7 @@ def save_thumbnail(self, filename, image):
                             storage.path(_upload_path)
                         )
                     except Exception as e:
-                        logger.debug(e)
+                        logger.exception(e)
 
                 try:
                     # Optimize the Thumbnail size and resolution
@@ -1518,9 +1518,9 @@ def save_thumbnail(self, filename, image):
                         (_default_thumb_size['width'], _default_thumb_size['height']),
                         resample=Image.ANTIALIAS)
                     cover = ImageOps.fit(im, (_default_thumb_size['width'], _default_thumb_size['height']))
-                    cover.save(storage.path(_upload_path), format='JPEG')
+                    cover.save(storage.path(_upload_path), format='PNG')
                 except Exception as e:
-                    logger.debug(e)
+                    logger.exception(e)
 
                 # check whether it is an URI or not
                 parsed = urlsplit(url)
@@ -1551,10 +1551,10 @@ def save_thumbnail(self, filename, image):
                     thumbnail_url=url
                 )
         except Exception as e:
-            logger.debug(
-                'Error when generating the thumbnail for resource %s. (%s)' %
-                (self.id, str(e)))
-            logger.warn('Check permissions for file %s.' % upload_path)
+            logger.error(
+                f'Error when generating the thumbnail for resource {self.id}. ({e})'
+            )
+            logger.error(f'Check permissions for file {upload_path}.')
             try:
                 Link.objects.filter(resource=self, name='Thumbnail').delete()
                 _thumbnail_url = staticfiles.static(settings.MISSING_THUMBNAIL)
@@ -1575,9 +1575,9 @@ def save_thumbnail(self, filename, image):
                     thumbnail_url=_thumbnail_url
                 )
             except Exception as e:
-                logger.debug(
-                    'Error when generating the thumbnail for resource %s. (%s)' %
-                    (self.id, str(e)))
+                logger.error(
+                    f'Error when generating the thumbnail for resource {self.id}. ({e})'
+                )
 
     def set_missing_info(self):
         """Set default permissions and point of contacts.

diff --git a/geonode/base/tests.py b/geonode/base/tests.py
@@ -81,6 +81,9 @@ def setUp(self):
         super(ThumbnailTests, self).setUp()
         self.rb = ResourceBase.objects.create()
 
+    def tearDown(self):
+        super().tearDown()
+
     def test_initial_behavior(self):
         """
         Tests that an empty resource has a missing image as default thumbnail.
@@ -102,10 +105,16 @@ def test_monochromatic_image(self, image):
         """
         Tests that an monochromatic image does not change the current resource thumbnail.
         """
+        filename = 'test-thumb'
+
         current = self.rb.get_thumbnail_url()
-        self.rb.save_thumbnail('test-thumb', image)
+        self.rb.save_thumbnail(filename, image)
         self.assertEqual(current, urlparse(self.rb.get_thumbnail_url()).path)
 
+        # cleanup: remove saved thumbnail
+        thumb_utils.remove_thumbs(filename)
+        self.assertFalse(thumb_utils.thumb_exists(filename))
+
     @patch('PIL.Image.open', return_value=test_image)
     def test_thumb_utils_methods(self, image):
         """
@@ -122,6 +131,10 @@ def test_thumb_utils_methods(self, image):
         self.assertTrue(thumb_utils.thumb_exists(filename))
         self.assertEqual(thumb_utils.thumb_size(upload_path), 10000)
 
+        # cleanup: remove saved thumbnail
+        thumb_utils.remove_thumbs(filename)
+        self.assertFalse(thumb_utils.thumb_exists(filename))
+
 
 class TestThumbnailUrl(GeoNodeBaseTestSupport):
 
@@ -950,6 +963,7 @@ def __get_last_thesaurus():
 
 @override_settings(THESAURUS_DEFAULT_LANG="en")
 class TestThesaurusAvailableForm(TestCase):
+    #  loading test thesausurs
     fixtures = [
         "test_thesaurus.json"
     ]

diff --git a/geonode/client/templates/ol/layers/layer_ol2_map.html b/geonode/client/templates/ol/layers/layer_ol2_map.html
@@ -61,8 +61,8 @@
                "https://c.tile.openstreetmap.org/${z}/${x}/${y}.png"]),
           	source
     	];
-
-    	var map = new OpenLayers.Map({
+        // olMap is used by thumbnail.js to POST to the thumbnail update view
+    	var map = window.olMap = new OpenLayers.Map({
             div: 'preview_map',
             projection: new OpenLayers.Projection('EPSG:900913'),
             layers: layers,

diff --git a/geonode/client/templates/ol/maps/map_ol2.html b/geonode/client/templates/ol/maps/map_ol2.html
@@ -46,7 +46,8 @@
         layers.push(source);
         {% endif %}
     {% endfor %}
-    var map = new OpenLayers.Map({
+    // olMap is used by thumbnail.js to POST to the thumbnail update view
+    window.olMap = new OpenLayers.Map({
         div: 'preview_map',
         projection: new OpenLayers.Projection('EPSG:900913'),
         layers: layers,