[Fixes #7060] Refactoring of thumbnail creation (#7061)

* Bump urllib3 from 1.26.2 to 1.26.3 (#6908) Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.2 to 1.26.3. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/1.26.3/CHANGES.rst) - [Commits](urllib3/urllib3@1.26.2...1.26.3) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni <[email protected]> * [Fixes #6880] Circle CI upload tests fail irregulary (#6881) * [Fixes #6880] Circle CI upload tests fail irregulary * CircleCI test fix: sometimes expires due to upload timeout in the test environment * - Avoid infinite loop on upload testing * Revert "CircleCI test fix: sometimes expires due to upload timeout in the test environment" This reverts commit 66139fd. Co-authored-by: Alessio Fabiani <[email protected]> Co-authored-by: afabiani <[email protected]> * [Fixes #6914] Remove "add to basket" tool for documents and maps (#6915) * Added malnajdi as contributor * [Fixes #6910] meaningful filename for document download (#6911) * get meaningful document filenames on download * - Strip extension from document title before slugify it (e.g.: image.jpg instead of imagejpg.jpg) Co-authored-by: afabiani <[email protected]> Co-authored-by: Alessio Fabiani <[email protected]> * - CircleCI Upload Tests: trying to reduce more the risk of infinite loop on "wait_for_progress" * [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION (#6913) * [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION * [Fixes #6916] unit test for GEOSERVER_LOCATION * Bump django-cors-headers from 3.6.0 to 3.7.0 (#6901) Bumps [django-cors-headers](https://github.com/adamchainz/django-cors-headers) from 3.6.0 to 3.7.0. - [Release notes](https://github.com/adamchainz/django-cors-headers/releases) - [Changelog](https://github.com/adamchainz/django-cors-headers/blob/master/HISTORY.rst) - [Commits](adamchainz/django-cors-headers@3.6.0...3.7.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump amqp from 5.0.3 to 5.0.5 (#6905) Bumps [amqp](https://github.com/celery/py-amqp) from 5.0.3 to 5.0.5. - [Release notes](https://github.com/celery/py-amqp/releases) - [Changelog](https://github.com/celery/py-amqp/blob/master/Changelog) - [Commits](celery/py-amqp@v5.0.3...v5.0.5) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump pip from 21.0 to 21.0.1 (#6900) Bumps [pip](https://github.com/pypa/pip) from 21.0 to 21.0.1. - [Release notes](https://github.com/pypa/pip/releases) - [Changelog](https://github.com/pypa/pip/blob/master/NEWS.rst) - [Commits](pypa/pip@21.0...21.0.1) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump coverage from 5.3.1 to 5.4 (#6903) Bumps [coverage](https://github.com/nedbat/coveragepy) from 5.3.1 to 5.4. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](nedbat/coveragepy@coverage-5.3.1...coverage-5.4) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump pytest from 6.2.1 to 6.2.2 (#6907) Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.1 to 6.2.2. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst) - [Commits](pytest-dev/pytest@6.2.1...6.2.2) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump djangorestframework-gis from 0.16 to 0.17 (#6902) Bumps [djangorestframework-gis](https://github.com/openwisp/django-rest-framework-gis) from 0.16 to 0.17. - [Release notes](https://github.com/openwisp/django-rest-framework-gis/releases) - [Changelog](https://github.com/openwisp/django-rest-framework-gis/blob/master/CHANGES.rst) - [Commits](openwisp/django-rest-framework-gis@v0.16.0...v0.17.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * - Algin setup.cfg to requirements.txt * [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has… (#6923) * [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has been uploaded * - Add REST APIs test suite to CircleCI * [Fixes #6918] Removal of QGIS support (#6919) * [Cleanup and Refactor] Remove QGIS server backend dependencies * [Cleanup and Refactor] Remove QGIS server backend dependencies * - Fix LGTM issues * allow Basic authenticated requests in LOCKDOWN mode * fix to avoid circular import * flake8 check fix * added tests * first draft * remove service integration * bbox structure fix * layers and maps endpoints integration * formatting * flake formatting * fix for longitude exceeding world width * description * fix for longitude exceeding world width 2 * fixes requested by PR review * exceeding latitude fix * immutability fix * thumbs background configs * poc of refactored wizard layer thumbnail * missing EPSG prefix * wizard map thumbnail * correct layers order in a map * security issue fix * fix for small thumbnails * comments to wizard wiget refactoring * preserve thumbnail's ratio * preserve thumbnail's ratio fix * fix thumbnail URL for layer's wizard panels * typo * bbox method fix * tests ongoing * tests ongoing * preparing the thumbnail.js for the refactoring in MapStore * tests and formatting * remove legacy thumbnail generation * test fix * minimal documentation of background generator options * remove outdated tests * add base.tests.ThumbnailTests cleanup * formatting * failing CI tests fix * - Add thumbs tests * - Fix test cases * - Added few more consistency checks in order to make sure we catch 'zero-area' cases * - Fix test cases * - Fix test cases * - Fix test cases * fixes to thumbnail set frm detail pages * - Fix test cases * - Fix test cases * - Fix test cases * - Test fixes * - Test fixes * - Test fixes Co-authored-by: allyoucanmap <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni <[email protected]> Co-authored-by: Alessio Fabiani <[email protected]> Co-authored-by: afabiani <[email protected]> Co-authored-by: Florian Hoedt <[email protected]> Co-authored-by: Mohammed Y. Alnajdi <[email protected]> Co-authored-by: biegan <[email protected]> (cherry picked from commit c6c6ec4) # Conflicts: # .circleci/config.yml # geonode/base/tests.py # geonode/geoserver/helpers.py # geonode/layers/tests.py # geonode/layers/utils.py # geonode/layers/views.py # geonode/maps/views.py # geonode/security/tests.py # geonode/services/serviceprocessors/arcgis.py # requirements.txt # scripts/docker/env/production/django.env # setup.cfg
GeoNode · Mar 17, 2021 · 2a431d5 · 2a431d5
1 parent 91325b3
commit 2a431d5
Show file tree

Hide file tree

Showing 78 changed files with 3,933 additions and 1,408 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -26,16 +26,16 @@ jobs:
           steps:
             - restore_cache:
                 keys:
-                  - v1-docker-images-{{ .Branch }}
+                  - v2-docker-images-{{ .Branch }}
             - run:
                 name: Load Docker layers cache
                 command: |
                   docker load -i ~/docker-layers-cache.tar || true
 
       - run:
           name: Build the stack
-          command: docker-compose -f docker-compose.yml build
-          working_directory: scripts/spcgeonode/
+          command: docker-compose -f docker-compose-test.yml build
+          working_directory: ./
 
       - when:
           condition: <<parameters.save_docker_cache>>
@@ -47,26 +47,26 @@ jobs:
                   docker save -o ~/docker-layers-cache.tar $(docker images -a --format "{{.ID}}")
                 when: always
             - save_cache:
-                key: v1-docker-images-{{ .Branch }}-{{ epoch }}
+                key: v2-docker-images-{{ .Branch }}-{{ epoch }}
                 paths:
                   - ~/docker-layers-cache.tar
                 when: always
 
       - run:
           name: Start the stack
-          command: docker-compose -f docker-compose.yml up -d --build
-          working_directory: scripts/spcgeonode/
+          command: docker-compose -f docker-compose-test.yml up -d
+          working_directory: ./
 
       - run:
           name: Wait for everything to start...
           command: |
             n=1
-            m=60
+            m=10
             until [ $n -gt $m ]
             do
               sleep 60
-              DJANGO_STATUS=$(docker inspect --format="{{json .State.Health.Status}}" spcgeonode_django_1)
-              GEOSERVER_STATUS=$(docker inspect --format="{{json .State.Health.Status}}" spcgeonode_geoserver_1)
+              DJANGO_STATUS=$(docker inspect --format="{{json .State.Health.Status}}" django4geonode)
+              GEOSERVER_STATUS=$(docker inspect --format="{{json .State.Health.Status}}" geoserver4geonode)
               echo ""
               echo "Waited $n min (out of $m min)"
               if [[ $DJANGO_STATUS == '"healthy"' ]] && [[ $GEOSERVER_STATUS == '"healthy"' ]]; then
@@ -85,12 +85,12 @@ jobs:
 
       - run:
           name: Geoserver logs (debug)
-          command: docker logs spcgeonode_geoserver_1 --tail 500
+          command: docker logs geoserver4geonode --tail 500
           when: on_fail
 
       - run:
           name: Django logs (debug)
-          command: docker logs spcgeonode_django_1 --tail 500
+          command: docker logs django4geonode --tail 500
           when: on_fail
 
       - when:
@@ -99,63 +99,101 @@ jobs:
             - run:
                 name: Run test suite
                 command: |
-                  docker-compose -f docker-compose.yml exec django bash -c 'echo "Running <<parameters.test_suite>>"'
-                  docker-compose -f docker-compose.yml exec postgres psql -U postgres -c 'SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid <> pg_backend_pid();'
-                  docker-compose -f docker-compose.yml exec postgres createdb -U postgres -T postgres test_postgres
-                  docker-compose -f docker-compose.yml exec postgres createdb -U postgres -T postgres test_geonode
-                  docker-compose -f docker-compose.yml exec postgres createdb -U postgres -T postgres test_geonode_data
-                  docker-compose -f docker-compose.yml run --rm django bash -c 'ASYNC_SIGNALS=False python manage.py test -v 3 --keepdb <<parameters.test_suite>>'
-                working_directory: scripts/spcgeonode/
+                  docker-compose -f docker-compose-test.yml exec db psql -U postgres -c 'SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid <> pg_backend_pid();'
+                  docker-compose -f docker-compose-test.yml exec db createdb -U postgres -T postgres test_postgres
+                  docker-compose -f docker-compose-test.yml exec db createdb -U postgres -T postgres test_geonode
+                  docker-compose -f docker-compose-test.yml exec db createdb -U postgres -T postgres test_geonode_data
+                  docker-compose -f docker-compose-test.yml exec db psql -U postgres -d test_geonode -c 'CREATE EXTENSION IF NOT EXISTS postgis;'
+                  docker-compose -f docker-compose-test.yml exec db psql -U postgres -d test_geonode_data -c 'CREATE EXTENSION IF NOT EXISTS postgis;'
+                  docker-compose -f docker-compose-test.yml exec django bash -c '<<parameters.test_suite>>'
+                working_directory: ./
+                no_output_timeout: 10m
+            - run:
+                name: Run pep8 checks
+                command: |
+                  docker-compose -f docker-compose-test.yml exec django bash -c 'flake8 geonode'
+                  docker-compose -f docker-compose-test.yml exec django bash -c 'codecov; bash <(curl -s https://codecov.io/bash) -t 2c0e7780-1640-45f0-93a3-e103b057d8c8'
+                working_directory: ./
 
 workflows:
 
   commit:
     jobs:
       - build:
-          name: setup
+          name: geonode_test_suite_smoke
           load_docker_cache: true
           save_docker_cache: true
-          test_suite: 'geonode.tests.smoke'
+          test_suite: ./test.sh geonode.tests.smoke geonode.tests.test_message_notifications geonode.tests.test_rest_api geonode.tests.test_search geonode.tests.test_utils
       - build:
-          name: tests_integration
+          name: geonode_test_suite
           load_docker_cache: true
           save_docker_cache: false
-          test_suite: 'geonode.tests.integration'
+          test_suite: ./test.sh $(python -c "import sys;from geonode import settings;sys.stdout.write('\'' '\''.join([a+'\''.tests'\'' for a in settings.GEONODE_APPS]))") geonode.thumbs.tests
           requires:
-            - setup
+            - geonode_test_suite_smoke
       - build:
-          name: tests_core
+          name: geonode_test_rest_apis
           load_docker_cache: true
           save_docker_cache: false
-          test_suite: $(python -c "import sys;from geonode import settings;sys.stdout.write('\'' '\''.join([a+'\''.tests'\'' for a in settings.GEONODE_CORE_APPS]))")
+          test_suite: ./test.sh geonode.base.api.tests geonode.layers.api.tests geonode.maps.api.tests geonode.documents.api.tests geonode.geoapps.api.tests
           requires:
-            - setup
+            - geonode_test_suite_smoke
       - build:
-          name: tests_internal
+          name: geonode_test_integration_csw
           load_docker_cache: true
           save_docker_cache: false
-          test_suite: $(python -c "import sys;from geonode import settings;sys.stdout.write('\'' '\''.join([a+'\''.tests'\'' for a in settings.GEONODE_INTERNAL_APPS]))")
+          test_suite: ./test_csw.sh
           requires:
-            - setup
+            - geonode_test_suite_smoke
       - build:
-          name: tests_contrib
+          name: geonode_test_integration_upload
           load_docker_cache: true
           save_docker_cache: false
-          test_suite: $(python -c "import sys;from geonode import settings;sys.stdout.write('\'' '\''.join([a+'\''.tests'\'' for a in settings.GEONODE_CONTRIB_APPS]))")
+          test_suite: ./test_upload.sh
           requires:
-            - setup
-
-  nightly:
-    triggers:
-      - schedule:
-          cron: "0 0 * * *"
-          filters:
-            branches:
-              only:
-                - spcgeonode-release
-                - spcgeonode
-    jobs:
+            - geonode_test_suite_smoke
       - build:
-          load_docker_cache: false
-          save_docker_cache: true
-          test_suite: geonode.tests.smoke geonode.tests.integration $(python -c "import sys;from geonode import settings;sys.stdout.write('\'' '\''.join([a+'\''.tests'\'' for a in settings.GEONODE_APPS]))")
+          name: geonode_test_integration_monitoring
+          load_docker_cache: true
+          save_docker_cache: false
+          test_suite: ./test.sh geonode.tests.smoke geonode.monitoring.tests.integration
+          requires:
+            - geonode_test_suite_smoke
+
+      # TODO
+      # - build:
+      #     name: tests_integration
+      #     load_docker_cache: true
+      #     save_docker_cache: false
+      #     test_suite: 'geonode.tests.integration'
+      #     requires:
+      #       - geonode_test_suite_smoke
+      # - build:
+      #     name: tests_geoserver_integration
+      #     load_docker_cache: true
+      #     save_docker_cache: false
+      #     test_suite: 'geonode.geoserver.tests.integration'
+      #     requires:
+      #       - geonode_test_suite_smoke
+      # - build:
+      #     name: tests_upload_integration
+      #     load_docker_cache: true
+      #     save_docker_cache: false
+      #     test_suite: 'geonode.upload.tests.integration'
+      #     requires:
+      #       - geonode_test_suite_smoke
+
+  # nightly:
+  #   triggers:
+  #     - schedule:
+  #         cron: "0 0 * * *"
+  #         filters:
+  #           branches:
+  #             only:
+  #               - master
+  #               - 3.x
+  #   jobs:
+  #     - build:
+  #         load_docker_cache: false
+  #         save_docker_cache: true
+  #         test_suite: geonode.tests.smoke geonode.tests.integration $(python -c "import sys;from geonode import settings;sys.stdout.write('\'' '\''.join([a+'\''.tests'\'' for a in settings.GEONODE_APPS]))")
diff --git a/.env_dev b/.env_dev
@@ -20,7 +20,7 @@ POSTGRES_PASSWORD=postgres
 GEONODE_DATABASE=geonode
 GEONODE_DATABASE_PASSWORD=geonode
 GEONODE_GEODATABASE=geonode_data
-GEONODE_GEODATABASE_PASSWORD=geonode_data
+GEONODE_GEODATABASE_PASSWORD=geonode
 GEONODE_DATABASE_SCHEMA=public
 GEONODE_GEODATABASE_SCHEMA=public
 DATABASE_URL=postgis://geonode:geonode@localhost:5432/geonode
@@ -40,9 +40,9 @@ SITEURL=http://localhost:8000/
 
 ALLOWED_HOSTS="['django', '*']"
 
-# DEFAULT_BACKEND_UPLOADER=geonode.importer
-# TIME_ENABLED=True
-# MOSAIC_ENABLED=False
+DEFAULT_BACKEND_UPLOADER=geonode.importer
+TIME_ENABLED=True
+MOSAIC_ENABLED=False
 HAYSTACK_SEARCH=False
 HAYSTACK_ENGINE_URL=http://elasticsearch:9200/
 HAYSTACK_ENGINE_INDEX_NAME=haystack

diff --git a/.gitignore b/.gitignore
@@ -7,7 +7,6 @@ pip-log.txt
 MANIFEST
 dist
 .idea/
-.vscode/
 .pytest_cache/
 pip-wheel-metadata/
 # supporting stuff pulled in by paver script

diff --git a/.vscode/launch.json b/.vscode/launch.json
@@ -0,0 +1,20 @@
+{
+    // Use IntelliSense to learn about possible attributes.
+    // Hover to view descriptions of existing attributes.
+    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "Python: Django",
+            "type": "python",
+            "request": "launch",
+            "program": "${workspaceFolder}/manage.py",
+            "envFile": "${workspaceFolder}/.env_dev",
+            "args": [
+                "runserver",
+                "0.0.0.0:8000"
+            ],
+            "django": true
+        }
+    ]
+}
diff --git a/codecov.yml b/codecov.yml
@@ -1,26 +1,26 @@
-codecov:
-  notify:
-    require_ci_to_pass: yes
-
-coverage:
-  precision: 2
-  round: down
-  range: "60...100"
-
-  status:
-    project: yes
-    patch: yes
-    changes: no
-
-parsers:
-  gcov:
-    branch_detection:
-      conditional: yes
-      loop: yes
-      method: no
-      macro: no
-
-comment:
-  layout: "header, diff"
-  behavior: default
-  require_changes: no
+codecov:
+  notify:
+    require_ci_to_pass: yes
+
+coverage:
+  precision: 2
+  round: down
+  range: "60...100"
+
+  status:
+    project: yes
+    patch: yes
+    changes: no
+
+parsers:
+  gcov:
+    branch_detection:
+      conditional: yes
+      loop: yes
+      method: no
+      macro: no
+
+comment:
+  layout: "header, diff"
+  behavior: default
+  require_changes: no
diff --git a/geonode/base/models.py b/geonode/base/models.py
@@ -1403,7 +1403,7 @@ def save_thumbnail(self, filename, image):
                             storage.path(_upload_path)
                         )
                     except Exception as e:
-                        logger.debug(e)
+                        logger.exception(e)
 
                 try:
                     # Optimize the Thumbnail size and resolution
@@ -1414,9 +1414,9 @@ def save_thumbnail(self, filename, image):
                         (_default_thumb_size['width'], _default_thumb_size['height']),
                         resample=Image.ANTIALIAS)
                     cover = ImageOps.fit(im, (_default_thumb_size['width'], _default_thumb_size['height']))
-                    cover.save(storage.path(_upload_path), format='JPEG')
+                    cover.save(storage.path(_upload_path), format='PNG')
                 except Exception as e:
-                    logger.debug(e)
+                    logger.exception(e)
 
                 # check whether it is an URI or not
                 parsed = urlsplit(url)
@@ -1447,10 +1447,10 @@ def save_thumbnail(self, filename, image):
                     thumbnail_url=url
                 )
         except Exception as e:
-            logger.debug(
-                'Error when generating the thumbnail for resource %s. (%s)' %
-                (self.id, str(e)))
-            logger.warn('Check permissions for file %s.' % upload_path)
+            logger.error(
+                f'Error when generating the thumbnail for resource {self.id}. ({e})'
+            )
+            logger.error(f'Check permissions for file {upload_path}.')
             try:
                 Link.objects.filter(resource=self, name='Thumbnail').delete()
                 _thumbnail_url = staticfiles.static(settings.MISSING_THUMBNAIL)
@@ -1471,9 +1471,9 @@ def save_thumbnail(self, filename, image):
                     thumbnail_url=_thumbnail_url
                 )
             except Exception as e:
-                logger.debug(
-                    'Error when generating the thumbnail for resource %s. (%s)' %
-                    (self.id, str(e)))
+                logger.error(
+                    f'Error when generating the thumbnail for resource {self.id}. ({e})'
+                )
 
     def set_missing_info(self):
         """Set default permissions and point of contacts.