[Fixes #6925] Thesauri improvements (#6951)

* Thesaurus: Add model ThesaurusLabel, improve load_thesaurus managment command and add test coverage * Thesaurus: Add new migration file, stab thesaurus rdf file and managment command test * Thesaurus: Add copyright, file reformatted with black * Thesaurus: Increase test coverage for thesaurus load and class refactor * Resolves #6925: New Thesaurus form generated dynamically based on thesaurus avaiable, fix test on thesaurus load, add migration file for updated thesaurus schema * Resolves #6925: refactor thesaurus form, fix validation issue * Resolves #6925: fix metadata bug that prevent to save multiple key for each thesaurus * Bump urllib3 from 1.26.2 to 1.26.3 (#6908) Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.2 to 1.26.3. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/1.26.3/CHANGES.rst) - [Commits](urllib3/urllib3@1.26.2...1.26.3) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni <[email protected]> * [Fixes #6880] Circle CI upload tests fail irregulary (#6881) * [Fixes #6880] Circle CI upload tests fail irregulary * CircleCI test fix: sometimes expires due to upload timeout in the test environment * - Avoid infinite loop on upload testing * Revert "CircleCI test fix: sometimes expires due to upload timeout in the test environment" This reverts commit 66139fd. Co-authored-by: Alessio Fabiani <[email protected]> Co-authored-by: afabiani <[email protected]> * [Fixes #6914] Remove "add to basket" tool for documents and maps (#6915) * Added malnajdi as contributor * [Fixes #6910] meaningful filename for document download (#6911) * get meaningful document filenames on download * - Strip extension from document title before slugify it (e.g.: image.jpg instead of imagejpg.jpg) Co-authored-by: afabiani <[email protected]> Co-authored-by: Alessio Fabiani <[email protected]> * - CircleCI Upload Tests: trying to reduce more the risk of infinite loop on "wait_for_progress" * [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION (#6913) * [Fixes #6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION * [Fixes #6916] unit test for GEOSERVER_LOCATION * Bump django-cors-headers from 3.6.0 to 3.7.0 (#6901) Bumps [django-cors-headers](https://github.com/adamchainz/django-cors-headers) from 3.6.0 to 3.7.0. - [Release notes](https://github.com/adamchainz/django-cors-headers/releases) - [Changelog](https://github.com/adamchainz/django-cors-headers/blob/master/HISTORY.rst) - [Commits](adamchainz/django-cors-headers@3.6.0...3.7.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump amqp from 5.0.3 to 5.0.5 (#6905) Bumps [amqp](https://github.com/celery/py-amqp) from 5.0.3 to 5.0.5. - [Release notes](https://github.com/celery/py-amqp/releases) - [Changelog](https://github.com/celery/py-amqp/blob/master/Changelog) - [Commits](celery/py-amqp@v5.0.3...v5.0.5) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump pip from 21.0 to 21.0.1 (#6900) Bumps [pip](https://github.com/pypa/pip) from 21.0 to 21.0.1. - [Release notes](https://github.com/pypa/pip/releases) - [Changelog](https://github.com/pypa/pip/blob/master/NEWS.rst) - [Commits](pypa/pip@21.0...21.0.1) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump coverage from 5.3.1 to 5.4 (#6903) Bumps [coverage](https://github.com/nedbat/coveragepy) from 5.3.1 to 5.4. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](nedbat/coveragepy@coverage-5.3.1...coverage-5.4) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump pytest from 6.2.1 to 6.2.2 (#6907) Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.1 to 6.2.2. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst) - [Commits](pytest-dev/pytest@6.2.1...6.2.2) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump djangorestframework-gis from 0.16 to 0.17 (#6902) Bumps [djangorestframework-gis](https://github.com/openwisp/django-rest-framework-gis) from 0.16 to 0.17. - [Release notes](https://github.com/openwisp/django-rest-framework-gis/releases) - [Changelog](https://github.com/openwisp/django-rest-framework-gis/blob/master/CHANGES.rst) - [Commits](openwisp/django-rest-framework-gis@v0.16.0...v0.17.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * - Algin setup.cfg to requirements.txt * [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has… (#6923) * [Fixes #6922][REST API v2] Expose the curated thumbnail URL if it has been uploaded * - Add REST APIs test suite to CircleCI * [Fixes #6918] Removal of QGIS support (#6919) * [Cleanup and Refactor] Remove QGIS server backend dependencies * [Cleanup and Refactor] Remove QGIS server backend dependencies * - Fix LGTM issues * Resolves #6925: add deprecation warning for old thesaurus settings * allow Basic authenticated requests in LOCKDOWN mode * Resolves #6925: change metadata template to let it reads the new thesaurus information as template tags, add tests for new template tags * Resolves #6925: add thesaurus in admin page * fix to avoid circular import * flake8 check fix * Resolves #6925: reformatting code for match Flake8 requirements * added tests * Resolves #6925: fix default values for all required in Thesaurus Model * Resolves #6925: add test for ThesaurusAvailableModelForm * Resolves #6925: minor refactor in ThesaurusForm initialization * Resolves #6925: thesaurus initial data * Resolves #6925: fix flake8 style errors * Refers #6925: add missing migration file * Resolves #6925: fix thesaurus show method by adding facet as filter * Resolves #6925: fix flake8 errors * Resolves #6925: fix lgmt bot issue * Resolves #6925: add missing prefix for thesaurus * Fixes #6925: remote RNDT information from the tests * Fixes #6925: italian wordings updated to english Co-authored-by: Giovanni Allegri <[email protected]> Co-authored-by: allyoucanmap <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni <[email protected]> Co-authored-by: Alessio Fabiani <[email protected]> Co-authored-by: afabiani <[email protected]> Co-authored-by: Florian Hoedt <[email protected]> Co-authored-by: Mohammed Y. Alnajdi <[email protected]> Co-authored-by: biegan <[email protected]>
GeoNode · Feb 17, 2021 · 68d2f25 · 68d2f25
1 parent c867f35
commit 68d2f25
Show file tree

Hide file tree

Showing 21 changed files with 920 additions and 70 deletions.
diff --git a/geonode/base/fixtures/initial_data.json b/geonode/base/fixtures/initial_data.json
@@ -4852,5 +4852,269 @@
       "fields": {
         "name": "TOPBAR_MENU"
       }
-    }
+    },
+    {
+      "model": "base.thesaurus",
+      "pk": 1,
+      "fields": {
+          "identifier": "inspire_themes",
+          "title": "GEMET - INSPIRE themes, version 1.0",
+          "date": "2018-05-23T10:25:56",
+          "description": "GEMET - INSPIRE themes, version 1.0",
+          "slug": "",
+          "about": "http://inspire.ec.europa.eu/theme",
+          "card_min": 1,
+          "card_max": 0,
+          "facet": true
+      }
+    },
+    {
+        "model": "base.thesaurus",
+        "pk": 4,
+        "fields": {
+            "identifier": "abc-all1",
+            "title": "Register of data of general interest",
+            "date": "2018-06-25",
+            "description": "Register of data of general interest",
+            "slug": "",
+            "about": "https://registry.geodati.gov.it/abc-all1",
+            "card_min": 0,
+            "card_max": 1,
+            "facet": true
+        }
+    },
+    {
+        "model": "base.thesaurus",
+        "pk": 6,
+        "fields": {
+            "identifier": "limitation_on_public_access",
+            "title": "Limitations on public access",
+            "date": "2020-10-22T13:21:33",
+            "description": "Limitations on public access",
+            "slug": "",
+            "about": "http://inspire.ec.europa.eu/metadata-codelist/LimitationsOnPublicAccess",
+            "card_min": 1,
+            "card_max": 1,
+            "facet": true
+        }
+    },
+    {
+        "model": "base.thesaurus",
+        "pk": 7,
+        "fields": {
+            "identifier": "limitation_on_public_access-2",
+            "title": "Conditions Applying To Access and Use",
+            "date": "2020-10-30T16:58:34",
+            "description": "Conditions Applying To Access and Use",
+            "slug": "",
+            "about": "http://inspire.ec.europa.eu/metadata-codelist/ConditionsApplyingToAccessAndUse",
+            "card_min": 0,
+            "card_max": -1,
+            "facet": true
+        }
+    },
+    {
+        "model": "base.thesaurus",
+        "pk": 8,
+        "fields": {
+            "identifier": "foo_name",
+            "title": "Mocked Title",
+            "date": "2018-05-23T10:25:56",
+            "description": "Mocked Title",
+            "slug": "abc",
+            "about": "http://inspire.ec.europa.eu/theme",
+            "card_min": 1,
+            "card_max": -1,
+            "facet": true
+        }
+    },
+    {
+      "model": "base.thesauruskeyword",
+      "pk": 1,
+      "fields": {
+          "about": "http://inspire.ec.europa.eu/theme/ad",
+          "alt_label": "ad",
+          "thesaurus": 1
+      }
+    },
+    {
+        "model": "base.thesauruskeyword",
+        "pk": 2,
+        "fields": {
+            "about": "http://inspire.ec.europa.eu/theme/au",
+            "alt_label": "au",
+            "thesaurus": 1
+        }
+    },
+    {
+        "model": "base.thesauruskeyword",
+        "pk": 3,
+        "fields": {
+            "about": "http://inspire.ec.europa.eu/theme/rs",
+            "alt_label": "rs",
+            "thesaurus": 1
+        }
+    },
+    {
+      "model": "base.thesauruskeyword",
+      "pk": 35,
+      "fields": {
+          "about": "http://inspire.ec.europa.eu/metadata-codelist/LimitationsOnPublicAccess/INSPIRE_Directive_Article13_1a",
+          "alt_label": "public access limited according to Article 13(1)(a) of the INSPIRE Directive [ITA]",
+          "thesaurus": 6
+      }
+    },
+    {
+      "model": "base.thesauruskeyword",
+      "pk": 44,
+      "fields": {
+          "about": "http://inspire.ec.europa.eu/metadata-codelist/ConditionsApplyingToAccessAndUse/noConditionsApply",
+          "alt_label": "No condition applicable",
+          "thesaurus": 7
+      }
+    },
+    {
+        "model": "base.thesauruskeyword",
+        "pk": 45,
+        "fields": {
+            "about": "http://inspire.ec.europa.eu/metadata-codelist/ConditionsApplyingToAccessAndUse/conditionsUnknown",
+            "alt_label": "Unknown Conditions",
+            "thesaurus": 7
+        }
+    },
+    {
+      "model": "base.thesauruskeyword",
+      "pk": 100,
+      "fields": {
+          "about": "http://test.com",
+          "alt_label": "Protected sites",
+          "thesaurus": 8
+      }
+    },
+    {
+      "model": "base.thesauruskeywordlabel",
+      "pk": 1,
+      "fields": {
+          "lang": "en",
+          "label": "Addresses",
+          "keyword": 1
+      }
+    },
+    {
+        "model": "base.thesauruskeywordlabel",
+        "pk": 2,
+        "fields": {
+            "lang": "de",
+            "label": "Adressen",
+            "keyword": 1
+        }
+    },
+    {
+        "model": "base.thesauruskeywordlabel",
+        "pk": 3,
+        "fields": {
+            "lang": "en",
+            "label": "Administrative units",
+            "keyword": 2
+        }
+    },
+    {
+        "model": "base.thesauruskeywordlabel",
+        "pk": 4,
+        "fields": {
+            "lang": "en",
+            "label": "Coordinate reference systems",
+            "keyword": 3
+        }
+    },
+    {
+      "model": "base.thesauruskeywordlabel",
+      "pk": 36,
+      "fields": {
+          "lang": "en",
+          "label": "public access limited according to Article 13(1)(a) of the INSPIRE Directive",
+          "keyword": 35
+      }
+    },
+    {
+      "model": "base.thesauruskeywordlabel",
+      "pk": 54,
+      "fields": {
+          "lang": "en",
+          "label": "no conditions to access and use",
+          "keyword": 44
+      }
+    },
+    {
+      "model": "base.thesauruskeywordlabel",
+      "pk": 56,
+      "fields": {
+          "lang": "en",
+          "label": "conditions to access and use unknown",
+          "keyword": 45
+      }
+    },
+    {
+        "model": "base.thesauruskeywordlabel",
+        "pk": 102,
+        "fields": {
+            "lang": "en",
+            "label": "Protected sites",
+            "keyword": 100
+        }
+    },
+    {
+      "model": "base.thesauruslabel",
+      "pk": 1,
+      "fields": {
+          "lang": "en",
+          "label": "Addresses",
+          "thesaurus": 1
+      }
+    },
+    {
+      "model": "base.thesauruslabel",
+      "pk": 2,
+      "fields": {
+          "lang": "de",
+          "label": "Adressen",
+          "thesaurus": 1
+      }
+    },
+    {
+      "model": "base.thesauruslabel",
+      "pk": 4,
+      "fields": {
+          "lang": "en",
+          "label": "Register of the reference data sets",
+          "thesaurus": 4
+      }
+    },
+    {
+      "model": "base.thesauruslabel",
+      "pk": 5,
+      "fields": {
+          "lang": "en",
+          "label": "Limitations on public access",
+          "thesaurus": 6
+      }
+    },
+    {
+      "model": "base.thesauruslabel",
+      "pk": 7,
+      "fields": {
+          "lang": "en",
+          "label": "Conditions Applying To Access and Use",
+          "thesaurus": 7
+      }
+    },
+    {
+      "model": "base.thesauruslabel",
+      "pk": 9,
+      "fields": {
+          "lang": "en",
+          "label": "Register of the reference data sets",
+          "thesaurus": 8
+      }
+  }
 ]
diff --git a/geonode/base/forms.py b/geonode/base/forms.py
@@ -20,43 +20,37 @@
 import re
 import html
 import logging
-
-from tinymce.widgets import TinyMCE
-
-from .fields import MultiThesauriField
-
+from django.db.models.query import QuerySet
+from bootstrap3_datetime.widgets import DateTimePicker
 from dal import autocomplete
-from taggit.forms import TagField
-
 from django import forms
 from django.conf import settings
 from django.contrib.auth import get_user_model
 from django.contrib.auth.models import Group
 from django.core import validators
 from django.db.models import Prefetch, Q
-from django.forms import models
-from django.forms import ModelForm
+from django.forms import ModelForm, models
 from django.forms.fields import ChoiceField
 from django.forms.utils import flatatt
+from django.utils.encoding import force_text
 from django.utils.html import format_html
 from django.utils.safestring import mark_safe
 from django.utils.translation import ugettext as _
-
-from django.utils.encoding import (
-    force_text,
-)
-
-from bootstrap3_datetime.widgets import DateTimePicker
 from modeltranslation.forms import TranslationModelForm
+from taggit.forms import TagField
+from tinymce.widgets import TinyMCE
 
-from geonode.base.models import HierarchicalKeyword, TopicCategory, Region, License, CuratedThumbnail, \
-    ResourceBase
-from geonode.base.models import ThesaurusKeyword, ThesaurusKeywordLabel
-from geonode.documents.models import Document
 from geonode.base.enumerations import ALL_LANGUAGES
+from geonode.base.models import (CuratedThumbnail, HierarchicalKeyword,
+                                 License, Region, ResourceBase, Thesaurus,
+                                 ThesaurusKeyword, ThesaurusKeywordLabel, ThesaurusLabel,
+                                 TopicCategory)
 from geonode.base.widgets import TaggitSelect2Custom
+from geonode.documents.models import Document
 from geonode.layers.models import Layer
 
+from .fields import MultiThesauriField
+
 logger = logging.getLogger(__name__)
 
 
@@ -320,6 +314,56 @@ class Meta:
     )
 
 
+class ThesaurusAvailableForm(forms.Form):
+    def __init__(self, *args, **kwargs):
+        super(ThesaurusAvailableForm, self).__init__(*args, **kwargs)
+        lang = settings.THESAURUS_DEFAULT_LANG if hasattr(settings, "THESAURUS_DEFAULT_LANG") else "en"
+        for item in Thesaurus.objects.all():
+            tname = ThesaurusLabel.objects.values_list("label", flat=True).filter(id=item.id).filter(lang=lang)
+            if item.card_max == 0:
+                continue
+            elif item.card_max == 1 and item.card_min == 0:
+                self.fields[f"{item.id}"] = self._define_choicefield(item, False, tname, lang)
+            elif item.card_max == 1 and item.card_min == 1:
+                self.fields[f"{item.id}"] = self._define_choicefield(item, True, tname, lang)
+            elif item.card_max == -1 and item.card_min == 0:
+                self.fields[f"{item.id}"] = self._define_multifield(item, False, tname, lang)
+            elif item.card_max == -1 and item.card_min == 1:
+                self.fields[f"{item.id}"] = self._define_multifield(item, True, tname, lang)
+
+    def cleanx(self, x):
+        cleaned_values = []
+        for key, value in x.items():
+            if isinstance(value, QuerySet):
+                for y in value:
+                    cleaned_values.append(y.id)
+            elif value:
+                cleaned_values.append(value.id)
+        return ThesaurusKeyword.objects.filter(id__in=cleaned_values)
+
+    @staticmethod
+    def _define_multifield(item, required, tname, lang):
+        return MultiThesauriField(
+            ThesaurusKeyword.objects.prefetch_related(
+                Prefetch(
+                    "keyword",
+                    queryset=ThesaurusKeywordLabel.objects.filter(keyword__thesaurus_id=item.id).filter(lang=lang),
+                )
+            ),
+            widget=autocomplete.ModelSelect2Multiple(url=f"/base/thesaurus_available/?sysid={item.id}&lang={lang}"),
+            label=_(f"{tname[0] if len(tname) > 0 else item.title}"),
+            required=required,
+        )
+
+    @staticmethod
+    def _define_choicefield(item, required, tname, lang):
+        return models.ModelChoiceField(
+            label=f"{tname[0] if len(tname) > 0 else item.title}",
+            required=required,
+            queryset=ThesaurusKeywordLabel.objects.filter(keyword__thesaurus_id=item.id).filter(lang=lang),
+        )
+
+
 class ResourceBaseDateTimePicker(DateTimePicker):
 
     def build_attrs(self, base_attrs=None, extra_attrs=None, **kwargs):