[Issue #4276] Updating django-autocomplete-light to version 3.5.0

[dhlambert]

Upgrading autocomplete_light dependency from 2.3.1 to 3.5.0 this will allow compatibility with django 2 and cross compatablity with python 2/3. Relates to #4276.

This has required significant changes as the DAL project usage has changed fairly significantly between versions 2.3.3 and 3.5.0. See docs below for more info.

New Docs: https://django-autocomplete-light.readthedocs.io/en/master/api.html
Old Docs: https://django-autocomplete-light.readthedocs.io/en/2.3.1/autocomplete.html

Additions/Fixes:
-> New keywords added in admin view do not error
-> Group category autocomplete now shows group categories not groups
-> Group category search works
-> No duplicates showing in autocomplete
-> Upgrading UI to use select2 widget
-> New UI styling for search input autocomplete dropdown - limits to 10 items
-> Manual implementation for Autocomplete on frontend using JQuery for input autocomplete.
-> Thesaurus keywords added to admin and advanced metadata edit views when activated
-> Thesaurus keywords do not get removed when saving data in advanced metadata edit views and admin
-> Search for groups works with queries with spaces

Removals/TODO
-> Keywords tree widget on backend and frontend needs updating to work with select2/autocomplete_light.
-> Did find there were issues with this custom widget when testing it on master. The heirarchy only seemed to open
sometimes when clicking an item, so this functionality would also need to be fixed.
-> Hack to get around advanced metadata styling for select2 elements, this might want to be updated at some point.

Checklist

Reviewing is a process done by project maintainers, mostly on a volunteer basis. We try to keep the overhead as small as possible and appreciate if you help us to do so by completing the following items. Feel free to ask in a comment if you have troubles with any of them.

For all pull requests:

• Confirm you have read the contribution guidelines
• You have sent a Contribution Licence Agreement (CLA) as necessary (not required for small changes, e.g., fixing typos in documentation)
• Make sure the first PR targets the master branch, eventual backports will be managed later. This can be ignored if the PR is fixing an issue that only happens in a specific branch, but not in newer ones.

The following are required only for core and extension modules (they are welcomed, but not required, for contrib modules):

• There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemptions is, changes not visible to end users)
• The issue connected to the PR must have Labels and Milestone assigned
• PR for bug fixes and small new features are presented as a single commit
• Commit message must be in the form "[Fixes #<issue_number>] Title of the Issue"
• New unit tests have been added covering the changes, unless there are explanation on why the tests are not necessary/implemented
• This PR passes all existing unit tests (test results will be reported by travis-ci after opening this PR)
• This PR passes the QA checks: flake8 geonode
• [] Commits changing the settings, UI, existing user workflows, or adding new functionality, need to include documentation updates

Submitting the PR does not require you to check all items, but by the time it gets merged, they should be either satisfied or inapplicable.

[afabiani]

Dear @dhlambert thanks for this awesome work and effort.

IMHO to get this PR accepted we will need at least to:

1. Figure out why selenium tests (not SPC ones) are failing with a 404 error. It looks like some .css is missing on something?

AssertionError: '404' == '404' : Error: "http://localhost/static/geonode/css/ux-all.css?v=3.0.dev1576191474" returned a 404!

2. Be sure there are opened issues marked as regression or feature reporting the things you listed among the Removals/TODO list

[codecov]

Codecov Report

Merging #5410 into master will increase coverage by 0.08%.
The diff coverage is 58.77%.

@@            Coverage Diff             @@
##           master    #5410      +/-   ##
==========================================
+ Coverage   57.57%   57.66%   +0.08%     
==========================================
  Files         172      167       -5     
  Lines       19299    19241      -58     
  Branches     3006     2999       -7     
==========================================
- Hits        11112    11095      -17     
+ Misses       7167     7130      -37     
+ Partials     1020     1016       -4

[dhlambert]

@afabiani Currently looking into fixing up those selenium tests, will also create those issues as you mentioned as well.

[afabiani]

There are still few things to fix.

In general, this seems to be a quite important upgrade, changing a lot of widely used froms and search filters.

I wondering if the existing "tests" and "docs" already cover those changes.

As an instance, are we fully sure all the configurations and settings on the "docs" are still valid?

[afabiani]

As far as I can see you introduced a new static JS file which is not part of the "package.json" nor of the "static_dependencies.json".
That means that trying to build GeoNode assets from scratch this will be completely lost.

Question:

• is this a custom JS or it is part of a common library?

If this is a custom JS this should be placed under "static/geonode/js/..." instead.

[dhlambert]

Yes, this is custom JS. It is used for the text input autocompletes used by all the searches (not the multiselects which is covered by select2). Thinking it can can be moved to the "static/geonode/js/search" directory rather than making a separate '/autocomplete directory.

[afabiani]

Yeps, go ahead

[dhlambert]

@afabiani Have had a look at the "docs" as you mentioned in your post earlier. It seems there are a few mentions of the hierarchical keyword tree widget in the wizard and advanced metadata edit docs. These might need updating if the tree widget is going to be removed temporarily.

• http://docs.geonode.org/en/2.10.x/admin/admin_panel/index.html#hierarchical-keywords
• http://docs.geonode.org/en/2.10.x/usage/managing_documents/document_metadata.html#filling-the-document-metadata
• http://docs.geonode.org/en/2.10.x/usage/managing_layers/layer_metadata.html#metadata-wizard

There also seems to inconsistencies in the screenshots and gifs throughout the docs regarding autocomplete styling for the searches i.e.

Current:

taken from http://docs.geonode.org/en/2.10.x/usage/data/finding_data.html#layers

New:

Happy to adjust the styling to fit the previous style if needed? Otherwise do you think it would be worth updating the screenshots/gifs to account for the new style or not worry about it?

The thesaurus setting seems to be working correctly in settings.py. Planning to check on the readonly keywords setting to see if this is working correctly. Not sure about other configurations which might be worth checking/testing.

[afabiani]

@afabiani Have had a look at the "docs" as you mentioned in your post earlier. It seems there are a few mentions of the hierarchical keyword tree widget in the wizard and advanced metadata edit docs. These might need updating if the tree widget is going to be removed temporarily.

• http://docs.geonode.org/en/2.10.x/admin/admin_panel/index.html#hierarchical-keywords
• http://docs.geonode.org/en/2.10.x/usage/managing_documents/document_metadata.html#filling-the-document-metadata
• http://docs.geonode.org/en/2.10.x/usage/managing_layers/layer_metadata.html#metadata-wizard

There also seems to inconsistencies in the screenshots and gifs throughout the docs regarding autocomplete styling for the searches i.e.

Current:

taken from http://docs.geonode.org/en/2.10.x/usage/data/finding_data.html#layers

New:

Happy to adjust the styling to fit the previous style if needed? Otherwise do you think it would be worth updating the screenshots/gifs to account for the new style or not worry about it?

The thesaurus setting seems to be working correctly in settings.py. Planning to check on the readonly keywords setting to see if this is working correctly. Not sure about other configurations which might be worth checking/testing.

Very good work @dhlambert thanks. I would say:

1. for the docs let's update the screenshots, this is more than enough and let's remove outdated or no more valid parts. We will recreate them in future PRs.

2. if you find something that might be worth testing or investigating, just open an issue. The same if you find some regression or bug or misbehavior introduced by this PR.

Thanks again.

[afabiani]

@dhlambert few feedbacks while testing your PR

1. When trying to edit the layer metadata I get the following error

GET http://localhost:8000/static/admin/css/autocomplete.css net::ERR_ABORTED 404 (Not Found)

2. If I try to insert new keywords containing substrings of already existing ones, they won't be inserted at all. As an instance, if I already have a keyword named "afabiani" in the system, trying to add a new one like "afa" or "afabi" or "iani" won't work.

[afabiani]

@dhlambert if you don't mind I did a commit to fix the two issues I mentioned above.

I'll let you squash them.

[dhlambert]

@afabiani thanks for fixing up those issues!

Will start making some of those changes to the docs as you suggested above.