GNIP-64: Python 3 & Django 2 Upgrade

[Coop56]

NGA is proposing to accelerate the work of upgrading GeoNode’s codebase from Python 2.7 to 3.7, Django 1.11 to 2.1, and replace all Angular 1.4 with React and JavaScript code. We will coordinate this work as part of our GEOINT Services contract to make sure this work is completed and pushed back to the GeoNode community by the end of March 2020. Work related to GeoNode as part of the GEOINT Services program will be pushed back to the GeoNode community as part of our open source commitment when appropriate. All work will be performed by Boundless Spatial at the direction of NGA. The following document outlines the work required to deliver on this upgrade work.

We’ve already begun work on the Angular 1.4 to React migration in another GNIP (#4129). This GNIP will focus on the Python 2 to 3 and Django 1.11 to 2.1 upgrades. We propose to complete the Python and Django upgrade process in the following order. Of note, this process is subject to change as we work on these items and uncover dependencies during the upgrade process.

1. Remove unnecessary dependencies, to ensure we only spend time upgrading the dependencies that are necessary to GeoNode.
2. Improve test coverage where possible, to make it easier to spot regressions.
3. Update dependencies with a version that supports Python 3.7 or replace those dependencies with another option if they do not have a version with Python 3.7 support.
4. There are certain pieces of Python 3 syntax that are already compatible with Python 2.7. We’ll make as many of these changes as possible in separate commits and get those PR’d into master before moving onto dropping Python 2.7 support entirely.
5. In one large PR, drop support for Python 2.7 for the entire GeoNode codebase and ensure all code is Python 3.7 compatible. (this PR will have to go on a new version, as it will of course break old systems, the community can decide what version it will be).
6. In a separate PR, update django-osgeo-importer to drop support for Python 2.7 and ensure all code is Python 3.7 compatible.
7. Since Django 1.11 supports Python 3 already, we’ll be able to finish the Python 3 upgrade and then move onto upgrading Django separately. We’ll upgrade any dependencies of GeoNode to a version that supports Django 2 first or replace them with another alternative if necessary.
8. Then we’ll move onto the GeoNode codebase itself and upgrade all of the code to Django 2 compatible syntax.
9. Finally, we’ll upgrade django-osgeo-importer to Django 2 compatible syntax.

Once all of this is done on master, the community will be able to decide which version we call this, and projects can start upgrading their own codebases if they choose to do so.

Key links:

• GeoNode dependency list: https://docs.google.com/spreadsheets/d/1buUuIeGPL8zfv5xFotfGMpxHt2i8njoCEh4Y4kna3rY/edit#gid=0
• Mailing list discussion thread: http://osgeo-org.1560.x6.nabble.com/GNIP-64-Status-and-Plan-td5420981.html
• Python 3 interpreter configuration (PyCharm): https://www.jetbrains.com/help/pycharm/configuring-python-interpreter.html
• Final testing document (need to update a sheet for this GNIP when we are near release): https://docs.google.com/spreadsheets/d/13aaZ1l_2_SHzdmz9DwExEO_jxwgNuXfMa6oF5eWIv_s/edit#gid=376642658
• Task sheet: https://docs.google.com/spreadsheets/d/1gr4ISkFBarY-7i_DPi0Vt_FnA5_uF30dGUgYpcIZ-xw/edit?usp=sharing

[travislbrundage]

Looks good. +1 👍

[copaskevin]

This work has been put on pause by NGA at the moment. We are working with them to identify who/when the work will be completed. Hope to have more information by the end of April.

[frafra]

@copaskevin Any news?

[copaskevin]

@frafra No news from NGA. The rumor is that they will be funding this work outside a different contract but it will more than likely not be coming from the contract awarded to us.

[srtonz]

Any update on this proposal? We may be able to contribute some dedicated time late in the year / early next year to help with upgrade efforts.

[afabiani]

Hi @geotob no particular updates up to now. GeoSolutions will probably invest some resources on this hopefully before the next year. It would be good to cooperate on this of course.

[gannebamm]

If our proposal gets funding, we will also be able to contribute. We will get more info later this year. Let´s do it together.

[coulonpascal]

Hi. SCISYS UK is sponsoring the code sprint at the up coming UK FOSS4G conference. We are using geonode in a number of uk government and private sector projects. It would be great to get some progress on this issue. Has there been any progress anywhere on this?

[gannebamm]

Regarding the mentioned proposal: If it is succesful the funding will not start before Oktober 2020. I am sorry.

[coulonpascal]

It would be great to get some help from the community at the up-coming UK FOSS4G, this is a great forum for this?

[travislbrundage]

Hey all, while I won't be at the up coming UK FOSS4G, I am going to be getting up to speed on this task and picking it up soon. The information is largely the same as described originally to my understanding, but for now the task is on my radar instead. Will keep you updated as I start making progress.

[srtonz]

Nice, we have time set aside from late November onwards if help is still needed by then :-)

[srtonz]

Just wondering if any one managed to make progress on this? If not I would like to propose that our team picks this up from 18 November. We would develop against a public fork of the latest stable release while keeping close to the plan outlined in the issue above.

Would obviously welcome any community contributions to our efforts.

[t-book]

@geotob What great news! We had this yesterday in gitter chat.
I hope it's okay to quote @travislbrundage here:

Well, I've been wrestling with that + trying to speed up the tests for GeoNode (there's some conversation before about it). So far I have only really done organization. I have a spreadsheet continued from the work coop had done with all the libraries that will need replacement or updating with Python3 (fortunately not too many). With this problem out of the way I'll start committing some updates to the unit tests to increase coverage/speed up if possible.

So it would be great if you two could get in contact and coordinate !

[afabiani]

@ all ... I guess GeoSolutions also will jump soon into this. Probably from November.

The priority now is to close the 2.10.1 release. Then we will focus on Python3 upgrade.

[srtonz]

We have two team members free after FOSS4G Oceania through to February - sounds like this won't be a problem to get done with everyone's help:) Let's coordinate in November?

[t-book]

** Dreamteam ** 👍

[travislbrundage]

This is excellent to hear! I welcome all the help I can getting this done. :)

There is not too much to update since I last posted as it took a while to get transitioned onto this task. I will reach out on the developer email list to create a more detailed post on progress and what I foresee going forward.

For now I am just trying to help with the 2.10.1 release + improve unit testing so the Python upgrade (hopefully) goes more smoothly. Looking forward to working with you all!

[travislbrundage]

Hey all, making a quick update to this. The main post will be edited with key information and links. I've made some documents to keep track of work that needs to be done so people can work in parallel. You can find more detailed information in the thread I've made on the mailing list, also linked. Thanks for all the help and feedback :)