[django1.11] fix file uploading, new user registration + make tests p…

…ass (#940)

* Add chef method unit tests and ensure that Django views return a HTTP response when they raise errors.

* Add more file upload and add node tests.

* Add finish_commit tests and return a HTTP 500 code instead of raising if an error occurs.

* Get Sentry reports for handled internal server errors in Internal API calls.

* Add yarn run devsetup command (#925)

* add new yarn command to set up dev environment

* change commands to remove redundant pieces to prioritize yarn run devsetup

Streamlines the new dev experience

* Updated translations with epub string

* Fixed default preview styling

* Add caching to the get_user_public_channels endpoint (#939)

* Move view get_public_channels logic to Channel model

* add the Channel.make_public function

* use StudioTestCase to create buckets

* add ChannelCacher class to implement channel caching

* add channel token related convenience functions

* make exporchannel use Channel.make_token()

* add channel specific cache for tokens

Cache that, one less query to make.

* add caching to channel token serializer endpoint

* move serializers.get_resource_count implementation to the Channel model

* use channel.get_thumbnail() function on serializers.py

* remove redefinition of generate_thumbnail_url

* add ChannelCacher.get_resource_count() cache function

Used to cache the channel's get_resource_count

* add caching to get_resource_count API attribute

* add channel.get_date_modified function

* make channelfieldmixin use channel.get_date_modified

* add channel.get_date_modified cache

* Use the channel cache for get_date_modified

* remove redundant generate_thumbnail_url

Already defined in ChannelFieldMixin

* pass cache get_public_channels args to real function

* cache the entire get_user_public_channels view

* fix tests for ChannelCacher.get_public_channels()

by comparing actual channel ids rather than objects

* Create SecretToken.exists() convenience method

* refactor make_token to definitely end after 100 attempts

And use for-else loop construct

* add extra non-public assert on the make_public test

* add test for SecretToken.exists()

* [WIP] Add nginx-level API endpoint caching (#943)

* add long running caching to the get_user_public_channels endpoint

* add caching to the get_user_edit_channels endpoint

* add 4 hour browser caching on all static files (#945)

* add caching to the get_user_edit_channels endpoint (#947)

* Unlock le-utils' version

* have pipenv update all updateable packages

* Added comment

* Made resource count required on storage requests

* Merge in develop

* Return 404s for any files not found in the zip, and have BadZipfile errors report the name, size, mode and download state to help us narrow down problems with opening. Also, tests. (#942)

* Remove the hardcoded server IP in tests so that configuration changes don't break them.

* Apply Micah's changes and also fix some new issues uncovered by tests to get the code to the same state as it was before revert.

* Fix login template name.

* Actually commit the updated dependency versions to requirements.txt...

* use `==` instead of `=` in template `if` statement

* Re-delete requirements.txt as a result of rebasing issues.

* Update Django to 1.11 in pipfile after rebase.

* Actually update the dependencies after accidentally using pip...

* objects.create automatically calls save, so no need to call it again afterwards.

* objects.create automatically calls save, so no need to call it again afterwards.

* Revert removed calls to save as we are creating models directly rather than using objects.create.

* Revert removed calls to save as we are creating models directly rather than using objects.create.

* update django to 1.11.15!

* use the recommended way of declaring our AppConfig

Through our INSTALLED_APPS! See https://docs.djangoproject.com/en/1.11/ref/applications/#configuring-applications

* use FormatPresetSerializer as a PrimaryKeyRelatedField serializer

* disable breakpoints

* attempted fix for file uploading issues

* file uploading works... yay!

* signup form multiselection area is working now

* enable offline_helper

* remove extra debugging leftovers from registration info template

* fixes for several tests... remember: don't manually hardcode constants from le_utils.  they may change some day!

* fixed authentication tests

* add a migration from django 1.11

Pipfile

@@ -18,7 +18,7 @@ djangorestframework-bulk = "==0.2.1"
 django-email-extras = "==0.3.3"
 kolibri = "==0.7.0"
 validators = "*"
-le-utils = ">=0.1.11"
+le-utils = "*"
 gunicorn = "==19.6.0"
 django-mailgun = "==0.9.1"
 pressurecooker = "==0.0.18"

README.md

@@ -222,24 +222,18 @@ All the javascript dependencies are listed in `package.json`. To install them ru
 
          CREATE DATABASE "gonano" WITH TEMPLATE = template0 OWNER = "learningequality";
 
-  5. Make sure the Redis server is running (used for job queue)
-
-         service redis-server start
-         # mac: redis-server /usr/local/etc/redis.conf
-
-  6. Start the minio server
-
-        MINIO_ACCESS_KEY=development MINIO_SECRET_KEY=development minio server ~/.minio_data
-
 
 
 ##### Run all database migrations and load constants
 
 You'll only need to run these commands once, to setup the necessary tables and
 constants in the database:
 
-    make migrate collectstatic
-    cd contentcuration; python manage.py setup --settings=contentcuration.dev_settings; cd ..
+    # On one terminal, run all external services 
+    $ yarn run services
+
+    # On another terminal, run devsetup to create all the necessary tables and buckets
+    $ yarn run devsetup
 
 ##### Start the dev server
 

contentcuration/contentcuration/context_processors.py

@@ -0,0 +1,5 @@
+from django.conf import settings
+
+
+def site_variables(request):
+    return {'INCIDENT': settings.INCIDENT}

contentcuration/contentcuration/decorators.py

@@ -6,6 +6,7 @@
 from django.template import RequestContext
 from contentcuration.models import Channel
 from contentcuration.utils.policies import check_policies
+from django.shortcuts import render
 
 ACCEPTED_BROWSERS = settings.HEALTH_CHECK_BROWSERS + settings.SUPPORTED_BROWSERS
 
@@ -30,7 +31,7 @@ def wrap(request, *args, **kwargs):
         if request.user.is_admin:
             return function(request, *args, **kwargs)
 
-        return render_to_response('unauthorized.html', context_instance=RequestContext(request), status=403)
+        return render(request, 'unauthorized.html', status=403)
 
     wrap.__doc__ = function.__doc__
     wrap.__name__ = function.__name__
@@ -49,7 +50,7 @@ def wrap(request, *args, **kwargs):
             request.user.is_admin:
             return function(request, *args, **kwargs)
 
-        return render_to_response('unauthorized.html', context_instance=RequestContext(request), status=403)
+        return render(request, 'unauthorized.html', status=403)
 
     wrap.__doc__ = function.__doc__
     wrap.__name__ = function.__name__
@@ -61,7 +62,7 @@ def wrap(request, *args, **kwargs):
             channel = Channel.objects.get(pk=kwargs['channel_id'], deleted=False)
 
             if not channel.editors.filter(id=request.user.id).exists() and not request.user.is_admin:
-                return render_to_response('unauthorized.html', context_instance=RequestContext(request), status=403)
+                return render(request, 'unauthorized.html', status=403)
 
             return function(request, *args, **kwargs)
         except ObjectDoesNotExist:

contentcuration/contentcuration/forms.py

@@ -89,7 +89,7 @@ def clean(self):
 
 
 class RegistrationInformationForm(UserCreationForm, ExtraFormMixin):
-    use = forms.ChoiceField(required=False, widget=forms.CheckboxSelectMultiple, label=_('How do you plan to use Kolibri Studio? (check all that apply)'), choices=USAGES)
+    use = forms.MultipleChoiceField(required=False, widget=forms.CheckboxSelectMultiple, label=_('How do you plan to use Kolibri Studio? (check all that apply)'), choices=USAGES)
     other_use = forms.CharField(required=False, widget=forms.TextInput)
     storage = forms.CharField(required=False, widget=forms.TextInput(attrs={"placeholder": _("e.g. 500MB")}), label=_("How much storage do you need?"))
 
@@ -112,7 +112,6 @@ def __init__(self, *args, **kwargs):
         )
 
         countries = [(c.name, translator.gettext(c.name)) for c in list(pycountry.countries)]
-
         self.fields['location'] = forms.ChoiceField(required=True, widget=forms.SelectMultiple, label=_('Where do you plan to use Kolibri? (select all that apply)'), choices=countries)
 
     def clean_email(self):
@@ -296,7 +295,7 @@ class StorageRequestForm(forms.Form, ExtraFormMixin):
     # Nature of content
     storage = forms.CharField(required=True, widget=forms.TextInput(attrs={"placeholder": _("e.g. 1GB"), "class": "short-field"}))
     kind = forms.CharField(required=True, widget=forms.TextInput(attrs={"placeholder": _("Mostly high resolution videos, some pdfs, etc."), "class": "long-field"}))
-    resource_count = forms.CharField(required=False, widget=forms.TextInput(attrs={"class": "short-field"}))
+    resource_count = forms.CharField(required=True, widget=forms.TextInput(attrs={"class": "short-field"}))
     resource_size = forms.CharField(required=False, widget=forms.TextInput(attrs={"placeholder": _("e.g. 10MB"), "class": "short-field"}))
     creators = forms.CharField(required=True, widget=forms.TextInput(attrs={"class": "long-field"}))
     sample_link = forms.CharField(required=False, widget=forms.TextInput(attrs={"class": "long-field"}))
@@ -354,6 +353,7 @@ def clean(self):
         self.check_field('storage', _("Please indicate how much storage you need"))
         self.check_field('kind', _("Please indicate what kind of content you are uploading"))
         self.check_field('creators', _("Please indicate the author, curator, and/or aggregator of your content"))
+        self.check_field('resource_count', _("Please indicate approximately how many resources you are planning to upload"))
 
         self.cleaned_data["license"] = ", ".join(self.cleaned_data.get('license') or [])
         self.check_field('license', _("Please indicate the licensing for your content"))

contentcuration/contentcuration/management/commands/exportchannel.py

@@ -582,19 +582,7 @@ def save_export_database(channel_id):
 def add_tokens_to_channel(channel):
     if not channel.secret_tokens.filter(is_primary=True).exists():
         logging.info("Generating tokens for the channel.")
-        token = proquint.generate()
-
-        # Try to generate the channel token, avoiding any infinite loops if possible
-        max_retries = 1000000
-        index = 0
-        while ccmodels.SecretToken.objects.filter(token=token).exists():
-            token = proquint.generate()
-            if index > max_retries:
-                raise ValueError("Cannot generate new token")
-
-        tk_human = ccmodels.SecretToken.objects.create(token=token, is_primary=True)
-        tk, _new = ccmodels.SecretToken.objects.get_or_create(token=channel.id)
-        channel.secret_tokens.add(tk_human, tk)
+        channel.make_token()
 
 def fill_published_fields(channel):
     published_nodes = channel.main_tree.get_descendants().filter(published=True).prefetch_related('files')

contentcuration/contentcuration/migrations/0093_auto_20180831_0745.py

@@ -0,0 +1,21 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.15 on 2018-08-31 07:45
+from __future__ import unicode_literals
+
+import contentcuration.models
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('contentcuration', '0092_auto_20180731_1024'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='file',
+            name='file_on_disk',
+            field=models.FileField(blank=True, max_length=500, upload_to=contentcuration.models.object_storage_name),
+        ),
+    ]

contentcuration/contentcuration/models.py

@@ -25,6 +25,7 @@
 from django.utils import timezone
 from django.utils.translation import ugettext as _
 from django.contrib.postgres.fields import JSONField
+from le_utils import proquint
 from le_utils.constants import (content_kinds, exercises, file_formats, licenses,
                                 format_presets, languages, roles)
 from mptt.models import (MPTTModel, TreeForeignKey, TreeManager,
@@ -231,7 +232,7 @@ def save(self, *args, **kwargs):
             changed = True
 
         if not self.clipboard_tree:
-            self.clipboard_tree = ContentNode.objects.create(title=self.email + " clipboard", kind_id="topic",
+            self.clipboard_tree = ContentNode.objects.create(title=self.email + " clipboard", kind_id=content_kinds.TOPIC,
                                                              sort_order=get_next_sort_order())
             self.clipboard_tree.save()
             changed = True
@@ -396,6 +397,14 @@ class SecretToken(models.Model):
     token = models.CharField(max_length=100, unique=True)
     is_primary = models.BooleanField(default=False)
 
+    @classmethod
+    def exists(cls, token):
+        """
+        Return true when the token string given by string already exists.
+        Returns false otherwise.
+        """
+        return cls.objects.filter(token=token).exists()
+
     def __str__(self):
         return self.token
 
@@ -551,6 +560,86 @@ def get_thumbnail(self):
 
         return '/static/img/kolibri_placeholder.png'
 
+    def get_date_modified(self):
+        return self.main_tree.get_descendants(include_self=True).aggregate(last_modified=Max('modified'))['last_modified']
+
+    def get_resource_count(self):
+        return self.main_tree.get_descendants().exclude(kind_id=content_kinds.TOPIC).count()
+
+    def get_human_token(self):
+        return self.secret_tokens.get(is_primary=True)
+
+    def get_channel_id_token(self):
+        return self.secret_tokens.get(token=self.id)
+
+
+    def make_token(self):
+        """
+        Creates a primary secret token for the current channel using a proquint
+        string. Creates a secondary token containing the channel id.
+
+        These tokens can be used to refer to the channel to download its content
+        database.
+        """
+        token = proquint.generate()
+
+        # Try 100 times to generate a unique token.
+        TRIALS = 100
+        for _ in range(TRIALS):
+            token = proquint.generate()
+            if SecretToken.exists(token):
+                continue
+            else:
+                break
+        # after TRIALS attempts and we didn't get a unique token,
+        # just raise an error.
+        # See https://stackoverflow.com/a/9980160 on what for-else loop does.
+        else:
+            raise ValueError("Cannot generate new token")
+
+        # We found a unique token! Save it
+        human_token = self.secret_tokens.create(token=token, is_primary=True)
+        self.secret_tokens.get_or_create(token=self.id)
+
+        return human_token
+
+    def make_public(self, bypass_signals=False):
+        """
+        Sets the current channel object to be public and viewable by anyone.
+
+        If bypass_signals is True, update the model in such a way that we
+        prevent any model signals from running due to the update.
+
+        Returns the same channel object.
+        """
+        if bypass_signals:
+            self.public = True     # set this attribute still, so the object will be updated
+            Channel.objects.filter(id=self.id).update(public=True)
+        else:
+            self.public = True
+            self.save()
+
+        return self
+
+    @classmethod
+    def get_public_channels(cls, defer_nonmain_trees=False):
+        """
+        Get all public channels.
+
+        If defer_nonmain_trees is True, defer the loading of all
+        trees except for the main_tree."""
+        if defer_nonmain_trees:
+            c = (Channel.objects
+                .filter(public=True)
+                .exclude(deleted=True)
+                .select_related('main_tree')
+                .prefetch_related('editors')
+                .defer('trash_tree', 'clipboard_tree', 'staging_tree', 'chef_tree', 'previous_tree', 'viewers'))
+        else:
+            c = Channel.objects.filter(public=True).exclude(deleted=True)
+
+        return c
+
     class Meta:
         verbose_name = _("Channel")
         verbose_name_plural = _("Channels")