Custom script for each request (#70)

* issue-38: Custom script for each request

Apply suggestions from code review

Co-authored-by: Eugene Morozov <[email protected]>

* issue-38: improves readme

issue-38: fixes readme

* issue-38: fixes models

issue-38: adds migration

* issue-38: adds celery

issue-38: adds base task

issue-38: adds call task in view

* issue-38: adds count workers in readme

* issue-38: adds restricted python to task

* issue-38: adds celery to docker compose

* issue-38: fixes makefile

* issue-38: improves celery settings

* issue-38: improves time limit message

* issue-38: fixes logs admin

* issue-38: adds lock file

* issue-38: adds request body in task

* issue-38: fixes linting

* Apply suggestions from code review

Co-authored-by: Eugene Morozov <[email protected]>

* issue-38: fixes readme

* issue-38: adds tests

* issue-38: adds the test for the absolute tag fix

* issue-38: fixes linting

* Apply suggestions from code review

Co-authored-by: Eugene Morozov <[email protected]>

* Update Makefile

Co-authored-by: Eugene Morozov <[email protected]>

.isort.cfg

@@ -0,0 +1,4 @@
+[isort]
+multi_line_output = 3
+include_trailing_comma = true
+use_parentheses = true

Makefile

@@ -60,7 +60,8 @@ build:  ## Build docker image
 		build parrot-app
 
 envfile:  ## Generate env file with variables with prefix PARROT_
-	env | egrep ^PARROT_ > .gen.env
+	$(shell env | egrep '^PARROT_' > .gen.env && echo '.gen.env has been generated' || touch .gen.env)
+	$(shell test -f .env && cat .env > .gen.env)
 
 runserver:  envfile  ## Local startup the app on docker with required services
 	docker-compose \

README.md

@@ -30,15 +30,15 @@ Code quality:
 1. Issue tracker: [https://github.com/Uma-Tech/parrot/issues](https://github.com/Uma-Tech/parrot/issues)
 1. Changelog: [https://github.com/Uma-Tech/parrot/blob/develop/CHANGELOG.md](https://github.com/Uma-Tech/parrot/blob/develop/CHANGELOG.md)
 
-## Quickstart
+## Quickstart (with Docker)
 1. Clone the repo
-    ```shell script
+    ```shell
     git clone [email protected]:Uma-Tech/parrot.git
     ```
 
 1. Build or download the docker image  
     _for build_
-    ```shell script
+    ```shell
     make build
     ```
     _for download_
@@ -47,7 +47,7 @@ Code quality:
     ```
 
 1. Apply migrations and create a superuser:
-    ```shell script
+    ```shell
     make shell
     # inside the container
     python manage.py migrate  # apply migrations
@@ -56,27 +56,99 @@ Code quality:
     ```
 
 1. Start app with required services
-    ```shell script
+    ```shell
     make runserver
     ```
 
 1. Service will be available at `http://127.0.0.1:8042/`
 
-## System requires
+## Start the project for development without Docker
+1. Clone the repo and go to the project directory
+    ```shell
+    git clone [email protected]:Uma-Tech/parrot.git
+    cd parrot
+    ```
+
+1. Install dependencies
+    ```shell
+    poetry install
+    ```
+
+1. Start databases  
+    _You can start databases in any other way_
+   ```shell
+    # start postgres
+   docker run \
+       --name postgres \
+       -d \
+       --rm \
+       -e POSTGRES_PASSWORD=parrot \
+       -e POSTGRES_USER=parrot \
+       --network host \
+       -v parrot_db:/var/lib/postgresql/data \
+       postgres
+   
+    # start redis
+   docker run \
+       --name redis \
+       -d \
+       --rm \
+       --network host \
+       redis
+    ```
+
+1. Setting required environment variables  
+   _Alternatively, you can create a local `.env` file with the variables_
+    ```shell
+    export PARROT_DB_HOST=127.0.0.1
+    export PARROT_SECRET_KEY=NO_SECRET
+    export PARROT_CELERY_BROKER_URL=redis://127.0.0.1
+    ```
+
+1. Build static
+    ```shell
+    poetry run python manage.py collectstatic --no-input
+    ```
+
+1. Apply database migrations
+    ```shell
+    poetry run python manage.py migrate
+    ```
+
+1. Create a django superuser
+    ```shell
+    poetry run python manage.py createsuperuser
+    ```
+
+1. Start the django app
+    ```shell
+    poetry run python manage.py runserver
+    ```
+
+1. Start the celery worker  
+   _Run in a separate terminal window_
+    ```shell
+    poetry run celery -A parrot worker -l INFO -c 4
+    ```
+
+
+## System requirements
 * docker ([https://www.docker.com/](https://www.docker.com/))
 * docker-compose
     ([https://github.com/docker/compose](https://github.com/docker/compose))
 * make
     ([https://www.gnu.org/software/make/](https://www.gnu.org/software/make/))
     _(all commands can be viewed by calling `make` without parameters)_
+* poetry _(for development)_
+  ([https://python-poetry.org/](https://python-poetry.org/))
 
 ## Components
 1. Database postgres.
+1. Database redis (for background celery-tasks).
 1. Python-app based on Django Web Framework.
 
 ## Contributing
 We welcome all contributions!  
 See [CONTRIBUTING.md](CONTRIBUTING.md) if you want to contribute.  
 You can start with [issues that need some help](https://github.com/Uma-Tech/parrot/issues)
 right now.
-

conftest.py

@@ -2,6 +2,10 @@
 
 from _pytest.fixtures import FixtureFunctionMarker
 
+from parrot import celery_app
+
+celery_app.conf['CELERY_ALWAYS_EAGER'] = True
+
 
 @pytest.fixture(autouse=True)
 def db_connect(db: FixtureFunctionMarker):

deploy/docker-compose.db.yml

@@ -17,6 +17,18 @@ services:
       - "5432"
     volumes:
       - "${PARROT_DB_VOLUME:-db-volume}:/var/lib/postgresql/data"
+    networks:
+      app:
+      celery:
+
+  parrot-celery-broker:
+    image: redis:6
+    restart: unless-stopped
+    container_name: parrot-celery-broker
+    networks:
+      app:
+      celery:
+
 
 volumes:
   db-volume:

deploy/docker-compose.develop.yml

@@ -4,6 +4,7 @@ services:
 
   parrot-app:
     command: bash /app/start-django-dev.sh
+    privileged: true
     environment:
       - DEBUG=1
     volumes:

deploy/docker-compose.yml

@@ -1,19 +1,32 @@
 version: "3.8"
 
 x-service-base: &service-base
-  image: umahighload/parrot-app:latest
+  image: umahighload/parrot-app:${PARROT_APP_VERSION:-latest}
   restart: unless-stopped
+  env_file:
+    - .gen.env
+  build:
+    context: .
+    dockerfile: ./deploy/app/Dockerfile
 
 services:
 
   parrot-app:
     <<: *service-base
-    build:
-      context: .
-      dockerfile: ./deploy/app/Dockerfile
-    env_file:
-      - .gen.env
-    ports:
-      - "8042:8042"
     container_name: parrot-app
     command: bash /app/start-django-uvicorn.sh
+    ports:
+      - "8042:8042"
+    networks:
+      app:
+
+  parrot-celery-worker:
+    <<: *service-base
+    container_name: parrot-celery-worker
+    command: celery -A parrot worker -l INFO -c 8
+    networks:
+      celery:
+
+networks:
+    app:
+    celery:

http_stubs/admin.py

@@ -41,6 +41,7 @@ def pretty_body(self, instance) -> str:
 
     list_filter = ('date', 'method')
     search_fields = ('path', 'source_ip')
+    list_display = ('pk', 'date', 'http_stub', 'source_ip')
     readonly_fields = (
         'pk',
         'date',
@@ -51,6 +52,7 @@ def pretty_body(self, instance) -> str:
         'headers',
         'body',
         'pretty_body',
+        'result_script',
     )
 
 

http_stubs/migrations/0005_httpstub_request_script_logentry_result_script.py

@@ -0,0 +1,23 @@
+# Generated by Django 3.1.3 on 2020-12-20 00:03
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('http_stubs', '0004_increase_logentry_path_max_length'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='httpstub',
+            name='request_script',
+            field=models.TextField(blank=True, help_text='Language: python 3.8. The script will run on each request.', verbose_name='Request script'),
+        ),
+        migrations.AddField(
+            model_name='logentry',
+            name='result_script',
+            field=models.CharField(blank=True, max_length=200, verbose_name='Result script'),
+        ),
+    ]

http_stubs/models.py

@@ -85,6 +85,11 @@ class HTTPStub(models.Model):
         default=dict,
         blank=True,
     )
+    request_script = models.TextField(
+        verbose_name='Request script',
+        help_text='Language: python 3.8. The script will run on each request.',
+        blank=True,
+    )
 
     class Meta:
         verbose_name = 'http stub'
@@ -136,6 +141,11 @@ class LogEntry(models.Model):
         blank=True,
         on_delete=models.CASCADE,
     )
+    result_script = models.CharField(
+        verbose_name='Result script',
+        max_length=200,
+        blank=True,
+    )
 
     class Meta:
         verbose_name = 'log'

http_stubs/tasks.py

@@ -0,0 +1,46 @@
+import json
+
+import requests
+from billiard.exceptions import SoftTimeLimitExceeded
+from RestrictedPython import (
+    compile_restricted,
+    limited_builtins,
+    safe_builtins,
+    utility_builtins,
+)
+from RestrictedPython.Guards import full_write_guard
+
+from http_stubs.models import LogEntry
+from parrot import celery_app
+
+restricted_builtins = {'__builtins__': {
+    'requests': requests,
+    'json': json,
+    '_getitem_': lambda dict_obj, key: dict_obj[key],
+    '_write_': full_write_guard,
+    **safe_builtins,
+    **limited_builtins,
+    **utility_builtins,
+}}
+
+
+@celery_app.task()
+def run_request_script(log_id: int, script: str, request_body: str) -> None:
+    """Task for run custom scripts from http stubs.
+
+    :param log_id: LogEntry.id
+    :param script: HTTPStub.request_script
+    :param request_body: text body from a request
+    """
+    log: LogEntry = LogEntry.objects.get(pk=log_id)
+    loc = {'request_body': request_body, **restricted_builtins}
+    byte_code = compile_restricted(script)
+    try:
+        exec(byte_code, loc, None)  # noqa: S102, WPS421
+    except SoftTimeLimitExceeded:
+        log.result_script = 'Error: Execution time limit exceeded'
+    except Exception as err:
+        log.result_script = f'Error: {err}'
+    else:
+        log.result_script = 'Done'
+    log.save()

http_stubs/templatetags/stub_tags.py

@@ -1,7 +1,7 @@
 import json
 from html import unescape
 from json import JSONDecodeError
-from typing import AnyStr, Dict, List
+from typing import AnyStr, Dict, List, Optional
 from urllib.parse import urlunparse
 
 from django import template
@@ -14,7 +14,7 @@
 
 
 @register.simple_tag(takes_context=True, name='absolute')
-def absolute_url(context: Dict, url: Url, fieldset: Fieldset) -> Url:
+def absolute_url(context: Dict, url: Url, fieldset: Fieldset) -> Optional[Url]:
     """Tag that returns an absolute url.
 
     :param context: context of request
@@ -23,6 +23,8 @@ def absolute_url(context: Dict, url: Url, fieldset: Fieldset) -> Url:
                      field from the form
     :returns: absolute url
     """
+    if not url or not fieldset:
+        return ''
     request = context['request']
     form = fieldset.form
     if form.initial.get('regex_path'):