From d635b46a8845e15d325a261ecc6bd82df6a94bf2 Mon Sep 17 00:00:00 2001 From: pld Date: Fri, 27 Jan 2017 13:11:47 +0300 Subject: [PATCH 1/4] add install and auth docs --- README.rst | 2 +- docs/api.md | 16 --- docs/authentication.rst | 220 ++++++++++++++++++++++++++++++++++ docs/conf.py | 6 +- docs/index.rst | 226 ++--------------------------------- docs/install.rst | 253 ++++++++++++++++++++++++++++++++++++++++ docs/quick_start.rst | 2 - install.md | 101 ---------------- 8 files changed, 489 insertions(+), 337 deletions(-) delete mode 100644 docs/api.md create mode 100644 docs/authentication.rst create mode 100644 docs/install.rst delete mode 100644 docs/quick_start.rst delete mode 100644 install.md diff --git a/README.rst b/README.rst index 62a149451e..2b21091a5d 100644 --- a/README.rst +++ b/README.rst @@ -14,7 +14,7 @@ Ona is derived from the excellent `formhub `_. +See the `installation documentation `_. Contributing ------------ diff --git a/docs/api.md b/docs/api.md deleted file mode 100644 index ce5e4a1d73..0000000000 --- a/docs/api.md +++ /dev/null @@ -1,16 +0,0 @@ -API changes -=========== - -## /users, /profiles - -- list of users/profiles is public - -## /forms - -- username and form id_string are no longer part of the url format -- use only a form id (number) to access a form -- form publishing, to publish a form to a different account you have to pass the `owner` param, which is the username of that account. - -## /data, /stats, /stats/submission -- username and form id_string are no longer part of the url format -- use only a form id (number) and optionally dataid (number) to access a data node diff --git a/docs/authentication.rst b/docs/authentication.rst new file mode 100644 index 0000000000..237d40b436 --- /dev/null +++ b/docs/authentication.rst @@ -0,0 +1,220 @@ +Authentication and Status Codes +******************************* + +Status Codes +------------ + +- **200** - Successful [``GET``, ``PATCH``, ``PUT``] +- **201** - Resource successfully created [``POST``\ ] +- **204** - Resouce successfully deleted [``DELETE``\ ] +- **403** - Permission denied to resource +- **404** - Resource was not found + +Request based Authentication +---------------------------- + +Ona JSON API enpoints support both Basic authentication and API Token +Authentication through the ``Authorization`` header. + +Basic Authentication +~~~~~~~~~~~~~~~~~~~~ + +Example using curl: + +:: + + curl -X GET https://api.ona.io/api/v1/ -u username:password + +Token Authentication +~~~~~~~~~~~~~~~~~~~~ + +Example using curl: + +:: + + curl -X GET https://api.ona.io/api/v1/ -H "Authorization: Token TOKEN_KEY" + +Temporary Token Authentication +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Example using curl: + +:: + + curl -X GET https://api.ona.io/api/v1/ -H "Authorization: TempToken TOKEN_KEY" + +The temporary token expires after ``DEFAULT_TEMP_TOKEN_EXPIRY_TIME`` seconds, +which defaults to 21600 seconds (6 hours). To expire the temporary token manually +use the `/user/expire` endpoint. Example using curl and password authentication: + +:: + + curl -X DELETE http://api.ona.io/api/v1/user/expire -u username:password + +You could use another type of authentication as well. + +To activate authentication via temporary token you must add the TemporaryToken +class to your local_settings.py file, for example: + +:: + REST_FRAMEWORK['DEFAULT_AUTHENTICATION_CLASSES'] = ( + 'onadata.libs.authentication.DigestAuthentication', + 'onadata.libs.authentication.TempTokenAuthentication', + ... + +Using Oauth2 with the Ona API +----------------------------- + +You can learn more about oauth2 `here`_. + +1. Register your client application with Ona - `register`_ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- ``name`` - name of your application +- ``client_type`` - Client Type: select confidential +- ``authorization_grant_type`` - Authorization grant type: Authorization code +- ``redirect_uri`` - Redirect urls: redirection endpoint + +Keep note of the ``client_id`` and the ``client_secret``, it is required +when requesting for an ``access_token``. + +.. _here: http://tools.ietf.org/html/rfc6749 +.. _register: /o/applications/register/ + +2. Authorize client application. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The authorization url is of the form: + +.. raw:: html + + 
+   GET /o/authorize?client_id=XXXXXX&response_type=code&state=abc
+ +example: + +:: + + http://api.ona.io/o/authorize?client_id=e8&response_type=code&state=xyz + +.. note:: + + Providing the url to any user will prompt for a password and + request for read and write permission for the application whose + ``client_id`` is specified. + +Where: + +- ``client_id`` - is the client application id - ensure its urlencoded +- ``response_type`` - should be code +- ``state`` - a random state string that you client application will + get when redirection happens + +What happens: + +1. a login page is presented, the username used to login determines the + account that provides access. +2. redirection to the client application occurs, the url is of the form: + + REDIRECT\_URI/?state=abc&code=YYYYYYYYY + +example redirect uri + +:: + + http://localhost:30000/?state=xyz&code=SWWk2PN6NdCwfpqiDiPRcLmvkw2uWd + +- ``code`` - is the code to use to request for ``access_token`` +- ``state`` - same state string used during authorization request + +Your client application should use the ``code`` to request for an +access\_token. + +3. Request for access token. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You need to make a ``POST`` request with ``grant_type``, ``code``, +``client_id`` and ``redirect_uri`` as ``POST`` payload params. You +should authenticate the request with ``Basic Authentication`` using your +``client_id`` and ``client_secret`` as ``username:password`` pair. + +Request: + +.. raw:: html + + 
+   POST/o/token
+ +Payload: + +:: + + grant_type=authorization_code&code=YYYYYYYYY&client_id=XXXXXX&redirect_uri=http://redirect/uri/path + +curl example: + +:: + + curl -X POST -d "grant_type=authorization_code& + code=PSwrMilnJESZVFfFsyEmEukNv0sGZ8& + client_id=e8x4zzJJIyOikDqjPcsCJrmnU22QbpfHQo4HhRnv& + redirect_uri=http://localhost:30000" "http://api.ona.io/o/token/" + --user "e8:xo7i4LNpMj" + +Response: + +:: + + { + "access_token": "Q6dJBs9Vkf7a2lVI7NKLT8F7c6DfLD", + "token_type": "Bearer", "expires_in": 36000, + "refresh_token": "53yF3uz79K1fif2TPtNBUFJSFhgnpE", + "scope": "read write groups" + } + +Where: + +- ``access_token`` - access token - expires +- ``refresh_token`` - token to use to request a new ``access_token`` in + case it has expored. + +Now that you have an ``access_token`` you can make API calls. + +4. Accessing the Ona API using the ``access_token``. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Example using curl: + +:: + + curl -X GET https://api.ona.io/api/v1 -H "Authorization: Bearer ACCESS_TOKEN" + +Making CORS - Cross-Origin Resource Sharing - requests to the Ona API +---------------------------------------------------------------------- +To find out more about CORS, you can read about it `here `_. The following is a javascript code snippet on how to make a CORS request. + +.. code-block:: javascript + + var xhr = new XMLHttpRequest(); + xhr.withCredentials = false; + xhr.open('GET', 'https://api.ona.io/api/v1/user', true); + xhr.setRequestHeader('Content-Type', 'application/json'); + xhr.setRequestHeader('Authorization', 'Token TOKEN_KEY'); + xhr.send(); + + +The following is a jquery code snippet on how to make a CORS request. + +.. code-block:: javascript + + $.ajax({ + method: "GET", + url: 'https://api.ona.io/api/v1/user', + dataType: 'json', + xhrFields: { + withCredentials: false + }, + headers: { + 'Authorization': 'Token TOKEN_KEY' + }, + }); diff --git a/docs/conf.py b/docs/conf.py index f42964a3fb..933bff90be 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -55,7 +55,7 @@ # General information about the project. project = u'Ona API' -copyright = u'2016, Ona' +copyright = u'2017, Ona' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the @@ -251,8 +251,8 @@ # dir menu entry, description, category) texinfo_documents = [ ('index', 'OnaAPI', u'Ona API Documentation', - u'Ona', 'OnaAPI', 'One line description of project.', - 'Miscellaneous'), + u'Ona', 'OnaAPI', 'Ona API Documentation.', + 'Miscellaneous'), ] # Documents to append as an appendix to all manuals. diff --git a/docs/index.rst b/docs/index.rst index 21c96c1946..6432b05057 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -3,12 +3,19 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -Welcome to Ona API's documentation! -======================================= +This is the Ona API and OnaData code documentation. Ona JSON Rest Api Endpoints =========================== +Authentication +-------------- + +.. toctree:: + :maxdepth: 2 + + authentication + Data Endpoints -------------- @@ -81,222 +88,13 @@ Ona Tagging API .. _List form data by tag.: data.html#query-submitted-data-of-a-specific-form-using-tags .. _Tag a specific submission: data.html#tag-a-submission-data-point -Authentication and Status Codes -=============================== - -Status Codes ------------- - -- **200** - Successful [``GET``, ``PATCH``, ``PUT``] -- **201** - Resource successfully created [``POST``\ ] -- **204** - Resouce successfully deleted [``DELETE``\ ] -- **403** - Permission denied to resource -- **404** - Resource was not found - -Authentication --------------- - -Ona JSON API enpoints support both Basic authentication and API Token -Authentication through the ``Authorization`` header. - -Basic Authentication -~~~~~~~~~~~~~~~~~~~~ - -Example using curl: - -:: - - curl -X GET https://api.ona.io/api/v1/ -u username:password - -Token Authentication -~~~~~~~~~~~~~~~~~~~~ - -Example using curl: - -:: - - curl -X GET https://api.ona.io/api/v1/ -H "Authorization: Token TOKEN_KEY" - -Temporary Token Authentication -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Example using curl: - -:: - - curl -X GET https://api.ona.io/api/v1/ -H "Authorization: TempToken TOKEN_KEY" - -Using Oauth2 with the Ona API ------------------------------ - -You can learn more about oauth2 `here`_. - -1. Register your client application with Ona - `register`_ -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -- ``name`` - name of your application -- ``client_type`` - Client Type: select confidential -- ``authorization_grant_type`` - Authorization grant type: Authorization code -- ``redirect_uri`` - Redirect urls: redirection endpoint - -Keep note of the ``client_id`` and the ``client_secret``, it is required -when requesting for an ``access_token``. - -.. _here: http://tools.ietf.org/html/rfc6749 -.. _register: /o/applications/register/ - -2. Authorize client application. -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The authorization url is of the form: - -.. raw:: html - - 
-   GET /o/authorize?client_id=XXXXXX&response_type=code&state=abc
- -example: - -:: - - http://api.ona.io/o/authorize?client_id=e8&response_type=code&state=xyz - -Note: Providing the url to any user will prompt for a password and -request for read and write permission for the application whose -``client_id`` is specified. - -Where: - -- ``client_id`` - is the client application id - ensure its urlencoded -- ``response_type`` - should be code -- ``state`` - a random state string that you client application will - get when redirection happens - -What happens: - -1. a login page is presented, the username used to login determines the - account that provides access. -2. redirection to the client application occurs, the url is of the form: - - REDIRECT\_URI/?state=abc&code=YYYYYYYYY - -example redirect uri - -:: - - http://localhost:30000/?state=xyz&code=SWWk2PN6NdCwfpqiDiPRcLmvkw2uWd - -- ``code`` - is the code to use to request for ``access_token`` -- ``state`` - same state string used during authorization request - -Your client application should use the ``code`` to request for an -access\_token. - -3. Request for access token. -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -You need to make a ``POST`` request with ``grant_type``, ``code``, -``client_id`` and ``redirect_uri`` as ``POST`` payload params. You -should authenticate the request with ``Basic Authentication`` using your -``client_id`` and ``client_secret`` as ``username:password`` pair. - -Request: - -.. raw:: html - - 
-   POST/o/token
- -Payload: - -:: - - grant_type=authorization_code&code=YYYYYYYYY&client_id=XXXXXX&redirect_uri=http://redirect/uri/path - -curl example: - -:: - - curl -X POST -d "grant_type=authorization_code& - code=PSwrMilnJESZVFfFsyEmEukNv0sGZ8& - client_id=e8x4zzJJIyOikDqjPcsCJrmnU22QbpfHQo4HhRnv& - redirect_uri=http://localhost:30000" "http://api.ona.io/o/token/" - --user "e8:xo7i4LNpMj" - -Response: - -:: - - { - "access_token": "Q6dJBs9Vkf7a2lVI7NKLT8F7c6DfLD", - "token_type": "Bearer", "expires_in": 36000, - "refresh_token": "53yF3uz79K1fif2TPtNBUFJSFhgnpE", - "scope": "read write groups" - } - -Where: - -- ``access_token`` - access token - expires -- ``refresh_token`` - token to use to request a new ``access_token`` in - case it has expored. - -Now that you have an ``access_token`` you can make API calls. - -4. Accessing the Ona API using the ``access_token``. -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Example using curl: - -:: - - curl -X GET https://api.ona.io/api/v1 -H "Authorization: Bearer ACCESS_TOKEN" - -Making CORS - Cross-Origin Resource Sharing - requests to the Ona API ----------------------------------------------------------------------- -To find out more about CORS, you can read about it `here `_. The following is a javascript code snippet on how to make a CORS request. - -.. code-block:: javascript - - var xhr = new XMLHttpRequest(); - xhr.withCredentials = false; - xhr.open('GET', 'https://api.ona.io/api/v1/user', true); - xhr.setRequestHeader('Content-Type', 'application/json'); - xhr.setRequestHeader('Authorization', 'Token TOKEN_KEY'); - xhr.send(); - - -The following is a jquery code snippet on how to make a CORS request. - -.. code-block:: javascript - - $.ajax({ - method: "GET", - url: 'https://api.ona.io/api/v1/user', - dataType: 'json', - xhrFields: { - withCredentials: false - }, - headers: { - 'Authorization': 'Token TOKEN_KEY' - }, - }); - -Formlist for preview urls --------------------------- -To generate a preview url for enketo, use the following formlist url format -:: - - https://api.ona.io//formList - - - -Quick start ------------ +Running your own server +======================= .. toctree:: :maxdepth: 2 - quick_start + install Indices and tables diff --git a/docs/install.rst b/docs/install.rst new file mode 100644 index 0000000000..bae407fbe6 --- /dev/null +++ b/docs/install.rst @@ -0,0 +1,253 @@ +Ubuntu installation instructions +================================ + +Prepare OS +---------- + +:: + ./script/install/ubuntu + +Database setup +-------------- + +In the base OS +~~~~~~~~~~~~~~ + +Replace username and db name accordingly. + +:: + sudo su postgres -c "psql -c \"CREATE USER onadata WITH PASSWORD 'onadata';\"" + sudo su postgres -c "psql -c \"CREATE DATABASE onadata OWNER onadata;\"" + sudo su postgres -c "psql -d onadata -c \"CREATE EXTENSION IF NOT EXISTS postgis;\"" + sudo su postgres -c "psql -d onadata -c \"CREATE EXTENSION IF NOT EXISTS postgis;\"" + sudo su postgres -c "psql -d onadata -c \"CREATE EXTENSION IF NOT EXISTS postgis_topology;\"" + +In Docker +~~~~~~~~~ + +These are just examples and you shouldn't run them as they are in production: +To run this use the Dockerfile in +`onaio/docker-builds `_ +for postgres 9.6.0 with postgis 2.3.0 and run: + +:: + mkdir ~/docker-images/postgres-9.6/ + cd ~/docker-images/postgres-9.6 + docker build -t postgres:9.6.0 . + +.. note:: + + This will be a persisted using ~/postgresql/data + +:: + + mkdir ~/postgresql/data + docker run -e POSTGRES_PASSWORD=pass -p 5432:5432 --volume ~/postgresql/data:/var/lib/postgresql/data --name onadata -d postgres:9.6.0 + +Connect using psql with: + +:: + + psql -h localhost -p 5432 -U postgres + +In psql: + +:: + + CREATE USER onadata WITH PASSWORD 'pass' + CREATE DATABASE onadata OWNER onadata + CONNECT onadata + CREATE EXTENSION IF NOT EXISTS postgis + CREATE EXTENSION IF NOT EXISTS postgis_topology;\"" + +From now onwards start your DB with ``docker start onadata`` provided you passed +the name "onadata" to Docker's ``--name`` option. + +Get the code +------------ + +:: + + git clone https://github.com/onaio/onadata.git + +Set up and start your virtual environment or sandbox +---------------------------------------------------- + +:: + + virtualenv <.venv> + source <.venv>/bin/activate + +Create a local_settings.py and update it accordingly +---------------------------------------------------- + +Make sure you have a ``onadata/settings/local_settings.py`` file. + +.. note:: + + This file is usually gitignored. + +Run make to set up core and for initial db setup +------------------------------------------------ + +:: + + make + +You may at this point start core with + +:: + python manage.py runserver --nothreading + +or go on and set up the rest. + +Compile api docs +---------------- + +:: + + cd docs + make html + cd .. + +Copy static files to static dir +------------------------------- + +:: + + python manage.py collectstatic --noinput + python manage.py createsuperuser + +Setup uwsgi init script +----------------------- + +:: + + pip install uwsgi + # edit uwsgi.ini accrodingly, change paths, user among other parmas + sudo cp script/etc/init/onadata.conf /etc/init/onadata.conf + # start the onadata service + sudo start onadata + # check that it started ok + # cat /path/to/onadata.log + +Setup celery service +-------------------- + +:: + + sudo apt-get install rabbitmq-server + # edit script/etc/default/celeryd-ona with correct paths and user, group + sudo cp script/etc/default/celeryd-ona /etc/default/celeryd-ona + # copy init script celeryd-ona + sudo cp script/etc/init.d/celeryd-ona /etc/init.d/celeryd-ona + sudo chmod +x /etc/init.d/celeryd-ona + sudo update-rc.d -f celeryd-ona defaults + sudo service celeryd-ona start + # confirm that the service started successfully + cat /tmp/w1-ona.log + +Setup nginx +----------- + +:: + + sudo apt-get install nginx + sudo cp script/etc/nginx/sites-available/onadata /etc/nginx/sites-available/onadata + sudo ln -s /etc/nginx/sites-available/onadata /etc/nginx/sites-enabled/onadata + # update and test /etc/nginx/sites-available/onadata + sudo service nginx configtest + # remove default nginx server config + sudo unlink /etc/nginx/sites-enabled/default + sudo service nginx restart + +Mac OS Installation Instructions +================================ + +Step 1: Install dependencies using brew +--------------------------------------- + +`Install homebrew `_ and run the following commands: + +:: + brew install mongo + brew install postgis + brew install gdal + brew install rabbitmq + brew install libmemcached + + +Add the following to your ``~/.bash_profile`` or ``~/.zprofile`` + +:: + export LIBMEMCACHED=/usr/local + export LC_ALL=en_US.UTF-8 + export LANG=en_US.UTF-8 + PATH=$PATH:/usr/local/sbin + +Rabbitmq is not automatically added to your path that's why we add ``PATH=$PATH:/usr/local/sbin``. + +Step 2: Install pip and virtualenv +---------------------------------- + +Install pip using `easy_install pip` if you don't have it already. + +Install `virtualenvwrapper `_ and then create a virtual environment. + +Step 3: Clone the sourcecode +---------------------------- + +Clone `onadata `_ in your directory of choice + +Step 4: Install app requirements +-------------------------------- + +Before you install dependencies from the requirements directory files, ensure you have activated your virtual environment and if not, use the ``workon `` to activate it. Then, run the following command: + +:: + pip install numpy --use-mirrors + pip install -r requirements/base.pip --allow-all-external + pip install -r requirements/dev.pip + +There is a known bug that prevents numpy from installing correctly when in requirements.pip file + +Step 5: Install postgres and create your database +------------------------------------------------- + +`Install postgres ``_ and access postgres in your +terminal using the command ``psql`` and use the following commands to create +your user and database: + +:: + CREATE USER WITH PASSWORD '' SUPERUSER CREATEDB LOGIN; + CREATE DATABASE WITH ENCODING='UTF8' LC_CTYPE='en_US.UTF-8' LC_COLLATE='en_US.UTF-8' OWNER= TEMPLATE=template0; + +You will also need to create some extensions in your newly created database. +Enter the command ``\c `` to connect to your database then run +the following commands to install the extensions: + +:: + CREATE EXTENSION IF NOT EXISTS postgis; + CREATE EXTENSION IF NOT EXISTS postgis_topology; + +Create `local_settings.py` file in the root of you cloned app if you don't have one already and update the `DATABASE` property with the details above. + +Step 6: Test installation using development server +-------------------------------------------------- + +Run + +:: + python manage.py runserver + +Step 7: Using celery +-------------------- + +Start rabbitmq with the command ``rabbitmq-server`` in a different terminal +window. + +Add ``CELERY_ALWAYS_EAGER = False`` to your local_settings if doesn't exist +already. + +Run ``python manage.py celeryd -l debug`` on the root the app directory in a +different terminal window. diff --git a/docs/quick_start.rst b/docs/quick_start.rst deleted file mode 100644 index 92029f5494..0000000000 --- a/docs/quick_start.rst +++ /dev/null @@ -1,2 +0,0 @@ -Quick Start -=========== \ No newline at end of file diff --git a/install.md b/install.md deleted file mode 100644 index ab76a46ac1..0000000000 --- a/install.md +++ /dev/null @@ -1,101 +0,0 @@ -# Ubuntu installation instructions. -## Prepare OS - $ ./script/install/ubuntu - -## Database setup - -### In the base OS -Replace username and db name accordingly. - - sudo su postgres -c "psql -c \"CREATE USER onadata WITH PASSWORD 'onadata';\"" - sudo su postgres -c "psql -c \"CREATE DATABASE onadata OWNER onadata;\"" - sudo su postgres -c "psql -d onadata -c \"CREATE EXTENSION IF NOT EXISTS postgis;\"" - sudo su postgres -c "psql -d onadata -c \"CREATE EXTENSION IF NOT EXISTS postgis;\"" - sudo su postgres -c "psql -d onadata -c \"CREATE EXTENSION IF NOT EXISTS postgis_topology;\"" - -### In Docker -These are just examples and you shouldn't run them as they are in production: -Use the Dockerfile in [onaio/docker-builds](https://github.com/onaio/docker-builds/tree/master/postgres) for postgres 9.6.0 with postgis 2.3.0. -``` -$ mkdir ~/docker-images/postgres-9.6/ -$ cd ~/docker-images/postgres-9.6 -$ docker build -t postgres:9.6.0 . -``` - -To run it. - -> This will be a persistent using ~/postgresql/data - -``` -$ mkdir ~/postgresql/data -$ docker run -e POSTGRES_PASSWORD=pass -p 5432:5432 --volume ~/postgresql/data:/var/lib/postgresql/data --name onadata -d postgres:9.6.0 -``` - -Connect using psql with: -`psql -h localhost -p 5432 -U postgres` - -In psql: -``` -CREATE USER onadata WITH PASSWORD 'pass' -CREATE DATABASE onadata OWNER onadata -CONNECT onadata -CREATE EXTENSION IF NOT EXISTS postgis -CREATE EXTENSION IF NOT EXISTS postgis_topology;\"" -``` - -From now onwards start your DB with `docker start onadata` provided you passed the name "onadata" to Docker's `--name` option. - -## Get the code - git clone https://github.com/onaio/onadata.git - -## Set up and start your virtual environment or sandbox. - $ virtualenv <.venv> - $ source <.venv>/bin/activate - -## Create a local_settings.py, update it accordingly. -Make sure you have a `onadata/settings/local_settings.py` file. -*This file is usually gitignored. - -## Run make to set up core and for initial db setup. - $ make -You may at this point start core with `$ python manage.py runserver --nothreading` or go on and set up the rest. - -## compile api docs - cd docs - make html - cd .. - -## copy static files to static dir - python manage.py collectstatic --noinput - python manage.py createsuperuser - -## Setup uwsgi init script - pip install uwsgi - # edit uwsgi.ini accrodingly, change paths, user among other parmas - sudo cp script/etc/init/onadata.conf /etc/init/onadata.conf - # start the onadata service - sudo start onadata - # check that it started ok - # cat /path/to/onadata.log - -## Setup celery service - sudo apt-get install rabbitmq-server - # edit script/etc/default/celeryd-ona with correct paths and user, group - sudo cp script/etc/default/celeryd-ona /etc/default/celeryd-ona - # copy init script celeryd-ona - sudo cp script/etc/init.d/celeryd-ona /etc/init.d/celeryd-ona - sudo chmod +x /etc/init.d/celeryd-ona - sudo update-rc.d -f celeryd-ona defaults - sudo service celeryd-ona start - # confirm that the service started successfully - cat /tmp/w1-ona.log - -## Setup nginx - sudo apt-get install nginx - sudo cp script/etc/nginx/sites-available/onadata /etc/nginx/sites-available/onadata - sudo ln -s /etc/nginx/sites-available/onadata /etc/nginx/sites-enabled/onadata - # update and test /etc/nginx/sites-available/onadata - sudo service nginx configtest - # remove default nginx server config - sudo unlink /etc/nginx/sites-enabled/default - sudo service nginx restart From 0cae9d5b8bf4ea75a51ff562bfe98d045e7ab3c6 Mon Sep 17 00:00:00 2001 From: pld Date: Fri, 27 Jan 2017 13:16:38 +0300 Subject: [PATCH 2/4] update auto docs --- docs/onadata.apps.api.management.commands.rst | 8 +++++ docs/onadata.apps.logger.migrations.rst | 24 ++++++++++++++ .../onadata.apps.main.management.commands.rst | 8 +++++ docs/onadata.apps.main.rst | 8 ----- docs/onadata.apps.restservice.services.rst | 8 ----- docs/onadata.apps.viewer.rst | 8 +++++ docs/onadata.libs.models.rst | 8 ----- docs/onadata.libs.serializers.fields.rst | 8 +++++ docs/onadata.libs.serializers.rst | 16 +++++----- docs/onadata.libs.tests.serializers.rst | 8 ----- docs/onadata.libs.tests.utils.rst | 16 +++++----- docs/onadata.libs.utils.rst | 32 ++++++++++++++----- docs/onadata.settings.rst | 8 +++++ 13 files changed, 104 insertions(+), 56 deletions(-) diff --git a/docs/onadata.apps.api.management.commands.rst b/docs/onadata.apps.api.management.commands.rst index e7f517a138..d9bef4eb48 100644 --- a/docs/onadata.apps.api.management.commands.rst +++ b/docs/onadata.apps.api.management.commands.rst @@ -28,6 +28,14 @@ onadata.apps.api.management.commands.create_default_project module :undoc-members: :show-inheritance: +onadata.apps.api.management.commands.fix_readonly_role_perms module +------------------------------------------------------------------- + +.. automodule:: onadata.apps.api.management.commands.fix_readonly_role_perms + :members: + :undoc-members: + :show-inheritance: + onadata.apps.api.management.commands.migrate_group_permissions module --------------------------------------------------------------------- diff --git a/docs/onadata.apps.logger.migrations.rst b/docs/onadata.apps.logger.migrations.rst index c3b4000fb5..7baabf90f0 100644 --- a/docs/onadata.apps.logger.migrations.rst +++ b/docs/onadata.apps.logger.migrations.rst @@ -196,6 +196,30 @@ onadata.apps.logger.migrations.0024_xform_has_hxl_support module :undoc-members: :show-inheritance: +onadata.apps.logger.migrations.0025_xform_last_updated_at module +---------------------------------------------------------------- + +.. automodule:: onadata.apps.logger.migrations.0025_xform_last_updated_at + :members: + :undoc-members: + :show-inheritance: + +onadata.apps.logger.migrations.0026_auto_20160913_0239 module +------------------------------------------------------------- + +.. automodule:: onadata.apps.logger.migrations.0026_auto_20160913_0239 + :members: + :undoc-members: + :show-inheritance: + +onadata.apps.logger.migrations.0027_auto_20161201_0730 module +------------------------------------------------------------- + +.. automodule:: onadata.apps.logger.migrations.0027_auto_20161201_0730 + :members: + :undoc-members: + :show-inheritance: + Module contents --------------- diff --git a/docs/onadata.apps.main.management.commands.rst b/docs/onadata.apps.main.management.commands.rst index a4d397ec71..673b115440 100644 --- a/docs/onadata.apps.main.management.commands.rst +++ b/docs/onadata.apps.main.management.commands.rst @@ -12,6 +12,14 @@ onadata.apps.main.management.commands.create_enketo_express_urls module :undoc-members: :show-inheritance: +onadata.apps.main.management.commands.create_metadata_for_kpi_deployed_forms module +----------------------------------------------------------------------------------- + +.. automodule:: onadata.apps.main.management.commands.create_metadata_for_kpi_deployed_forms + :members: + :undoc-members: + :show-inheritance: + onadata.apps.main.management.commands.export_user_emails module --------------------------------------------------------------- diff --git a/docs/onadata.apps.main.rst b/docs/onadata.apps.main.rst index ddcdd8731a..8d41cd9c5e 100644 --- a/docs/onadata.apps.main.rst +++ b/docs/onadata.apps.main.rst @@ -39,14 +39,6 @@ onadata.apps.main.forms module :undoc-members: :show-inheritance: -onadata.apps.main.google_export module --------------------------------------- - -.. automodule:: onadata.apps.main.google_export - :members: - :undoc-members: - :show-inheritance: - onadata.apps.main.registration_urls module ------------------------------------------ diff --git a/docs/onadata.apps.restservice.services.rst b/docs/onadata.apps.restservice.services.rst index fb97ae3b9a..1e09dff048 100644 --- a/docs/onadata.apps.restservice.services.rst +++ b/docs/onadata.apps.restservice.services.rst @@ -28,14 +28,6 @@ onadata.apps.restservice.services.generic_xml module :undoc-members: :show-inheritance: -onadata.apps.restservice.services.google_sheets module ------------------------------------------------------- - -.. automodule:: onadata.apps.restservice.services.google_sheets - :members: - :undoc-members: - :show-inheritance: - onadata.apps.restservice.services.textit module ----------------------------------------------- diff --git a/docs/onadata.apps.viewer.rst b/docs/onadata.apps.viewer.rst index 35351089c5..279daea758 100644 --- a/docs/onadata.apps.viewer.rst +++ b/docs/onadata.apps.viewer.rst @@ -23,6 +23,14 @@ onadata.apps.viewer.admin module :undoc-members: :show-inheritance: +onadata.apps.viewer.parsed_instance_tools module +------------------------------------------------ + +.. automodule:: onadata.apps.viewer.parsed_instance_tools + :members: + :undoc-members: + :show-inheritance: + onadata.apps.viewer.tasks module -------------------------------- diff --git a/docs/onadata.libs.models.rst b/docs/onadata.libs.models.rst index 2a39ed4ada..cda91271f1 100644 --- a/docs/onadata.libs.models.rst +++ b/docs/onadata.libs.models.rst @@ -20,14 +20,6 @@ onadata.libs.models.clone_xform module :undoc-members: :show-inheritance: -onadata.libs.models.google_sheet_service module ------------------------------------------------ - -.. automodule:: onadata.libs.models.google_sheet_service - :members: - :undoc-members: - :show-inheritance: - onadata.libs.models.share_project module ---------------------------------------- diff --git a/docs/onadata.libs.serializers.fields.rst b/docs/onadata.libs.serializers.fields.rst index e15bd00236..2e2a4160e3 100644 --- a/docs/onadata.libs.serializers.fields.rst +++ b/docs/onadata.libs.serializers.fields.rst @@ -36,6 +36,14 @@ onadata.libs.serializers.fields.json_field module :undoc-members: :show-inheritance: +onadata.libs.serializers.fields.organization_field module +--------------------------------------------------------- + +.. automodule:: onadata.libs.serializers.fields.organization_field + :members: + :undoc-members: + :show-inheritance: + onadata.libs.serializers.fields.project_field module ---------------------------------------------------- diff --git a/docs/onadata.libs.serializers.rst b/docs/onadata.libs.serializers.rst index c8deac9d84..540802507b 100644 --- a/docs/onadata.libs.serializers.rst +++ b/docs/onadata.libs.serializers.rst @@ -67,14 +67,6 @@ onadata.libs.serializers.geojson_serializer module :undoc-members: :show-inheritance: -onadata.libs.serializers.google_serializer module -------------------------------------------------- - -.. automodule:: onadata.libs.serializers.google_serializer - :members: - :undoc-members: - :show-inheritance: - onadata.libs.serializers.metadata_serializer module --------------------------------------------------- @@ -91,6 +83,14 @@ onadata.libs.serializers.note_serializer module :undoc-members: :show-inheritance: +onadata.libs.serializers.organization_member_serializer module +-------------------------------------------------------------- + +.. automodule:: onadata.libs.serializers.organization_member_serializer + :members: + :undoc-members: + :show-inheritance: + onadata.libs.serializers.organization_serializer module ------------------------------------------------------- diff --git a/docs/onadata.libs.tests.serializers.rst b/docs/onadata.libs.tests.serializers.rst index 7a4044e418..4ab4f20ae7 100644 --- a/docs/onadata.libs.tests.serializers.rst +++ b/docs/onadata.libs.tests.serializers.rst @@ -20,14 +20,6 @@ onadata.libs.tests.serializers.test_dataview_serializer module :undoc-members: :show-inheritance: -onadata.libs.tests.serializers.test_google_sheet_serializer module ------------------------------------------------------------------- - -.. automodule:: onadata.libs.tests.serializers.test_google_sheet_serializer - :members: - :undoc-members: - :show-inheritance: - onadata.libs.tests.serializers.test_organization_serializer module ------------------------------------------------------------------ diff --git a/docs/onadata.libs.tests.utils.rst b/docs/onadata.libs.tests.utils.rst index 49c0d8117b..56d01b7df0 100644 --- a/docs/onadata.libs.tests.utils.rst +++ b/docs/onadata.libs.tests.utils.rst @@ -12,6 +12,14 @@ onadata.libs.tests.utils.test_api_export_tools module :undoc-members: :show-inheritance: +onadata.libs.tests.utils.test_async_status module +------------------------------------------------- + +.. automodule:: onadata.libs.tests.utils.test_async_status + :members: + :undoc-members: + :show-inheritance: + onadata.libs.tests.utils.test_backup_tools module ------------------------------------------------- @@ -60,14 +68,6 @@ onadata.libs.tests.utils.test_export_tools module :undoc-members: :show-inheritance: -onadata.libs.tests.utils.test_google_sheets_tools module --------------------------------------------------------- - -.. automodule:: onadata.libs.tests.utils.test_google_sheets_tools - :members: - :undoc-members: - :show-inheritance: - onadata.libs.tests.utils.test_image_tools module ------------------------------------------------ diff --git a/docs/onadata.libs.utils.rst b/docs/onadata.libs.utils.rst index 4b8b1d7ef3..4090cefee3 100644 --- a/docs/onadata.libs.utils.rst +++ b/docs/onadata.libs.utils.rst @@ -12,6 +12,14 @@ onadata.libs.utils.api_export_tools module :undoc-members: :show-inheritance: +onadata.libs.utils.async_status module +-------------------------------------- + +.. automodule:: onadata.libs.utils.async_status + :members: + :undoc-members: + :show-inheritance: + onadata.libs.utils.audit module ------------------------------- @@ -116,6 +124,14 @@ onadata.libs.utils.dict_tools module :undoc-members: :show-inheritance: +onadata.libs.utils.export_builder module +---------------------------------------- + +.. automodule:: onadata.libs.utils.export_builder + :members: + :undoc-members: + :show-inheritance: + onadata.libs.utils.export_tools module -------------------------------------- @@ -132,14 +148,6 @@ onadata.libs.utils.google module :undoc-members: :show-inheritance: -onadata.libs.utils.google_sheets_tools module ---------------------------------------------- - -.. automodule:: onadata.libs.utils.google_sheets_tools - :members: - :undoc-members: - :show-inheritance: - onadata.libs.utils.gravatar module ---------------------------------- @@ -188,6 +196,14 @@ onadata.libs.utils.model_tools module :undoc-members: :show-inheritance: +onadata.libs.utils.mongo module +------------------------------- + +.. automodule:: onadata.libs.utils.mongo + :members: + :undoc-members: + :show-inheritance: + onadata.libs.utils.numeric module --------------------------------- diff --git a/docs/onadata.settings.rst b/docs/onadata.settings.rst index f856ba8ba1..8e11380a74 100644 --- a/docs/onadata.settings.rst +++ b/docs/onadata.settings.rst @@ -36,6 +36,14 @@ onadata.settings.drone_test module :undoc-members: :show-inheritance: +onadata.settings.local_settings module +-------------------------------------- + +.. automodule:: onadata.settings.local_settings + :members: + :undoc-members: + :show-inheritance: + onadata.settings.production_example module ------------------------------------------ From 0f1de40ea0a67573ea4a4a420cb3335cd52c8f6c Mon Sep 17 00:00:00 2001 From: pld Date: Fri, 27 Jan 2017 13:24:23 +0300 Subject: [PATCH 3/4] update doc building instructions, remove $s --- README.rst | 44 +++++++++++++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 15 deletions(-) diff --git a/README.rst b/README.rst index 2b21091a5d..62ca828b3e 100644 --- a/README.rst +++ b/README.rst @@ -29,7 +29,7 @@ Edit top level requirements in the file `requirements/base.in `_. + +Run sphinx in autobuild mode using: + .. code-block:: sh - $ cd docs - $ make html - $ python manage.py collectstatic + sphinx-autobuild docs docs/_build/html + +Requires sphinx-autobuild, install with ``pip install sphinx-autobuild``. + Django Debug Toolbar -------------------- @@ -105,11 +119,11 @@ Fake initial migration of `guardian`, `django_digest`, `registration`. Migrate ` .. code-block:: sh - $ python manage.py migrate contenttypes - $ python manage.py migrate --fake-initial django_digest - $ python manage.py migrate --fake-initial guardian - $ python manage.py migrate --fake-initial registration - $ python manage.py migrate + python manage.py migrate contenttypes + python manage.py migrate --fake-initial django_digest + python manage.py migrate --fake-initial guardian + python manage.py migrate --fake-initial registration + python manage.py migrate **Major django changes affecting Ona** From b8c81c038e9c3f9c9f818387b60f20fb3de9073c Mon Sep 17 00:00:00 2001 From: pld Date: Fri, 27 Jan 2017 13:24:32 +0300 Subject: [PATCH 4/4] add codeblock format --- docs/install.rst | 50 +++++++++++++++++++++++++++++------------------- notes.md | 8 -------- 2 files changed, 30 insertions(+), 28 deletions(-) delete mode 100644 notes.md diff --git a/docs/install.rst b/docs/install.rst index bae407fbe6..92d5688b20 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -4,7 +4,8 @@ Ubuntu installation instructions Prepare OS ---------- -:: +.. code-block:: sh + ./script/install/ubuntu Database setup @@ -15,7 +16,8 @@ In the base OS Replace username and db name accordingly. -:: +.. code-block:: sh + sudo su postgres -c "psql -c \"CREATE USER onadata WITH PASSWORD 'onadata';\"" sudo su postgres -c "psql -c \"CREATE DATABASE onadata OWNER onadata;\"" sudo su postgres -c "psql -d onadata -c \"CREATE EXTENSION IF NOT EXISTS postgis;\"" @@ -30,7 +32,8 @@ To run this use the Dockerfile in `onaio/docker-builds `_ for postgres 9.6.0 with postgis 2.3.0 and run: -:: +.. code-block:: sh + mkdir ~/docker-images/postgres-9.6/ cd ~/docker-images/postgres-9.6 docker build -t postgres:9.6.0 . @@ -39,20 +42,20 @@ for postgres 9.6.0 with postgis 2.3.0 and run: This will be a persisted using ~/postgresql/data -:: +.. code-block:: sh mkdir ~/postgresql/data docker run -e POSTGRES_PASSWORD=pass -p 5432:5432 --volume ~/postgresql/data:/var/lib/postgresql/data --name onadata -d postgres:9.6.0 Connect using psql with: -:: +.. code-block:: sh psql -h localhost -p 5432 -U postgres In psql: -:: +.. code-block:: sql CREATE USER onadata WITH PASSWORD 'pass' CREATE DATABASE onadata OWNER onadata @@ -66,14 +69,14 @@ the name "onadata" to Docker's ``--name`` option. Get the code ------------ -:: +.. code-block:: sh git clone https://github.com/onaio/onadata.git Set up and start your virtual environment or sandbox ---------------------------------------------------- -:: +.. code-block:: sh virtualenv <.venv> source <.venv>/bin/activate @@ -90,13 +93,14 @@ Make sure you have a ``onadata/settings/local_settings.py`` file. Run make to set up core and for initial db setup ------------------------------------------------ -:: +.. code-block:: sh make You may at this point start core with -:: +.. code-block:: sh + python manage.py runserver --nothreading or go on and set up the rest. @@ -104,7 +108,7 @@ or go on and set up the rest. Compile api docs ---------------- -:: +.. code-block:: sh cd docs make html @@ -113,7 +117,7 @@ Compile api docs Copy static files to static dir ------------------------------- -:: +.. code-block:: sh python manage.py collectstatic --noinput python manage.py createsuperuser @@ -121,7 +125,7 @@ Copy static files to static dir Setup uwsgi init script ----------------------- -:: +.. code-block:: sh pip install uwsgi # edit uwsgi.ini accrodingly, change paths, user among other parmas @@ -134,7 +138,7 @@ Setup uwsgi init script Setup celery service -------------------- -:: +.. code-block:: sh sudo apt-get install rabbitmq-server # edit script/etc/default/celeryd-ona with correct paths and user, group @@ -150,7 +154,7 @@ Setup celery service Setup nginx ----------- -:: +.. code-block:: sh sudo apt-get install nginx sudo cp script/etc/nginx/sites-available/onadata /etc/nginx/sites-available/onadata @@ -169,7 +173,8 @@ Step 1: Install dependencies using brew `Install homebrew `_ and run the following commands: -:: +.. code-block:: sh + brew install mongo brew install postgis brew install gdal @@ -180,6 +185,7 @@ Step 1: Install dependencies using brew Add the following to your ``~/.bash_profile`` or ``~/.zprofile`` :: + export LIBMEMCACHED=/usr/local export LC_ALL=en_US.UTF-8 export LANG=en_US.UTF-8 @@ -204,7 +210,8 @@ Step 4: Install app requirements Before you install dependencies from the requirements directory files, ensure you have activated your virtual environment and if not, use the ``workon `` to activate it. Then, run the following command: -:: +.. code-block:: sh + pip install numpy --use-mirrors pip install -r requirements/base.pip --allow-all-external pip install -r requirements/dev.pip @@ -218,7 +225,8 @@ Step 5: Install postgres and create your database terminal using the command ``psql`` and use the following commands to create your user and database: -:: +.. code-block:: sql + CREATE USER WITH PASSWORD '' SUPERUSER CREATEDB LOGIN; CREATE DATABASE WITH ENCODING='UTF8' LC_CTYPE='en_US.UTF-8' LC_COLLATE='en_US.UTF-8' OWNER= TEMPLATE=template0; @@ -226,7 +234,8 @@ You will also need to create some extensions in your newly created database. Enter the command ``\c `` to connect to your database then run the following commands to install the extensions: -:: +.. code-block:: sql + CREATE EXTENSION IF NOT EXISTS postgis; CREATE EXTENSION IF NOT EXISTS postgis_topology; @@ -237,7 +246,8 @@ Step 6: Test installation using development server Run -:: +.. code-block:: sh + python manage.py runserver Step 7: Using celery diff --git a/notes.md b/notes.md deleted file mode 100644 index 009e00ec13..0000000000 --- a/notes.md +++ /dev/null @@ -1,8 +0,0 @@ -- PostgreSQL - PostgreSQL is the most capable of all the databases here in terms of schema support; the only caveat is that adding columns with default values will cause a full rewrite of the table, for a time proportional to its size. - For this reason, it’s recommended you always create new columns with null=True, as this way they will be added immediately. - -Might be worthing looking into: -- prefetch_related - https://docs.djangoproject.com/en/1.7/ref/models/querysets/#django.db.models.query.QuerySet.prefetch_related -- custom lookups - https://docs.djangoproject.com/en/1.7/howto/custom-lookups/ -- Django Guardian: User.get_all_permissions