INSTALL

# HOWTO install OS2Webscanner on your own server

## System requirements

We recommend that any server that you wish to use for running this scanner must
have at least 4GB RAM, a CPU with 4 cores and at least as much free disk space
as used by the web sites you want to scan.

*Note:* The installer is currently only set up to work with Debian-like
GNU/Linux systems, as it depends on the `apt` package management system.
Magenta has tested the system on _Ubuntu 12.04 only_, but it should work
without problem on all recent Debian-like systems.

If you wish to run the system on a non-Debian-like GNU/Linux system or on
FreeBSD or OSX, you will have to modify `install.sh` yourself to handle the
package managers, package names and dependencies in these systems. Also note
that the setup of Apache is different on such systems.

*Note:* The Apache configuration file delivered with version 1.1.0 of
OS2Webscanner is for Apache <= 2.2  and will *not* work on Apache 2.4.
A configuration file for Apache 2.4 will be supplied in future versions.

In the meantime, please refer to
http://httpd.apache.org/docs/2.4/upgrading.html for instructions on how to
upgrade the configuration file for running Apache 2.4. As Apache 2.2 is not
available in Ubuntu 14.04, please note that the present instructions will _not_
work without modification in Ubuntu 14.04. We do not recommend running a
production server on non-LTS versions of Ubuntu.

If you wish to run the system on a Windows system, you need to install a C
compiler environment, e.g. MinGW or Cygwin. Running on Windows has not been
tested.


## Select installation directory

We recommend that you install the system as a dedicated user, e.g. with
username `os2`. To proceed with the installation, create the user, prevent
remote login and login as that user:

```
sudo adduser os2
sudo adduser os2 sudo
sudo su - os2
```


Note, you are now logged in as the user `os2`. We assume this in the rest of
this guide. Also note that this is only a convention - you are free to install
the system wherever you want. Paths are not hard coded in the system.

*Note:* You should probably remove `os2`'s sudo rights once the installation is
complete.

## Install Git, Apache and WSGI

```
sudo apt-get install git apache2 libapache2-mod-wsgi
```

## Get the code

```
git clone https://github.com/os2webscanner/os2webscanner.git
```

## Install Django and Scrapy, including system dependencies

```
cd os2webscanner
./install.sh
```

You will be prompted for the `sudo` password.

*Note:* This requires an Internet connection and may take a while.


## Set up the database

If you performed the previous step, you've already installed `postgresql`
and `postgresql-client`.

Assuming this, do:

```
sudo su - postgres
createdb os2webscanner
createuser os2webscanner
psql
```

The following commands must be executed IN psql shell.

```
grant all on database os2webscanner to os2webscanner;
alter user os2webscanner with password '<PASSWORD>';
alter user os2webscanner createdb;
```

You may (of course) use another database name, database user etc. as you
please.

Now log out from psql (`\q`) and logout as the postgres user (by pressing
`Ctrl+D`) to proceed, and return to where you cloned the `os2webscanner` source
directory (e.g. `/home/os2/os2webscanner`).

Next, change into the `webscanner_site/webscanner` directory:

```
cd webscanner_site/webscanner
```

Copy the file `local_settings.py.example` to `local_settings.py` and open
`local_settings.py` for editing:

```
cp local_settings.py.example local_settings.py
<< edit local_settings.py with your favorite editor>>
```

In order to make your database setup work, you must override the default
DATABASES configuration.

A sample `local_settings.py` configured for your server could look like this:

```
SITE_URL = 'http://webscanner.kommune.dk'
STATIC_ROOT = '/home/os2/os2webscanner/webscanner_site/static'
MEDIA_ROOT = '/home/os2/os2webscanner/webscanner_site/uploads'
DEFAULT_FROM_EMAIL = 'your@email'
ADMIN_EMAIL = 'your@email'

# Database
# https://docs.djangoproject.com/en/1.6/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'os2webscanner',
        'USER': 'os2webscanner',
        'PASSWORD': 'YOUR_PASSWORD',
        'HOST': '127.0.0.1',
    }
}
```

## Test and initialize


```
cd /home/os2/os2webscanner/webscanner_site
source ../python-env/bin/activate
python manage.py test os2webscanner
```

The test should pass. Now do:

```
python manage.py migrate
```

and create a user with a password you can remember.

## Deployment with Apache

First, collect content to be served statically:

```
cd /home/os2/os2webscanner/webscanner_site
python manage.py collectstatic
```

Next, deploy Apache configuration:
```
cd /home/os2/os2webscanner
sudo cp config/apache.conf /etc/apache2/sites-available/webscanner
```

Now, before activating the site, please _edit_ the Apache configuration.

* If you're using SSL, please supply paths to your certificate files.
* Change the `ServerName` directive to the FQDN of your own server.
* If you're not installing to the directory `/home/os2/os2webscanner/`, please
  change all paths accordingly.
* If you're _not_ using SSL, please delete the first VirtualHost, specify port
  80 for the second one and delete all directives starting with the letters
  "SSL".

If using SSL, you need to enable the extensions `mod_rewrite` and `mod_ssl`:

```
sudo a2enmod rewrite
sudo a2enmod ssl
```

You also need to create the Apache log directories:

```
sudo mkdir -p /var/log/os2webscanner/
```

With all this in place, you may now enable the Apache site:

```
sudo a2ensite webscanner
sudo service apache2 restart
```

The webscanner should now be available at the URL you specified as ServerName
in your VirtualHost, e.g. "https://webscanner.kommune.dk".

## Start the scanning processors

First, make the logs directory writable by the web server user:

```
sudo chown -R www-data:os2 /home/os2/os2webscanner/var
```

Next, start the _process manager_ background process in order to get scans
which scan non-text files (e.g. PDF files or Office documents) to work.

```
sudo -u www-data -b /home/os2/os2webscanner/scrapy-webscanner/start_process_manager.sh
```

*Note:* You may want to have the scanners `var` dir somewhere else, e.g. in
`/var/lib/os2webscanner`, which is the location we (the developers) prefer for
production environments. To achieve this, please overwrite the Django setting
`VAR_DIR` in your `local_settings.py` accordingly and set ownership for the
directory as indicated above. 


## Setting up scheduled scans

To setup scheduled scans, you need to add an entry to the user www-data's
crontab:

```
sudo crontab -u www-data -e
```

Add the following line below the commented lines (beginning with '#'), and
then save the file:

```
*/15 * * * *    /home/os2/os2webscanner/cron/run_cron_script.sh
```


## Setting up scheduled summary reports

The system may send out summary reports describing the performance, results,
etc., of different scanners. 

To have summaries emailed to recipients, edit `crontab` as described in the
previous section, adding the line

```
0    7 * * * /home/os2/os2webscanner/cron/dispatch_summaries.sh
```

to have summaries emailed every day at 7AM. You can of course change this as
you wish, but summaries should be mailed no more than once a day as this may
cause reports to be sent twice.


## Creating an organization and adding a user to it:

Visit your webscanner site URL + `/admin/` to enter the Django admin interface.

Login with the Django superuser you created (when running `python manage.py
syncdb`). Click on "Organization" and hit the button labeled "Tilføj
Organisation" or "Add Organization" to add an "Organization". This is
necessary - the system will not work without at least one organization. Give
your new organization a name, email address and phone number and save it by
clicking "Gem" or "Save" at the bottom of the page.

Return to the main admin page and click "Brugere".
Click the username that you would like to add to the organization.

At the bottom of the page, under "User profiles", change the "Organisation"
to the organization you created and save.

OS2Webscanner is now ready to be used.