Skip to content

Latest commit

 

History

History
611 lines (458 loc) · 18.7 KB

chapter_making_deployment_production_ready.asciidoc

File metadata and controls

611 lines (458 loc) · 18.7 KB

Getting to a Production-Ready Deployment

Chapter info

shortname:

chapter_making_deployment_production_ready
Warning
 2017-03-13: Book upgraded to Python 3.6 and Django 1.11 beta. Before that, there was a big upgrade to Selenium 3 on 2017-01-30. More info on the blog.

In this chapter we’ll make some changes to our site to move to a configuration that’s more production-ready. As we make each change, we’ll use the tests to tell us whether things are still working.

What’s wrong with our hacky deployment? Well, we can’t use the Django dev server for production, it’s not designed for "real-life" loads. We’ll use something called Gunicorn instead to run our Django code, and we’ll get Nginx to serve our static files.

Our 'settings.py' currently has DEBUG=True, and that’s strongly recommended against for production (you don’t want users staring at debug tracebacks of your code when your site errors for example). We’ll also need to set ALLOWED_HOSTS for security.

We want our site to start up automatically whenever the server reboots. For that we’ll write a Systemd config file.

Finally, hard-coding port 8000 won’t let us run multiple sites on this server, so we’ll switch to using "unix sockets" to communicate between nginx and Django.

Switching to Gunicorn

Do you know why the Django mascot is a pony? The story is that Django comes with so many things you want: an ORM, all sorts of middleware, the admin site…​ "What else do you want, a pony?" Well, Gunicorn stands for "Green Unicorn", which I guess is what you’d want next if you already had a pony…​

elspeth@server:$ ../virtualenv/bin/pip install gunicorn

Gunicorn will need to know a path to a WSGI server, which is usually a function called application. Django provides one in 'superlists/wsgi.py':

elspeth@server:$ ../virtualenv/bin/gunicorn superlists.wsgi:application
2013-05-27 16:22:01 [10592] [INFO] Starting gunicorn 0.19.6
2013-05-27 16:22:01 [10592] [INFO] Listening at: http://127.0.0.1:8000 (10592)
[...]

If you now take a look at the site, you’ll find the CSS is all broken, as in Broken CSS.

And if we run the functional tests, you’ll see they confirm that something is wrong. The test for adding list items passes happily, but the test for layout + styling fails. Good job tests!

$ STAGING_SERVER=superlists-staging.ottg.eu python manage.py test functional_tests
[...]
AssertionError: 125.0 != 512 within 3 delta
FAILED (failures=1)

The reason that the CSS is broken is that although the Django dev server will serve static files magically for you, Gunicorn doesn’t. Now is the time to tell Nginx to do it instead.

The site is up, but CSS is broken
Figure 1. Broken CSS

One step forward, one step backward, but at least the tests are there to help. Moving on!

Getting Nginx to Serve Static Files

First we run collectstatic to copy all the static files to a folder where Nginx can find them:

elspeth@server:$ ../virtualenv/bin/python manage.py collectstatic --noinput
elspeth@server:$ ls ../static/
base.css  bootstrap

Now we tell Nginx to start serving those static files for us:

server: /etc/nginx/sites-available/superlists-staging.ottg.eu
server {
    listen 80;
    server_name superlists-staging.ottg.eu;

    location /static {
        alias /home/elspeth/sites/superlists-staging.ottg.eu/static;
    }

    location / {
        proxy_pass http://localhost:8000;
    }
}

Reload Nginx and restart Gunicorn…​

elspeth@server:$ sudo systemctl reload nginx
elspeth@server:$ ../virtualenv/bin/gunicorn superlists.wsgi:application

And if we take another look at the site, things are looking much healthier. We can rerun our FTs:

$ STAGING_SERVER=superlists-staging.ottg.eu python manage.py test functional_tests
[...]

...
 ---------------------------------------------------------------------
Ran 3 tests in 10.718s

OK

Phew.

Switching to Using Unix Sockets

When we want to serve both staging and live, we can’t have both servers trying to use port 8000. We could decide to allocate different ports, but that’s a bit arbitrary, and it would be dangerously easy to get it wrong and start the staging server on the live port, or vice versa.

A better solution is to use Unix domain sockets—​they’re like files on disk, but can be used by Nginx and Gunicorn to talk to each other. We’ll put our sockets in '/tmp'. Let’s change the proxy settings in Nginx:

server: /etc/nginx/sites-available/superlists-staging.ottg.eu
[...]
    location / {
        proxy_set_header Host $host;
        proxy_pass http://unix:/tmp/superlists-staging.ottg.eu.socket;
    }
}

proxy_set_header is used to make sure Gunicorn and Django know what domain it’s running on. We need that for the ALLOWED_HOSTS security feature, which we’re about to switch on.

Now we restart Gunicorn, but this time telling it to listen on a socket instead of on the default port:

elspeth@server:$ sudo systemctl reload nginx
elspeth@server:$ ../virtualenv/bin/gunicorn --bind \
    unix:/tmp/superlists-staging.ottg.eu.socket superlists.wsgi:application

And again, we rerun the functional test again, to make sure things still pass:

$ STAGING_SERVER=superlists-staging.ottg.eu python manage.py test functional_tests
[...]
OK

A couple more steps!

Switching DEBUG to False and Setting ALLOWED_HOSTS

Django’s DEBUG mode is all very well for hacking about on your own server, but leaving those pages full of tracebacks available isn’t secure.

You’ll find the DEBUG setting at the top of 'settings.py'. When we set this to False, we also need to set another setting called ALLOWED_HOSTS. This was added as a security feature in Django 1.5. Unfortunately it doesn’t have a helpful comment in the default 'settings.py', but we can add one ourselves. Do this on the server:

server: superlists/settings.py
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = False

TEMPLATE_DEBUG = DEBUG

# Needed when DEBUG=False
ALLOWED_HOSTS = ['superlists-staging.ottg.eu']
[...]

And, once again, we restart Gunicorn and run the FT to check things still work.

Note
 Don’t commit these changes on the server. At the moment this is just a hack to get things working, not a change we want to keep in our repo. In general, to keep things simple, I’m only going to do Git commits from the local PC, using git push and git pull when I need to sync them up to the server.

One more test run to reassure ourselves that things still work?

$ STAGING_SERVER=superlists-staging.ottg.eu python manage.py test functional_tests
[...]
OK

Good.

Using Systemd to Make Sure Gunicorn Starts on Boot

Our final step is to make sure that the server starts up Gunicorn automatically on boot, and reloads it automatically if it crashes. On Ubuntu, the way to do this is using Systemd:

server: /etc/systemd/system/gunicorn-superlists-staging.ottg.eu.service
[Unit]
Description=Gunicorn server for superlists-staging.ottg.eu

[Service]
Restart=on-failure  (1)
User=elspeth  (2)
WorkingDirectory=/home/elspeth/sites/superlists-staging.ottg.eu/source  (3)
ExecStart=/home/elspeth/sites/superlists-staging.ottg.eu/virtualenv/bin/gunicorn \
    --bind unix:/tmp/superlists-staging.ottg.eu.socket \
    superlists.wsgi:application  (4)

[Install]
WantedBy=multi-user.target (5)

Systemd is joyously simple to configure (especially if you’ve ever had the dubious pleasure of writing an init.d script), and is fairly self-explanatory.

  1. Restart=on-failure will restart the process automatically if it crashes.

  2. User=elspeth makes the process run as the "elspeth" user.

  3. WorkingDirectory sets the current working directory.

  4. ExecStart is the actual process to execute. We use the `\ ` line continuation characters to split the full command over multiple lines, for readability, but it could all go on one line.

  5. WantedBy in the [Install] section is what tells Systemd we want this service to start on boot.

Systemd scripts live in '/etc/systemd/system', and their names must end in '.service'.

Now we tell Systemd to start Gunicorn with the systemctl command:

# this command is necessary to tell Systemd to load our new config file
elspeth@server:$ sudo systemctl daemon-reload
# this command tells Systemd to always load our service on boot
elspeth@server:$ sudo systemctl enable gunicorn-superlists-staging.ottg.eu
# this command actually starts our service
elspeth@server:$ sudo systemctl start gunicorn-superlists-staging.ottg.eu

(you should find the systemctl command responds to tab-completion, including of the service name, by the way)

Now we can rerun the FTs to see that everything still works. You can even test that the site comes back up if you reboot the server!

More Debugging Tips

  • Check the Systemd logs for using sudo journalctl -u gunicorn-superlists-staging.ottg.eu

  • You can ask Systemd to check the validity of your service configuration: systemd-analyze verify /path/to/my.service

  • Remember to restart both services whenever you make changes.

  • If you make changes to the Systemd config file, you need to run daemon-reload before systemctl restart to see the effect of your changes.

Saving Our Changes: Adding Gunicorn to Our requirements.txt

Back in the 'local' copy of your repo, we should add Gunicorn to the list of packages we need in our virtualenvs:

$ pip install gunicorn
$ pip freeze | grep gunicorn >> requirements.txt
$ git commit -am "Add gunicorn to virtualenv requirements"
$ git push

Note
 On Windows, at the time of writing, Gunicorn would pip install quite happily, but it wouldn’t actually work if you tried to use it. Thankfully we only ever run it on the server, so that’s not a problem. And, Windows support is being discussed…​

Thinking about Automating

Let’s recap our provisioning and deployment procedures:

Provisioning

  1. Assume we have a user account and home folder

  2. add-apt-repository ppa:fkrull/deadsnakes

  3. apt-get install nginx git python3.6 python3.6-venv

  4. Add Nginx config for virtual host

  5. Add Systemd job for Gunicorn

Deployment

  1. Create directory structure in '~/sites'

  2. Pull down source code into folder named 'source'

  3. Start virtualenv in '../virtualenv'

  4. pip install -r requirements.txt

  5. manage.py migrate for database

  6. collectstatic for static files

  7. Set DEBUG = False and ALLOWED_HOSTS in 'settings.py'

  8. Restart Gunicorn job

  9. Run FTs to check everything works

Assuming we’re not ready to entirely automate our provisioning process, how should we save the results of our investigation so far? I would say that the Nginx and Systemd config files should probably be saved somewhere, in a way that makes it easy to reuse them later. Let’s save them in a new subfolder in our repo:

Saving templates for our provisioning config files

$ mkdir deploy_tools
deploy_tools/nginx.template.conf
server {
    listen 80;
    server_name SITENAME;

    location /static {
        alias /home/elspeth/sites/SITENAME/static;
    }

    location / {
        proxy_set_header Host $host;
        proxy_pass http://unix:/tmp/SITENAME.socket;
    }
}
deploy_tools/gunicorn-systemd.template.service
[Unit]
Description=Gunicorn server for SITENAME

[Service]
Restart=on-failure
User=elspeth
WorkingDirectory=/home/elspeth/sites/SITENAME/source
ExecStart=/home/elspeth/sites/SITENAME/virtualenv/bin/gunicorn \
    --bind unix:/tmp/SITENAME.socket \
    superlists.wsgi:application

[Install]
WantedBy=multi-user.target

Then it’s easy for us to use those two files to generate a new site, by doing a find & replace on SITENAME.

For the rest, just keeping a few notes is OK. Why not keep them in a file in the repo too?

deploy_tools/provisioning_notes.md
Provisioning a new site
=======================

## Required packages:

* nginx
* Python 3.6
* virtualenv + pip
* Git

eg, on Ubuntu:

    sudo add-apt-repository ppa:fkrull/deadsnakes
    sudo apt-get install nginx git python36 python3.6-venv

## Nginx Virtual Host config

* see nginx.template.conf
* replace SITENAME with, e.g., staging.my-domain.com

## Systemd service

* see gunicorn-systemd.template.service
* replace SITENAME with, e.g., staging.my-domain.com

## Folder structure:
Assume we have a user account at /home/username

/home/username
└── sites
    └── SITENAME
         ├── database
         ├── source
         ├── static
         └── virtualenv

We can do a commit for those:

$ git add deploy_tools
$ git status # see three new files
$ git commit -m "Notes and template config files for provisioning"

Our source tree will now look something like this:

.
├── deploy_tools
│   ├── gunicorn-systemd.template.service
│   ├── nginx.template.conf
│   └── provisioning_notes.md
├── functional_tests
│   ├── [...]
├── lists
│   ├── __init__.py
│   ├── models.py
│   ├── [...]
│   ├── static
│   │   ├── base.css
│   │   └── bootstrap
│   │       ├── [...]
│   ├── templates
│   │   ├── base.html
│   │   ├── [...]
│   ├── tests.py
│   ├── urls.py
│   └── views.py
├── manage.py
├── requirements.txt
└── superlists
    ├── [...]

Saving our Progress

Being able to run our FTs against a staging server can be very reassuring. But, in most cases, you don’t want to run your FTs against your "real" server. In order to "save our work", and reassure ourselves that the production server will work just as well as the real server, we need to make our deployment process repeatable.

Automation is the answer, and it’s the topic of the next chapter.

Production-readiness for server deployments

A few things to think about when trying to build a production-ready server environment.

Don’t use the Django dev server in production

Something like Gunicorn or uWSGI is a better tool for running Django; they will let you run multiple workers, for example.

Don’t use Django to serve your static files

There’s no point in using a Python process to do the simple job of serving static files. Nginx can do it, but so can other web servers like Apache or uWSGI.

Check your settings.py for dev-only settings

DEBUG=True and ALLOWED_HOSTS are the two we looked at, but you will probably have others (we’ll see more when we start to send emails from the server).

Security

A serious discussion of server security is beyond the scope of this book, and I’d warn against running your own servers without learning a good bit more about it. (One reason people choose to use a PaaS to host their code, because that means there’s a few less security issues to worry about). If you’d like a place to start, here’s as good a place as any: My first 5 minutes on a server. I can definitely recommend the eye-opening experience of installing fail2ban and watching its logfiles to see just how quickly it picks up on random drive-by attempts to brute force your SSH login. The Internet is a dangerous place!