Setup Django with Supervisor, Gunicorn, and Nginx Part 2

In our previous post, we’d looked at the basic steps setting up a Django site.  This final part will show you how to expose the site to the public, and enable others to access your Django powered system/website.

Gunicorn

For a public site, you really don’t want to use Django’s inbuilt server since it’s not built to handle such traffic, only development testing.  For this, we’ll be using gunicorn server.  Install it in your application’s virtualenv

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ pip install gunicorn
Downloading/unpacking gunicorn
Downloading gunicorn-19.3.0-py2.py3-none-any.whl (110kB): 110kB downloaded
Installing collected packages: gunicorn
Successfully installed gunicorn
Cleaning up…

You can now do a basic test to ensure gunicorn serves your application

(djangosms)osboxes@osboxes:~/projects/active/django_project$ gunicorn django_project.wsgi:application –bind 0.0.0.0:8002
[2015-04-19 15:37:14 +0000] [3430] [INFO] Starting gunicorn 19.3.0
[2015-04-19 15:37:14 +0000] [3430] [INFO] Listening at: http://0.0.0.0:8002 (3430)
[2015-04-19 15:37:14 +0000] [3430] [INFO] Using worker: sync
[2015-04-19 15:37:14 +0000] [3435] [INFO] Booting worker with pid: 3435

Navigating to http://127.0.0.1:8002 should reward you with the default Django splash page.  But that’s not all…we’ll need to beef up this stack if it’s to serve traffic to millions of users.  Since Gunicorn is not installed and ready to serve your users, we can automate the startup process using supervisor.  This way, you don’t always have to login to your server on every reboot to start your Gunicorn server process.  For this, we’ll combine a simple shell script with supervisor.

First, we’ll create the shell script, which we’ll save at the directory root as gunicorn.sh:

#!/bin/bash
NAME=”djangotut” # Name of the application
DJANGODIR=/home/osboxes/projects/active/django_project # Django project directory
SOCKFILE=/home/osboxes/projects/active/django_project/gunicorn.sock # we will communicte using this unix socket

USER=osboxes # the user to run as
GROUP=osboxes # the group to run as
NUM_WORKERS=3 # how many worker processes should Gunicorn spawn

MAX_REQUESTS=1 # reload the application server for each request
DJANGO_SETTINGS_MODULE=django_project.settings # which settings file should Django use
DJANGO_WSGI_MODULE=django_project.wsgi # WSGI module name

echo “Starting $NAME as `whoami`”

# Activate the virtual environment
cd $DJANGODIR
source ~/.virtualenvs/django-tutorial-env/bin/activate

export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH

# Create the run directory if it doesn’t exist
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR

# Start your Django Unicorn

# Programs meant to be run under supervisor should not daemonize themselves (do not use –daemon)
exec ~/.virtualenvs/django-tutorial-env/bin/gunicorn ${DJANGO_WSGI_MODULE}:application \
–name $NAME \
–workers $NUM_WORKERS \
–max-requests $MAX_REQUESTS \
–user=$USER –group=$GROUP \
–bind=0.0.0.0:3000 \
–log-level=error \
–log-file=-

Make this script executable

sudo chmod u+x gunicorn.sh

You can test this script (make sure you run it as the user specified in USER)

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ ./gunicorn.sh
Starting djangotut as osboxes

As before, navigating to http://127.0.0.1:300 will load the same page as the previous test.  Modify the parameters in the script to fit your project setup.  Take note to:

  1. set the –workers (NUM_WORKERS) argument according to the following formula: 2 * CPUs + 1.
  2. set the –name (NAME) argument according to how your application will identify itself in programs such as top or ps. This makes it easy to distinguish it from others if you have multiple Gunicorn apps on the same server since it defaults to ‘gunicorn’
  3. install setproctitle to make the above setting (#2) work.  To build it, pip needs to have access to Python’s C header files.  You can install them by running sudo apt-get install python-dev then install setproctitle from within your virtualenv via pip.

Suoervisor

sudo apt-get install supervisor

After installing supervisor, we need to tell it the programs to watch and start by creating configuration files in /etc/supervisor/conf.d.  We’ll create ours as /etc/supervisor/conf.d/djangotut.conf

[program:djangotut]
command = /home/osboxes/projects/active/django_project/gunicorn.sh ; Command to start app
user = osboxes ; User to run as
stdout_logfile = /home/osboxes/projects/active/django_project/logs/supervisor.log ; Where to write log messages
redirect_stderr = true ; Save stderr in the same log
environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8 ; Set UTF-8 as default encoding

Note that we specified a directory within the project’s folder where we’ll store supervisor logs for our app.  It doesn’t exist, so it needs to be created

mkdir /home/osboxes/projects/active/django_project/logs

After saving the configuration file, we’ll then have supervisor reread configuration files and update (subsequently starting the newly registered app).

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ sudo supervisorctl reread
djangotut: available
(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ sudo supervisorctl update
djangotut: added process group

Our application will now always run on system reboot, and if it crashes, it’ll also be restarted.

Nginx

Right now, users can access our simple site at port 3000.  However, static files (images, javascript and stylesheets) won’t be served.  To do this, we’ll install nginx

sudo apt-get install nginx
sudo service nginx restart

We’ll now proceed to create an Nginx virtual server configuration for our app.  This will be under /etc/nginx/sites-available/djangotut, then we’ll create a symbolic link to /etc/nginx/sites-enabled.

upstream app_server {
# fail_timeout=0 means we always retry an upstream even if it failed
# to return a good HTTP response (in case the Unicorn master nukes a
# single worker for timing out).
server unix:/home/osboxes/projects/active/django_project/gunicorn.sock fail_timeout=0;
}

server {
listen 80;
server_name 0.0.0.0;
client_max_body_size 4G;
access_log /home/osboxes/projects/active/django_project/logs/nginx-access.log;
error_log /home/osboxes/projects/active/django_project/logs/nginx-error.log;
location /static/ {
alias /home/osboxes/projects/active/django_project/static/;
}

location /media/ {
alias /home/osboxes/projects/active/django_project/media/;
}

location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
# Try to serve static files from nginx, no point in making an
# *application* server like Unicorn/Rainbows! serve static files.
if (!-f $request_filename) {
proxy_pass http://app_server;
break;
}
}
}

Create the symbolic link to the sites-enabled folder

sudo ln -s /etc/nginx/sites-available/djangotut /etc/nginx/sites-enabled/djangotut

Restart nginx

sudo service nginx restart

And that’s all you’ll need to setup your Django application 🙂  As with any setup, I encountered a few issues (just two):

  1. 403 forbidden on accessing any of the media paths nginx serves (media and static).  To resolve this, specify a user at the top of your /etc/nginx/nginx.conf (above the server section).  The default is www-data, but if the user who owns those folders is different, then you’ll need to set this to the correct user (adding that user to the same group that www-data belongs to is also another solution)
  2. Anything that uses Pillow or PIL crashes with an error.  On Ubuntu, you need to:
    1. install libjpeg-dev with apt: sudo apt-get install libjpeg-dev
    2. reinstall pillow: pip install -I pillow
    3. if that doesn’t work:
      1. For Ubuntu x64:sudo ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib
        sudo ln -s /usr/lib/x86_64-linux-gnu/libfreetype.so /usr/lib
        sudo ln -s /usr/lib/x86_64-linux-gnu/libz.so /usr/lib

        Or for Ubuntu 32bit:

        sudo ln -s /usr/lib/i386-linux-gnu/libjpeg.so /usr/lib/
        sudo ln -s /usr/lib/i386-linux-gnu/libfreetype.so.6 /usr/lib/
        sudo ln -s /usr/lib/i386-linux-gnu/libz.so /usr/lib/

      2. Then reinstall pillow
Advertisements

Setup Django with Supervisor, Gunicorn, and Nginx Part 1

When I first began my long journey with developing web-based systems on Django, the recommended setup was centered around Apache and mod_wsgi.  That deployment setup has however advanced gracefully and evolved into more efficient, resilient and complex process involving supervisor, gunicorn and nginx.  We’ll be going through this process in this two part. This first part will show you setting up Django and your Database

What You’ll Need

A server with root access (I’m using Ubuntu 14.10).  If you’re on Windows, you can follow along if you run your server inside Virtual Box (or VMware).  For those on RPM-based distros like CentOS, replace apt-get with the yum equivalent, and those on BSD-like distros, replace the apt-get step with the equivalent from the ports tree.  Alternatively, get an inexpensive VPS server such as the fine setups available at Digital Ocean

Preparation

Make sure your system is up to date

sudo apt-get update -y
sudo apt-get upgrade -y

Database Setup: PostgreSQL

Install PostgreSQL

sudo apt-get install postgresql postgresql-contrib

Create a database user and a new database for the app

osboxes@osboxes:~$ sudo su postgres
[sudo] password for osboxes:
postgres@osboxes:/home/osboxes$ createuser --interactive -P
Enter name of role to add: djangodeploy
Enter password for new role:
Enter it again:
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
postgres@osboxes:/home/osboxes$ createdb --owner djangodeploy tutorial
postgres@osboxes:/home/osboxes$ exit
exit

Create a virtualenv for your app

virtualenv is a tool to create isolated Python environments.  It’s handy when you want to run applications with different (e.g. one needs Django 1.6, while another needs Django 1.8). or if you can’t install packages into the global site-packages directory (on a shared host).  We’ll also go ahead and install the helpful virtualenvwrapper extension, which makes managing multiple virtualenvs a breeze.

We’ll be installing these from pip.  If you don’t have pip installed, you’ll also need to install it prior to continuing

sudo apt-get install python-pip -y

Install virtualenv

pip install --user virtualenv

Now install virtualenvwrapper

pip install --user virtualenvwrapper

Just to show you what I have so far on my side:

osboxes@osboxes:~$ pip show virtualenv
---
Name: virtualenvwrapper
Version: 1.11.6
Location: /home/osboxes/.local/lib/python2.7/site-packages
Requires:
osboxes@osboxes:~$ pip show virtualenvwrapper
---
Name: virtualenvwrapper
Version: 4.4.1
Location: /home/osboxes/.local/lib/python2.7/site-packages
Requires: stevedore, virtualenv-clone, virtualenv

Finally, we add some information to our ~/.bashrc file (depends on your default shell, mine is bash). As usual for these things, they go to the end of the file. The actual contents for you will be different.  Here’s how my setup will be:

  • virtual environments will go to ~/.virtualenvs
  • active projects will be saved in ~/projects/active
  • because I installed as a user, the path to my virtualenvwrapper.sh is in ~/.local/bin/

# where to store our virtual envs
export WORKON_HOME=$HOME/.virtualenvs
# where projects will reside
export PROJECT_HOME=$HOME/projects/active
# where is the virtualenvwrapper.sh
source $HOME/.local/bin/virtualenvwrapper.sh

Save these changes and make them active

source ~/.bashrc

Create and activate the virtualenv

osboxes@osboxes:~$ mkvirtualenv django-tutorial-env
Running virtualenv with interpreter /usr/bin/python2
New python executable in django-tutorial-env/bin/python2
Also creating executable in django-tutorial-env/bin/python
Installing setuptools, pip...done.
(django-tutorial-env)osboxes@osboxes:~$

Start your django project

(django-tutorial-env)osboxes@osboxes:~$ pip install django
Downloading/unpacking django
Downloading Django-1.8-py2.py3-none-any.whl (6.2MB): 6.2MB downloaded
Installing collected packages: django
Successfully installed django
Cleaning up...
(django-tutorial-env)osboxes@osboxes:~$ cd ~/projects/active
(django-tutorial-env)osboxes@osboxes:~/projects/active$ django-admin.py startproject django_project
(django-tutorial-env)osboxes@osboxes:~/projects/active$ ls
django_project
(django-tutorial-env)osboxes@osboxes:~/projects/active$

One of the things I usually do at this point to make life easier is to make manage.py executable, so you can just type ./manage.py <command> rather than needing to type python manage.py <command> all the time

(django-tutorial-env)osboxes@osboxes:~/projects/active$ cd django_project
(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ chmod +x manage.py

At this pont, you can test the development server

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ ./manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).

You have unapplied migrations; your app may not work properly until they are applied.
Run 'python manage.py migrate' to apply them.

April 12, 2015 - 16:17:07
Django version 1.8, using settings 'django_project.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

You should now be able to access your development server from http://127.0.0.1:8000/

Configure PostgreSQL to work with Django

Install the psycopg2 database adapter

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ pip install psycopg2

You can now configure the databases section in your settings.py

DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'tutorial',
'USER': 'djangodeploy',
'PASSWORD': '1sbL?5oGMx@P1@',
'HOST': '',
'PORT': '',
}
}

Initialise the database

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ python manage.py syncdb
/home/osboxes/.virtualenvs/django-tutorial-env/local/lib/python2.7/site-packages/django/core/management/commands/syncdb.py:24: RemovedInDjango19Warning: The syncdb command will be removed in Django 1.9
warnings.warn("The syncdb command will be removed in Django 1.9", RemovedInDjango19Warning)

Operations to perform:
Synchronize unmigrated apps: staticfiles, messages
Apply all migrations: admin, contenttypes, auth, sessions
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying sessions.0001_initial... OK

You have installed Django's auth system, and don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'osboxes'):
Email address: admin@localhost.com
Password:
Password (again):
Superuser created successfully.
(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$

And that leaves you with a basic Django website. If you start the server once more, you can access your app at the same URL.  In the second part, we’ll look at how to install Gunicorn, Nginx and supervisor to tie this all togetehr into one production ready setup.