Setup Django with Supervisor, Gunicorn, and Nginx Part 2

In our previous post, we’d looked at the basic steps setting up a Django site.  This final part will show you how to expose the site to the public, and enable others to access your Django powered system/website.

Gunicorn

For a public site, you really don’t want to use Django’s inbuilt server since it’s not built to handle such traffic, only development testing.  For this, we’ll be using gunicorn server.  Install it in your application’s virtualenv

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ pip install gunicorn
Downloading/unpacking gunicorn
Downloading gunicorn-19.3.0-py2.py3-none-any.whl (110kB): 110kB downloaded
Installing collected packages: gunicorn
Successfully installed gunicorn
Cleaning up…

You can now do a basic test to ensure gunicorn serves your application

(djangosms)osboxes@osboxes:~/projects/active/django_project$ gunicorn django_project.wsgi:application –bind 0.0.0.0:8002
[2015-04-19 15:37:14 +0000] [3430] [INFO] Starting gunicorn 19.3.0
[2015-04-19 15:37:14 +0000] [3430] [INFO] Listening at: http://0.0.0.0:8002 (3430)
[2015-04-19 15:37:14 +0000] [3430] [INFO] Using worker: sync
[2015-04-19 15:37:14 +0000] [3435] [INFO] Booting worker with pid: 3435

Navigating to http://127.0.0.1:8002 should reward you with the default Django splash page.  But that’s not all…we’ll need to beef up this stack if it’s to serve traffic to millions of users.  Since Gunicorn is not installed and ready to serve your users, we can automate the startup process using supervisor.  This way, you don’t always have to login to your server on every reboot to start your Gunicorn server process.  For this, we’ll combine a simple shell script with supervisor.

First, we’ll create the shell script, which we’ll save at the directory root as gunicorn.sh:

#!/bin/bash
NAME=”djangotut” # Name of the application
DJANGODIR=/home/osboxes/projects/active/django_project # Django project directory
SOCKFILE=/home/osboxes/projects/active/django_project/gunicorn.sock # we will communicte using this unix socket

USER=osboxes # the user to run as
GROUP=osboxes # the group to run as
NUM_WORKERS=3 # how many worker processes should Gunicorn spawn

MAX_REQUESTS=1 # reload the application server for each request
DJANGO_SETTINGS_MODULE=django_project.settings # which settings file should Django use
DJANGO_WSGI_MODULE=django_project.wsgi # WSGI module name

echo “Starting $NAME as `whoami`”

# Activate the virtual environment
cd $DJANGODIR
source ~/.virtualenvs/django-tutorial-env/bin/activate

export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH

# Create the run directory if it doesn’t exist
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR

# Start your Django Unicorn

# Programs meant to be run under supervisor should not daemonize themselves (do not use –daemon)
exec ~/.virtualenvs/django-tutorial-env/bin/gunicorn ${DJANGO_WSGI_MODULE}:application \
–name $NAME \
–workers $NUM_WORKERS \
–max-requests $MAX_REQUESTS \
–user=$USER –group=$GROUP \
–bind=0.0.0.0:3000 \
–log-level=error \
–log-file=-

Make this script executable

sudo chmod u+x gunicorn.sh

You can test this script (make sure you run it as the user specified in USER)

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ ./gunicorn.sh
Starting djangotut as osboxes

As before, navigating to http://127.0.0.1:300 will load the same page as the previous test.  Modify the parameters in the script to fit your project setup.  Take note to:

  1. set the –workers (NUM_WORKERS) argument according to the following formula: 2 * CPUs + 1.
  2. set the –name (NAME) argument according to how your application will identify itself in programs such as top or ps. This makes it easy to distinguish it from others if you have multiple Gunicorn apps on the same server since it defaults to ‘gunicorn’
  3. install setproctitle to make the above setting (#2) work.  To build it, pip needs to have access to Python’s C header files.  You can install them by running sudo apt-get install python-dev then install setproctitle from within your virtualenv via pip.

Suoervisor

sudo apt-get install supervisor

After installing supervisor, we need to tell it the programs to watch and start by creating configuration files in /etc/supervisor/conf.d.  We’ll create ours as /etc/supervisor/conf.d/djangotut.conf

[program:djangotut]
command = /home/osboxes/projects/active/django_project/gunicorn.sh ; Command to start app
user = osboxes ; User to run as
stdout_logfile = /home/osboxes/projects/active/django_project/logs/supervisor.log ; Where to write log messages
redirect_stderr = true ; Save stderr in the same log
environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8 ; Set UTF-8 as default encoding

Note that we specified a directory within the project’s folder where we’ll store supervisor logs for our app.  It doesn’t exist, so it needs to be created

mkdir /home/osboxes/projects/active/django_project/logs

After saving the configuration file, we’ll then have supervisor reread configuration files and update (subsequently starting the newly registered app).

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ sudo supervisorctl reread
djangotut: available
(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ sudo supervisorctl update
djangotut: added process group

Our application will now always run on system reboot, and if it crashes, it’ll also be restarted.

Nginx

Right now, users can access our simple site at port 3000.  However, static files (images, javascript and stylesheets) won’t be served.  To do this, we’ll install nginx

sudo apt-get install nginx
sudo service nginx restart

We’ll now proceed to create an Nginx virtual server configuration for our app.  This will be under /etc/nginx/sites-available/djangotut, then we’ll create a symbolic link to /etc/nginx/sites-enabled.

upstream app_server {
# fail_timeout=0 means we always retry an upstream even if it failed
# to return a good HTTP response (in case the Unicorn master nukes a
# single worker for timing out).
server unix:/home/osboxes/projects/active/django_project/gunicorn.sock fail_timeout=0;
}

server {
listen 80;
server_name 0.0.0.0;
client_max_body_size 4G;
access_log /home/osboxes/projects/active/django_project/logs/nginx-access.log;
error_log /home/osboxes/projects/active/django_project/logs/nginx-error.log;
location /static/ {
alias /home/osboxes/projects/active/django_project/static/;
}

location /media/ {
alias /home/osboxes/projects/active/django_project/media/;
}

location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
# Try to serve static files from nginx, no point in making an
# *application* server like Unicorn/Rainbows! serve static files.
if (!-f $request_filename) {
proxy_pass http://app_server;
break;
}
}
}

Create the symbolic link to the sites-enabled folder

sudo ln -s /etc/nginx/sites-available/djangotut /etc/nginx/sites-enabled/djangotut

Restart nginx

sudo service nginx restart

And that’s all you’ll need to setup your Django application 🙂  As with any setup, I encountered a few issues (just two):

  1. 403 forbidden on accessing any of the media paths nginx serves (media and static).  To resolve this, specify a user at the top of your /etc/nginx/nginx.conf (above the server section).  The default is www-data, but if the user who owns those folders is different, then you’ll need to set this to the correct user (adding that user to the same group that www-data belongs to is also another solution)
  2. Anything that uses Pillow or PIL crashes with an error.  On Ubuntu, you need to:
    1. install libjpeg-dev with apt: sudo apt-get install libjpeg-dev
    2. reinstall pillow: pip install -I pillow
    3. if that doesn’t work:
      1. For Ubuntu x64:sudo ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib
        sudo ln -s /usr/lib/x86_64-linux-gnu/libfreetype.so /usr/lib
        sudo ln -s /usr/lib/x86_64-linux-gnu/libz.so /usr/lib

        Or for Ubuntu 32bit:

        sudo ln -s /usr/lib/i386-linux-gnu/libjpeg.so /usr/lib/
        sudo ln -s /usr/lib/i386-linux-gnu/libfreetype.so.6 /usr/lib/
        sudo ln -s /usr/lib/i386-linux-gnu/libz.so /usr/lib/

      2. Then reinstall pillow
Advertisements

Setup Django with Supervisor, Gunicorn, and Nginx Part 1

When I first began my long journey with developing web-based systems on Django, the recommended setup was centered around Apache and mod_wsgi.  That deployment setup has however advanced gracefully and evolved into more efficient, resilient and complex process involving supervisor, gunicorn and nginx.  We’ll be going through this process in this two part. This first part will show you setting up Django and your Database

What You’ll Need

A server with root access (I’m using Ubuntu 14.10).  If you’re on Windows, you can follow along if you run your server inside Virtual Box (or VMware).  For those on RPM-based distros like CentOS, replace apt-get with the yum equivalent, and those on BSD-like distros, replace the apt-get step with the equivalent from the ports tree.  Alternatively, get an inexpensive VPS server such as the fine setups available at Digital Ocean

Preparation

Make sure your system is up to date

sudo apt-get update -y
sudo apt-get upgrade -y

Database Setup: PostgreSQL

Install PostgreSQL

sudo apt-get install postgresql postgresql-contrib

Create a database user and a new database for the app

osboxes@osboxes:~$ sudo su postgres
[sudo] password for osboxes:
postgres@osboxes:/home/osboxes$ createuser --interactive -P
Enter name of role to add: djangodeploy
Enter password for new role:
Enter it again:
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
postgres@osboxes:/home/osboxes$ createdb --owner djangodeploy tutorial
postgres@osboxes:/home/osboxes$ exit
exit

Create a virtualenv for your app

virtualenv is a tool to create isolated Python environments.  It’s handy when you want to run applications with different (e.g. one needs Django 1.6, while another needs Django 1.8). or if you can’t install packages into the global site-packages directory (on a shared host).  We’ll also go ahead and install the helpful virtualenvwrapper extension, which makes managing multiple virtualenvs a breeze.

We’ll be installing these from pip.  If you don’t have pip installed, you’ll also need to install it prior to continuing

sudo apt-get install python-pip -y

Install virtualenv

pip install --user virtualenv

Now install virtualenvwrapper

pip install --user virtualenvwrapper

Just to show you what I have so far on my side:

osboxes@osboxes:~$ pip show virtualenv
---
Name: virtualenvwrapper
Version: 1.11.6
Location: /home/osboxes/.local/lib/python2.7/site-packages
Requires:
osboxes@osboxes:~$ pip show virtualenvwrapper
---
Name: virtualenvwrapper
Version: 4.4.1
Location: /home/osboxes/.local/lib/python2.7/site-packages
Requires: stevedore, virtualenv-clone, virtualenv

Finally, we add some information to our ~/.bashrc file (depends on your default shell, mine is bash). As usual for these things, they go to the end of the file. The actual contents for you will be different.  Here’s how my setup will be:

  • virtual environments will go to ~/.virtualenvs
  • active projects will be saved in ~/projects/active
  • because I installed as a user, the path to my virtualenvwrapper.sh is in ~/.local/bin/

# where to store our virtual envs
export WORKON_HOME=$HOME/.virtualenvs
# where projects will reside
export PROJECT_HOME=$HOME/projects/active
# where is the virtualenvwrapper.sh
source $HOME/.local/bin/virtualenvwrapper.sh

Save these changes and make them active

source ~/.bashrc

Create and activate the virtualenv

osboxes@osboxes:~$ mkvirtualenv django-tutorial-env
Running virtualenv with interpreter /usr/bin/python2
New python executable in django-tutorial-env/bin/python2
Also creating executable in django-tutorial-env/bin/python
Installing setuptools, pip...done.
(django-tutorial-env)osboxes@osboxes:~$

Start your django project

(django-tutorial-env)osboxes@osboxes:~$ pip install django
Downloading/unpacking django
Downloading Django-1.8-py2.py3-none-any.whl (6.2MB): 6.2MB downloaded
Installing collected packages: django
Successfully installed django
Cleaning up...
(django-tutorial-env)osboxes@osboxes:~$ cd ~/projects/active
(django-tutorial-env)osboxes@osboxes:~/projects/active$ django-admin.py startproject django_project
(django-tutorial-env)osboxes@osboxes:~/projects/active$ ls
django_project
(django-tutorial-env)osboxes@osboxes:~/projects/active$

One of the things I usually do at this point to make life easier is to make manage.py executable, so you can just type ./manage.py <command> rather than needing to type python manage.py <command> all the time

(django-tutorial-env)osboxes@osboxes:~/projects/active$ cd django_project
(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ chmod +x manage.py

At this pont, you can test the development server

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ ./manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).

You have unapplied migrations; your app may not work properly until they are applied.
Run 'python manage.py migrate' to apply them.

April 12, 2015 - 16:17:07
Django version 1.8, using settings 'django_project.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

You should now be able to access your development server from http://127.0.0.1:8000/

Configure PostgreSQL to work with Django

Install the psycopg2 database adapter

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ pip install psycopg2

You can now configure the databases section in your settings.py

DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'tutorial',
'USER': 'djangodeploy',
'PASSWORD': '1sbL?5oGMx@P1@',
'HOST': '',
'PORT': '',
}
}

Initialise the database

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ python manage.py syncdb
/home/osboxes/.virtualenvs/django-tutorial-env/local/lib/python2.7/site-packages/django/core/management/commands/syncdb.py:24: RemovedInDjango19Warning: The syncdb command will be removed in Django 1.9
warnings.warn("The syncdb command will be removed in Django 1.9", RemovedInDjango19Warning)

Operations to perform:
Synchronize unmigrated apps: staticfiles, messages
Apply all migrations: admin, contenttypes, auth, sessions
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying sessions.0001_initial... OK

You have installed Django's auth system, and don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'osboxes'):
Email address: admin@localhost.com
Password:
Password (again):
Superuser created successfully.
(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$

And that leaves you with a basic Django website. If you start the server once more, you can access your app at the same URL.  In the second part, we’ll look at how to install Gunicorn, Nginx and supervisor to tie this all togetehr into one production ready setup.

Filter by generic field

One of my favorite features in Django is the contenttypes framework.  Through this, you can easily create a generic model that can reference to any model within your project’s apps.  Let’s say my project has these models:

class Transaction(models.Model):
transaction_type = models.CharField(max_length=1, choices=CATEGORY_TRANSACTION_TYPES)
paid_by = models.CharField(max_length=2, choices=PAYMENT_METHOD, default='CH')
amount = models.DecimalField(max_digits=11, decimal_places=2, verbose_name='Amount')
notes = models.CharField(max_length=255, blank=True, verbose_name='Optional additional information about payment')
content_type = models.ForeignKey(ContentType, related_name='related_user_record')
object_id = models.PositiveIntegerField(blank=True)
content_object = generic.GenericForeignKey('content_type', 'object_id')
def __unicode__(self):
amount = "{0:,f}".format(self.amount)
return u"%s: %s (%s)" % (self.get_transaction_type_display(), amount, self.get_paid_by_display())

class Student(models.Model):
admission_number = models.CharField(max_length=15)
first_name = models.CharField(max_length=255)
middle_name = models.CharField(max_length=255, null=True, blank=True)
last_name = models.CharField(max_length=255)
profile_picture = ImageField(upload_to=settings.USER_AVATAR_PATH + '/students', null=True, blank=True)

One of the possible queries you might need based on these two classes is filtering the Transaction class by model instance to get all payments related to a student record.  Unfortunately, we can’t query directly by the content_object field like this:

transactions = Transaction.objects.filter(content_object=Student.objects.get(pk = 1))

Luckily, there’s a simple way to do this. There may be another way to do this, but this is the solution I came up with:

from django.contrib.contenttypes.models import ContentType
c_type= ContentType.objects.get(model=Student._meta.module_name)
Transaction.objects.filter(content_type=c_type, object_id=pk)

And finally to make all this DRY, I plugged it into the Student class as a property like so:

@property
def payments(self):
from ..finances.models import Transaction
from django.contrib.contenttypes.models import ContentType
content_type = ContentType.objects.get_for_model(self)
return Transaction.objects.filter(content_type=content_type, object_id=self.pk)

And that’s all we need to filter our records

Python and QRCodes

Recently, I wanted to generate QR codes for a Django project I’ve been working on for a while.  As I always say, in Django, there’s a plugin for almost everything.  So to get things rolling, first you need to install the qrcode on PyPi Just run this:

pip install qrcode

You’ll also need PIL (I recommend you use Pillow though).  So once you have those two, time to roll up our sleeves and get down to the code.

First, import the library somewhere in your script:

import qrcode

Then instanciate the qrcode object

qr = qrcode.QRCode(
version=1,
error_correction=qrcode.constants.ERROR_CORRECT_L,
box_size=10,
border=4,
)

The first parameter, version, is of course the QR version.  It should be an integer between 0 and 40 which define the size of the barcode and the amount of data we’ll be able to store.  The second parameter, error_correction, is the redundancy level.  This can be:

  • ERROR_CORRECT_L: 7% of codewords can be restored
  • ERROR_CORRECT_M (default): 15% can be restored
  • ERROR_CORRECT_Q: 25% can be restored
  • ERROR_CORRECT_H: 30% can be restored

This basically ensures decoding even if the data is damaged. More info on these redunancy levels can be obtained from Wikipedia. The box_size parameter controls how many pixels each “box” of the resulting QR code is, while the border parameter controls how many boxes thick the border should be (the default is 4, which is the minimum according to the specs).

So once you have the initiliased object, you can add the data like so:

qr.add_data("This is the data")
qr.make(fit=True)
# im contains a PIL.Image.Image object
img = qr.make_image()

From there, depending on your backend, you need to save this image.  For Django I used the InMemoryUploadedFile class to convert this Image object into a File object that I could pass into the model class as a variable so that it goes into the usual File handling flow.  There could be another way to do this, but this worked for me:

from StringIO import StringIO
from django.core.files.uploadedfile import InMemoryUploadedFile
buffer = StringIO()
img.save(buffer, "PNG")
image_file = InMemoryUploadedFile(buffer,  None,  ("%s.png" % identifier),  "image/png",  buffer.len,  None)

Then finally, you save the QR code onto your model class object like normal:

myobject.qr_code.save(("%s.png" % identifier), image_file)

Remove Specific HTML Tags in Django

It seems every day I learn something new in Django.  Recently, I was working on a few customizations to a zinnia installation and wanted to get the latest 4 blog posts on the home page.  Now, in zinnia, this is very straightfoward:

{% load zinnia_tags %}

{% get_recent_entries 4 template="zinnia/tags/entries_recent.html" %}

The extra template variable is because I was using a custom layout to render the snippets.  My main issue was that some of the blog posts might have embedded iframes.  So I wanted to find a way to remove the iframe tags.  Luckily, it’s quite easy to do this in Django (versions 1.7 downwards…for the latest versions, use bleach):

At code level (this can then be extended into a custom templatetag)

from django.template.defaultfilters import removetags
html = '


stripped = removetags(html, 'iframe')

At the template level

{{ value|removetags:"iframe"|safe }}

And it’s as simple as that

Enabling iframe and other content in django-ckeditor-updated

One of the most used features in any Django project that allows page editing…while I have my favourite (django-summernote), I recently integrated ckeditor using the django-ckeditor-updated package from the cheeseshop.  It was an excellent choice and I was loving all the new features and extensibility.  However, I hit a stop when I wanted to insert HTML content from and ajax request I had being fired from the UI.  First of all, inserting just plain HTML is actually very easy:

CKEDITOR.instances["<id-of-ckeditor-textarea>"].insertHtml("<your-html-here>");

The fun all starts when you want to insert an iframe (or the more popular term, embedding content eg a Youtube video or for my case, a link to a Geonode map).  This is due to the Advanced Content Filter introduced in CKEditor 4.1.x. To start things off, you can check whether your editor will display iframes correctly by executing this simple JS code from your console:

CKEDITOR.instances.yourInstance.filter.check( 'iframe' );
>>> true // it's allowed

If the result is false, you can:

  • enable the mediaembed plugin in your editor instance: more info from the docs
  • extend config.extraAllowedContent to re-enable it

For the second solution, you need to add this code toy your editor’s config:

config.extraAllowedContent = 'iframe[*]'

or you can also just simply have it as:

CKEDITOR.config.allowedContent = true;

The beauty of this is you don’t have to enable the mediaembed plugin.  SO, that’s for the JS version of the plugin.  FOr Django users, this all goes into settings.py.  Mine looks something like this:


CKEDITOR_CONFIGS = {
    'default': {
        'toolbar': 'Full',
        'height': 300,
        'width': '100%',
        'removePlugins': 'stylesheetparser',
        'extraAllowedContent': 'iframe[*]',
    },
}

And with those simple changes, you’ll be able to insert iframe content without and problems

Setting related_name for django abstract base classes

In Django, ForeignKey and ManyToManyField fields have a corresponding backward relation is automatically created for you.  By default, this relation is given a name automatically, but if you want, you can specify your own using the related_name parameter like: ForeignKey (User, related_name='<name-here>') However, things get a bit different when you use abstract classes.  Not sure if other Django versions have the same behavior, but in 1.6.3, this code breaks things if more than one class inherits from the base abstract class:

class TimeStampedModel(models.Model):
    created = models.DateTimeField(auto_now_add=True)
    updated = models.DateTimeField(auto_now=True)

    class Meta:
        abstract = True

class TimeStampedAuthModel(TimeStampedModel):
    created_by = models.ForeignKey(User)
    updated_by = models.ForeignKey(User)

    class Meta:
        abstract = True

django_abstract_foreign_keys_1

The reason for this can be found in the ever helpful Django docs page

As explained there, you resolve  this as follows:

To work around this problem, when you are using related_name in an abstract base class (only), part of the name should contain '%(app_label)s' and'%(class)s'

So we patch up our declaration to be:

created_by = models.ForeignKey(User, related_name='%(app_label)s_%(class)s_created_by')
updated_by = models.ForeignKey(User, related_name='%(app_label)s_%(class)s_updated_by')

Also, as pointed out in this blog post, any name ending with a ‘+’ also works