puts, print and p in ruby

I’ve been trying to pick up Ruby and one of the things I was asking myself was the difference in the various ways you can output text.

  • puts: displays text and adds a newline character
  • print: displays text and doesn’t add a newline character (this means you must add newlines manually when using print)
  • p: while it’s similar to puts in that it adds a newline, it calls inspect instead of to_s This is why p can be more useful for debugging since it shows what class an object belongs to.

Screen Shot 2016-02-08 at 19.02.38

More references for those interested

 

Advertisements

Developing on OS X – Part 1

This multipart series will take you through how to setup a Python development environment on OS X Yosemite & El Capitan.  We will setup Python…and the various helpers I use for Django development.  We’ll also install a few other applications I make use of in my day-to-day coding activities.  For this first part, we’ll go through settings up the various helper applications we’ll end up using for our Python setup

Enabling Writing to NTFS

Install Homebrew and Homebrew Cask

For anyone new to OS X, homebrew is a package manager, and it enables you to install command-line applications and libraries that would require manual compilation.  While Homebrew Cask extends Homebrew.

To install homebrew, run this command on your terminal:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

To get Homebrew Cask run this command on your terminal:

brew tap caskroom/cask

We’re now ready to install osxfuse…El Capitan users will want to install the 3.x.x version (3.1.0 or higher would be best) to avoid System integrity Protection issues https://github.com/osxfuse/osxfuse/releases.  For Yosemite, brew will suffice

brew cask install osxfuse

The final step to give you write-access to NTFS will be to install ntfs3g.

brew install homebrew/fuse/ntfs-3g

If using El Capitan, System Integrity Protection’s fs role will first need to be disabled.  This is needed to enable writing to /sbin

  • reboot and hold CMD+R to go into recovery mode
  • Run this on a terminal: $ csrutil disable
  • reboot normally

Create a symlink to mount NTFS

sudo mv /sbin/mount_ntfs /sbin/mount_ntfs.original
sudo ln -s /usr/local/sbin/mount_ntfs /sbin/mount_ntfs

Remember to re-enable System Integrity Protection (and re-lock the system directories) if you’d previously disabled it (same steps as before), only run $ csrutil enable

After that you should now be able to read and write to your NTFS drives.

Xcode

Xcode is an integrated development environment (IDE) containing a suite of software development tools developed by Apple for developing software for OS X and iOS.  While you don’t need the entire suite, you’ll want to install the command line tools.  Run this command then follow the instructions:

xcode-select --install

Terminal

I always like having a good terminal at my disposal…iterm2 is my custom terminal of choice on OS X.  To get iTerm2, download and install it as a normal OS X pkg file

Version Control

It’s always recommended to have a VCS in place when you’re writing your code….this way you can always revert to a previously working version of your code if you break something along the line.  It’s a matter of personal preference which VCS you use, but I like git.  And for anyone new to VCS, this handy guide explains it all.

To install git:

brew install git

 

Next step is to configure git for your setup.  If you haven’t configured your SSH keys, that would be the first thing.  If you get stuck, this Github link will help


ssh-keygen -t rsa -C "<your-email-address>"
git config --global user.name "<your-name>"
git config --global user.email "<your-email-address>"

 

After this add your key to your Github account (same process for Bitbucket or any other code hosting site where you can upload your SSH keys).

PRO TIPS:

  • You can never be too careful with your code so it’s always a good idea to enable two-factor authentication whenever possible.
  • For Android users, if you change phones frequently, instead of the Google Authenticator app, you can check out Authy…it’s slightly better since it has a clud backup feature, and if you’re a Chrome user, you can even register your browser as an MFA device

That’s it for the basics…in the next blog post we’ll get down and dirty with getting Python running

Setup Django with Supervisor, Gunicorn, and Nginx Part 2

In our previous post, we’d looked at the basic steps setting up a Django site.  This final part will show you how to expose the site to the public, and enable others to access your Django powered system/website.

Gunicorn

For a public site, you really don’t want to use Django’s inbuilt server since it’s not built to handle such traffic, only development testing.  For this, we’ll be using gunicorn server.  Install it in your application’s virtualenv

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ pip install gunicorn
Downloading/unpacking gunicorn
Downloading gunicorn-19.3.0-py2.py3-none-any.whl (110kB): 110kB downloaded
Installing collected packages: gunicorn
Successfully installed gunicorn
Cleaning up…

You can now do a basic test to ensure gunicorn serves your application

(djangosms)osboxes@osboxes:~/projects/active/django_project$ gunicorn django_project.wsgi:application –bind 0.0.0.0:8002
[2015-04-19 15:37:14 +0000] [3430] [INFO] Starting gunicorn 19.3.0
[2015-04-19 15:37:14 +0000] [3430] [INFO] Listening at: http://0.0.0.0:8002 (3430)
[2015-04-19 15:37:14 +0000] [3430] [INFO] Using worker: sync
[2015-04-19 15:37:14 +0000] [3435] [INFO] Booting worker with pid: 3435

Navigating to http://127.0.0.1:8002 should reward you with the default Django splash page.  But that’s not all…we’ll need to beef up this stack if it’s to serve traffic to millions of users.  Since Gunicorn is not installed and ready to serve your users, we can automate the startup process using supervisor.  This way, you don’t always have to login to your server on every reboot to start your Gunicorn server process.  For this, we’ll combine a simple shell script with supervisor.

First, we’ll create the shell script, which we’ll save at the directory root as gunicorn.sh:

#!/bin/bash
NAME=”djangotut” # Name of the application
DJANGODIR=/home/osboxes/projects/active/django_project # Django project directory
SOCKFILE=/home/osboxes/projects/active/django_project/gunicorn.sock # we will communicte using this unix socket

USER=osboxes # the user to run as
GROUP=osboxes # the group to run as
NUM_WORKERS=3 # how many worker processes should Gunicorn spawn

MAX_REQUESTS=1 # reload the application server for each request
DJANGO_SETTINGS_MODULE=django_project.settings # which settings file should Django use
DJANGO_WSGI_MODULE=django_project.wsgi # WSGI module name

echo “Starting $NAME as `whoami`”

# Activate the virtual environment
cd $DJANGODIR
source ~/.virtualenvs/django-tutorial-env/bin/activate

export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH

# Create the run directory if it doesn’t exist
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR

# Start your Django Unicorn

# Programs meant to be run under supervisor should not daemonize themselves (do not use –daemon)
exec ~/.virtualenvs/django-tutorial-env/bin/gunicorn ${DJANGO_WSGI_MODULE}:application \
–name $NAME \
–workers $NUM_WORKERS \
–max-requests $MAX_REQUESTS \
–user=$USER –group=$GROUP \
–bind=0.0.0.0:3000 \
–log-level=error \
–log-file=-

Make this script executable

sudo chmod u+x gunicorn.sh

You can test this script (make sure you run it as the user specified in USER)

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ ./gunicorn.sh
Starting djangotut as osboxes

As before, navigating to http://127.0.0.1:300 will load the same page as the previous test.  Modify the parameters in the script to fit your project setup.  Take note to:

  1. set the –workers (NUM_WORKERS) argument according to the following formula: 2 * CPUs + 1.
  2. set the –name (NAME) argument according to how your application will identify itself in programs such as top or ps. This makes it easy to distinguish it from others if you have multiple Gunicorn apps on the same server since it defaults to ‘gunicorn’
  3. install setproctitle to make the above setting (#2) work.  To build it, pip needs to have access to Python’s C header files.  You can install them by running sudo apt-get install python-dev then install setproctitle from within your virtualenv via pip.

Suoervisor

sudo apt-get install supervisor

After installing supervisor, we need to tell it the programs to watch and start by creating configuration files in /etc/supervisor/conf.d.  We’ll create ours as /etc/supervisor/conf.d/djangotut.conf

[program:djangotut]
command = /home/osboxes/projects/active/django_project/gunicorn.sh ; Command to start app
user = osboxes ; User to run as
stdout_logfile = /home/osboxes/projects/active/django_project/logs/supervisor.log ; Where to write log messages
redirect_stderr = true ; Save stderr in the same log
environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8 ; Set UTF-8 as default encoding

Note that we specified a directory within the project’s folder where we’ll store supervisor logs for our app.  It doesn’t exist, so it needs to be created

mkdir /home/osboxes/projects/active/django_project/logs

After saving the configuration file, we’ll then have supervisor reread configuration files and update (subsequently starting the newly registered app).

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ sudo supervisorctl reread
djangotut: available
(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ sudo supervisorctl update
djangotut: added process group

Our application will now always run on system reboot, and if it crashes, it’ll also be restarted.

Nginx

Right now, users can access our simple site at port 3000.  However, static files (images, javascript and stylesheets) won’t be served.  To do this, we’ll install nginx

sudo apt-get install nginx
sudo service nginx restart

We’ll now proceed to create an Nginx virtual server configuration for our app.  This will be under /etc/nginx/sites-available/djangotut, then we’ll create a symbolic link to /etc/nginx/sites-enabled.

upstream app_server {
# fail_timeout=0 means we always retry an upstream even if it failed
# to return a good HTTP response (in case the Unicorn master nukes a
# single worker for timing out).
server unix:/home/osboxes/projects/active/django_project/gunicorn.sock fail_timeout=0;
}

server {
listen 80;
server_name 0.0.0.0;
client_max_body_size 4G;
access_log /home/osboxes/projects/active/django_project/logs/nginx-access.log;
error_log /home/osboxes/projects/active/django_project/logs/nginx-error.log;
location /static/ {
alias /home/osboxes/projects/active/django_project/static/;
}

location /media/ {
alias /home/osboxes/projects/active/django_project/media/;
}

location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
# Try to serve static files from nginx, no point in making an
# *application* server like Unicorn/Rainbows! serve static files.
if (!-f $request_filename) {
proxy_pass http://app_server;
break;
}
}
}

Create the symbolic link to the sites-enabled folder

sudo ln -s /etc/nginx/sites-available/djangotut /etc/nginx/sites-enabled/djangotut

Restart nginx

sudo service nginx restart

And that’s all you’ll need to setup your Django application 🙂  As with any setup, I encountered a few issues (just two):

  1. 403 forbidden on accessing any of the media paths nginx serves (media and static).  To resolve this, specify a user at the top of your /etc/nginx/nginx.conf (above the server section).  The default is www-data, but if the user who owns those folders is different, then you’ll need to set this to the correct user (adding that user to the same group that www-data belongs to is also another solution)
  2. Anything that uses Pillow or PIL crashes with an error.  On Ubuntu, you need to:
    1. install libjpeg-dev with apt: sudo apt-get install libjpeg-dev
    2. reinstall pillow: pip install -I pillow
    3. if that doesn’t work:
      1. For Ubuntu x64:sudo ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib
        sudo ln -s /usr/lib/x86_64-linux-gnu/libfreetype.so /usr/lib
        sudo ln -s /usr/lib/x86_64-linux-gnu/libz.so /usr/lib

        Or for Ubuntu 32bit:

        sudo ln -s /usr/lib/i386-linux-gnu/libjpeg.so /usr/lib/
        sudo ln -s /usr/lib/i386-linux-gnu/libfreetype.so.6 /usr/lib/
        sudo ln -s /usr/lib/i386-linux-gnu/libz.so /usr/lib/

      2. Then reinstall pillow

Setup Django with Supervisor, Gunicorn, and Nginx Part 1

When I first began my long journey with developing web-based systems on Django, the recommended setup was centered around Apache and mod_wsgi.  That deployment setup has however advanced gracefully and evolved into more efficient, resilient and complex process involving supervisor, gunicorn and nginx.  We’ll be going through this process in this two part. This first part will show you setting up Django and your Database

What You’ll Need

A server with root access (I’m using Ubuntu 14.10).  If you’re on Windows, you can follow along if you run your server inside Virtual Box (or VMware).  For those on RPM-based distros like CentOS, replace apt-get with the yum equivalent, and those on BSD-like distros, replace the apt-get step with the equivalent from the ports tree.  Alternatively, get an inexpensive VPS server such as the fine setups available at Digital Ocean

Preparation

Make sure your system is up to date

sudo apt-get update -y
sudo apt-get upgrade -y

Database Setup: PostgreSQL

Install PostgreSQL

sudo apt-get install postgresql postgresql-contrib

Create a database user and a new database for the app

osboxes@osboxes:~$ sudo su postgres
[sudo] password for osboxes:
postgres@osboxes:/home/osboxes$ createuser --interactive -P
Enter name of role to add: djangodeploy
Enter password for new role:
Enter it again:
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
postgres@osboxes:/home/osboxes$ createdb --owner djangodeploy tutorial
postgres@osboxes:/home/osboxes$ exit
exit

Create a virtualenv for your app

virtualenv is a tool to create isolated Python environments.  It’s handy when you want to run applications with different (e.g. one needs Django 1.6, while another needs Django 1.8). or if you can’t install packages into the global site-packages directory (on a shared host).  We’ll also go ahead and install the helpful virtualenvwrapper extension, which makes managing multiple virtualenvs a breeze.

We’ll be installing these from pip.  If you don’t have pip installed, you’ll also need to install it prior to continuing

sudo apt-get install python-pip -y

Install virtualenv

pip install --user virtualenv

Now install virtualenvwrapper

pip install --user virtualenvwrapper

Just to show you what I have so far on my side:

osboxes@osboxes:~$ pip show virtualenv
---
Name: virtualenvwrapper
Version: 1.11.6
Location: /home/osboxes/.local/lib/python2.7/site-packages
Requires:
osboxes@osboxes:~$ pip show virtualenvwrapper
---
Name: virtualenvwrapper
Version: 4.4.1
Location: /home/osboxes/.local/lib/python2.7/site-packages
Requires: stevedore, virtualenv-clone, virtualenv

Finally, we add some information to our ~/.bashrc file (depends on your default shell, mine is bash). As usual for these things, they go to the end of the file. The actual contents for you will be different.  Here’s how my setup will be:

  • virtual environments will go to ~/.virtualenvs
  • active projects will be saved in ~/projects/active
  • because I installed as a user, the path to my virtualenvwrapper.sh is in ~/.local/bin/

# where to store our virtual envs
export WORKON_HOME=$HOME/.virtualenvs
# where projects will reside
export PROJECT_HOME=$HOME/projects/active
# where is the virtualenvwrapper.sh
source $HOME/.local/bin/virtualenvwrapper.sh

Save these changes and make them active

source ~/.bashrc

Create and activate the virtualenv

osboxes@osboxes:~$ mkvirtualenv django-tutorial-env
Running virtualenv with interpreter /usr/bin/python2
New python executable in django-tutorial-env/bin/python2
Also creating executable in django-tutorial-env/bin/python
Installing setuptools, pip...done.
(django-tutorial-env)osboxes@osboxes:~$

Start your django project

(django-tutorial-env)osboxes@osboxes:~$ pip install django
Downloading/unpacking django
Downloading Django-1.8-py2.py3-none-any.whl (6.2MB): 6.2MB downloaded
Installing collected packages: django
Successfully installed django
Cleaning up...
(django-tutorial-env)osboxes@osboxes:~$ cd ~/projects/active
(django-tutorial-env)osboxes@osboxes:~/projects/active$ django-admin.py startproject django_project
(django-tutorial-env)osboxes@osboxes:~/projects/active$ ls
django_project
(django-tutorial-env)osboxes@osboxes:~/projects/active$

One of the things I usually do at this point to make life easier is to make manage.py executable, so you can just type ./manage.py <command> rather than needing to type python manage.py <command> all the time

(django-tutorial-env)osboxes@osboxes:~/projects/active$ cd django_project
(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ chmod +x manage.py

At this pont, you can test the development server

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ ./manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).

You have unapplied migrations; your app may not work properly until they are applied.
Run 'python manage.py migrate' to apply them.

April 12, 2015 - 16:17:07
Django version 1.8, using settings 'django_project.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

You should now be able to access your development server from http://127.0.0.1:8000/

Configure PostgreSQL to work with Django

Install the psycopg2 database adapter

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ pip install psycopg2

You can now configure the databases section in your settings.py

DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'tutorial',
'USER': 'djangodeploy',
'PASSWORD': '1sbL?5oGMx@P1@',
'HOST': '',
'PORT': '',
}
}

Initialise the database

(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$ python manage.py syncdb
/home/osboxes/.virtualenvs/django-tutorial-env/local/lib/python2.7/site-packages/django/core/management/commands/syncdb.py:24: RemovedInDjango19Warning: The syncdb command will be removed in Django 1.9
warnings.warn("The syncdb command will be removed in Django 1.9", RemovedInDjango19Warning)

Operations to perform:
Synchronize unmigrated apps: staticfiles, messages
Apply all migrations: admin, contenttypes, auth, sessions
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying sessions.0001_initial... OK

You have installed Django's auth system, and don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'osboxes'):
Email address: admin@localhost.com
Password:
Password (again):
Superuser created successfully.
(django-tutorial-env)osboxes@osboxes:~/projects/active/django_project$

And that leaves you with a basic Django website. If you start the server once more, you can access your app at the same URL.  In the second part, we’ll look at how to install Gunicorn, Nginx and supervisor to tie this all togetehr into one production ready setup.

Add text to beginning of a file in bash

If you’ve ever worked with bash or the Linux command line, you already know you can add text at the end of a file using the >> operator.  I however found myself in need of adding text to the beginning of the file, and started to wonder what the opposite of >> was (definitely not << 🙂 )

Due to lack of a prepend operator, you can get creative with sed, or even combine echo and cat

  • sed -i “1itext to prepend” myfile.txt:  -i will update the contents of the file and automatically save it, while 1i will insert to the 1st line of the file.
  • echo -e “text to prepend\n$(cat myfile.txt)” > myfile.txt: the command inside of $(…) is executed before the file is overwritten with >myfile.txt.  The -e is to ensure any backslashes in myfile.txt are interpreted…otherwise the command will crash and burn.

Of course given the flexibility of some of the Linux tools at our disposal, other solutions are definitely possible.  Personally, I went with sed, but if you use echo and cat regularly, the second format might be more easier to remember

 

Notepadqq – great Notepad++ alternative on Ubuntu

I’ve recently had to switch by dev setup from Windows+VMWare to working from Ubuntu since I needed to use Vagrant+Ansible, and that dev setup never plays well with Windows.  This meant having to go through a new setup and look for alternatives to most of my favourite Windows tools.  One of these was Notepad++ I could of course use Wine and emulate Windows, but I felt there was an existing package that can run on Ubuntu without going through Wine.  After a brief search on Google, I came across Notepadqq.  It’s an opensource (free as in speech) editor for the Linus desktop that has a Notepad++ look and feel.

Screenshot from 2014-11-11 13:09:32

Some of the features include:

  • Syntax highlighting for about 100 different languages.
  • Code folding
  • Colour schemes
  • File monitoring
  • Multiple selection
  • Search text using the power of regular expressions
  • Organize documents side by side
  • Real-time highlighting to find near identifiers in no time.

To install the latest version (0.39.0 at the time of writing), you need to follow these steps:

Install Notepadqq 0.39.0 on Ubuntu 14.04 & Ubuntu 14.10
sudo add-apt-repository ppa:notepadqq-team/notepadqq
sudo apt-get update
sudo apt-get install notepadqq

Install Notepadqq 0.39.0 on Debian OS (change version number as per the desired one)
sudo apt-get install gdebi

For 32bit OS
wget "https://launchpad.net/~notepadqq-team/+archive/ubuntu/notepadqq/+files/notepadqq_0.39.0-1%7Etrusty1_i386.deb" -O notepadqq_0.39.0_i386.deb
sudo gdebi notepadqq_0.39.0_i386.deb

For 64bit OS
wget "https://launchpad.net/~notepadqq-team/+archive/ubuntu/notepadqq/+files/notepadqq_0.39.0-1%7Etrusty1_i386.deb" -O notepadqq_0.39.0_amd64.deb
sudo gdebi notepadqq_0.39.0_amd64.deb

After that, you should have your text editor ready for use 🙂

Filter by generic field

One of my favorite features in Django is the contenttypes framework.  Through this, you can easily create a generic model that can reference to any model within your project’s apps.  Let’s say my project has these models:

class Transaction(models.Model):
transaction_type = models.CharField(max_length=1, choices=CATEGORY_TRANSACTION_TYPES)
paid_by = models.CharField(max_length=2, choices=PAYMENT_METHOD, default='CH')
amount = models.DecimalField(max_digits=11, decimal_places=2, verbose_name='Amount')
notes = models.CharField(max_length=255, blank=True, verbose_name='Optional additional information about payment')
content_type = models.ForeignKey(ContentType, related_name='related_user_record')
object_id = models.PositiveIntegerField(blank=True)
content_object = generic.GenericForeignKey('content_type', 'object_id')
def __unicode__(self):
amount = "{0:,f}".format(self.amount)
return u"%s: %s (%s)" % (self.get_transaction_type_display(), amount, self.get_paid_by_display())

class Student(models.Model):
admission_number = models.CharField(max_length=15)
first_name = models.CharField(max_length=255)
middle_name = models.CharField(max_length=255, null=True, blank=True)
last_name = models.CharField(max_length=255)
profile_picture = ImageField(upload_to=settings.USER_AVATAR_PATH + '/students', null=True, blank=True)

One of the possible queries you might need based on these two classes is filtering the Transaction class by model instance to get all payments related to a student record.  Unfortunately, we can’t query directly by the content_object field like this:

transactions = Transaction.objects.filter(content_object=Student.objects.get(pk = 1))

Luckily, there’s a simple way to do this. There may be another way to do this, but this is the solution I came up with:

from django.contrib.contenttypes.models import ContentType
c_type= ContentType.objects.get(model=Student._meta.module_name)
Transaction.objects.filter(content_type=c_type, object_id=pk)

And finally to make all this DRY, I plugged it into the Student class as a property like so:

@property
def payments(self):
from ..finances.models import Transaction
from django.contrib.contenttypes.models import ContentType
content_type = ContentType.objects.get_for_model(self)
return Transaction.objects.filter(content_type=content_type, object_id=self.pk)

And that’s all we need to filter our records