Web GIS application development step by step Part 1

GeoNetwork2

Many people asked me about web GIS  development and from where to start. Some questions were basic like ‘how to show a shape file on Google maps’ to advance level as ’tiles are coming from local tile cache or GIS server’. Since I am into Open Source web GIS application development, thus, I decided to write a blog about it that can help the beginners and answer some medium to advance level questions.

First step is to setup a development environment. Keeping the development environment Open source and free from end to end (from Operating system to client side APIs), I selected Ubuntu as the operating system. I installed Ubuntu 12.04 and rest of the story is as follows

Package updates are required, thus, do the following.

sudo apt-get update
sudo apt-get upgrade

this will take some time, depends on your internet speed. I found synaptic package manager missing from my newly installed Ubuntu 12.04 (desktop), thus, I Installed synaptic package manager using commands

Note: Remember, answers to questions on console are case sensitive.

 sudo apt-get install synaptic

I like Google Chrome because it has an easy to use debugger, thus, installed Chrome using synaptic package installer. Keep in mind, chrome for Ubuntu 64 bit is only available for AMD by-default.

To keep a backup backup of some important files, I always use Dropbox. To install Dropbox, python-gpgme package needs to be installed (again using synaptic package installer) for verification of binary signature. Finally, I downloaded Dropbox installation package and installed it.

Behind every successful web application there is a database :). Not that we will be needing a database for spatial purposes (in this tutorial) but it is required for many other purposes in a web application e.g. keeping users details etc. Few choices from Open Source community are MySQL, SQLite and PostgreSQL. PostgreSQL is the best Open Source database especially when it comes to spatial functions. Again, I used synaptic package manager to install postgreSQL (as you can see in in the image below, I installed postgreSQL 9.1.x)

postgreSQL

 

At this moment, postgreSQL is installed but One has to create a role, user and database, thus, I executed following commands through terminal

sudo -u postgres createuser

following questions required user response

Enter name of role to add: scriptNdebug
Shall the new role be a superuser? (y/n) y
Shall the new role be allowed to create databases? (y/n) y
Shall the new role be allowed to create more new roles? (y/n) y
CREATE ROLE 

The above command will create a user, now for database

sudo -u postgres createdb myDB

response will be

CREATE DATABASE

Now user scriptNdebug needs to have access to myDB database, thus, logged in to command line tool of PostgreSQL

i.e. psql using super user postgres

sudo -u postgres psql 
postgres=# alter user scriptNdebug with encrypted password 'password';

response is

ALTER ROLE
postgres=# grant all privileges on database myDB to scriptNdebug;

response is

GRANT

What if I want to stop PostgreSQL database server. Following are some useful commands (use them from terminal)

Start

/etc/init.d/postgresql start

Stop

/etc/init.d/postgresql stop

Restart

/etc/init.d/postgresql restart

If you want server to listen on a specific IP, then change pg_hba_config file. If the file does not exists, then simply rename pg_hba_config.sample at location /usr/share/postgresql/9.1/ to pg_hba_config and make the following changes

TYPE    DATABASE   USER    ADDRESS            METHOD
host    all        all     192.168.0.1/32     @authmethod@

 the above change will make the server listen on IPs from 192.168.0.1 to 192.168.0.32. I want to list on all IPs for all users and ‘password’ as authentication method, thus, I made the following changes

TYPE    DATABASE   USER    ADDRESS METHOD
 host    all       all     all     password 

Similarly, if you want to listen on localhost, then make the following changes in file postgresql.config. If the file does not exists, then simply rename postgresql.config.sample at location /usr/share/postgresql/9.1/ to postgresql.config and make the following changes

listen_addresses = 'localhost'

This file is to be changed if one wants to change port 5432. Do not forget to restart PostgreSQL server load the new settings.

Now web server needs to be installed, I selected apache, used synaptic package manager to install apache (version 2.2.22-1ubuntu1.4). This will create a folder ‘www’ in folder /var/. We will use the www folder when we will create a test web GIS application later in this tutorial. If you want to use php as server side development language, install php module for apache server. Just use synaptic to install libapache2-mod-php5. Also install (using synaptic) php5-pgsql module to use postgreSQL with php. Restart apache (use command below) to load these modules.

sudo /etc/init.d/apache2 restart

Finally it is time to install the most important component of Web GIS application i.e. GIS Server. It is important to know what a GIS Server Is? It is a software that listens for specific request (by clients). These requests could be for different services  (read OGC specification on GIS server for details) e.g. getMap request etc. A GIS server loads a dataset (e.g. a shape file or a Geo-Tiff) renders it, cut image into tiles and sends it back to the requesting client. Every time a client pans or zooms, GIS server receives requests and sends image tiles as response. Format of image tiles is specifically selected due to its size. In Free and Open Source Software for Geo  (FOSS4Geo) there are multiple choices for GIS Servers e.g. MapServer, GeoServer, mapnik. qGIS server, MapGuide etc. Some are written in C and some are in Java.

GeoServer has an edge over others because of its administration tool i.e. web based administration interface. With GeoServer, deploying a dataset (let say a shape file) as a WMS is very easy. Thus I decided to install GeoServer. Before installing GeoServer, Java needs to be installed as GeoServer require Java. I used  synaptic package manager to install openjdk-6-jre.  Then downloaded GeoServer from its web site. Unzipped the downloaded file into a folder. Moved to bin folder and edited startup.sh file to add JAVA_HOME variable as shown below (your path might be different).

JAVA_HOME="/usr/lib/jvm/java-6-openjdk-i386"

All is set now, started GeoServer from terminal by using the following command

 sudo ./startup.sh

Note: Do not close this terminal this terminal window because it will shutdown GeoServer.

Started a web browser, and typed http://localhost/geoserver to open administration application and used admin as user name and geoserver as password. Application menu is on left side, clicked on Layer preview to view sample WMS layers. One can click on Open Layers link against any sample layer (as shown in figure below) to open it in a new tab.

geoserver

 

Please read GeoServer user manual to know how to deploy your datasets as WMS layers (I will create a WMS in next part of this tutorial). In next part, I will also create a web GIS sample application using Open Layers and explain web GIS application functionality.

Advertisements

Essential Linux commands, version 1.0

I have horrible memory when it comes to names and numbers, same is true for Linux commands. I use and then quickly forget useful commands or switches. I always promises my self to note down these commands but its not working. This blog entry (and the future versions) will serve as personal manual.

Package Information

Show installed packages (all)

dpkg -l

Search for a specific package e.g. gdal

dpkg -l | grep gdal

What if I want to know if  a package is installed or not.

dpkg-query -l 

for example, package gdal is installed or not (Note you have to provide full package name, its not like grep)

dpkg-query -l libgdal-dev

or just the -S switch

dpkg -S libgdal-dev

and if you want to know dependencies of a package, run the following command

apt-cache showpkg package_name

e.g. 

apt-cache showpkg libgdal-dev

Note: Not only the above command shows package dependencies for libgdal-dev package but will also show packages depending on libgdal-dev,see section Reverse Depends in the output.

Folders

List files, sorted by creation date and time

ls -ltrh

Sort by file size

ls -lhS

List hidden folders

ls -ld .?*

List folder with permissions

ls -ld */

Locate command is dependent on a DB, thus, it is necessary to run the following command from time to time (use sudo)

updatedb

Locate command is perfect but some times I want to find folder under a specific path

find /path -name "name" -type d

e.g.

find /var -name "apache2" -type d

-type d is for directory

and to find out folder size

du -hs

switch h is for human readable format and s is summarized. 

Create multiple folders with one command (the command below will create folder 1990, 1991….2010)

mkdir {1990..2010}

Delete folders (and files in them)

rm -rf foldername
or 
rmdir foldername

-f is for force (you will not be asked again and again for ‘yes’)

Files

File size using the following command

ls -sh

h is for human readable format.

Copy files from one location into other

cp -v file*.html /var/www/website

Note: Use -r switch to copy directories.

copy all files from one folder to an other

cp Folder1/* Folder2/

Move files

mv filename folder_name

Example:

 mv test.png /var/www/images

or for folder 

mv -r /home/user/folder /location/to/cop

Rename files

mv original.txt renamed.txt

sudo might be required

Cat command is one of Linux powerful commands. It sequentially concatenates multiple files and can be redirected into a file or pipe for an other command.

cat *.txt > merged-file.txt

or

cat *.txt > grep "some search string"

In a console, how to create files (no mouse to right click and select “create new file” 🙂 ). Use the touch command

touch newfile.txt

ls command can tell a lot about file and folders e.g. if file/folder permission information is require then use

ls -l

It show  user, group, date and time of creation and type of permission on a file and folder. How to understand permission information e.g

drwxrwxr-x Tony:stark ...............

Lets break it down

directory  owner   group  others
d          rwx     rwx    r-x 

it shows owner, group and others permission information e.g. the above item is a folder and owner “Tony” has read, write and execute permissions. user belong to group “stark” have read, write and execute on this item and all other users have read and execute permissions but they cannot write to this item.

What if a user wants to have permissions to read, write or execute. Here comes change mode command

chmod a=rwx file/foler 

e.g

chmod a=rwx Iron-man.txt

sudo is required.

If item is a folder and contains sub-folders and files, the recursive switch can help to set the same permissions for files/sub-folders inside the parent folder

chmod -R a=rwx file/folder

a=rwx means that all owner, group and others will have read, write and execution permission.

If just one type of user is to be given permissions then use the following versions

Give owner read and write permissions
chmod u+rw file/folder
Give group read and execute permissions
chmod g+rx file/folder
Give others read and execute permissions
chmod o+rw file/folders

or combine user and group in one command

chmod u+r,g+x file/folder

Similarly,  permission can be revoked using change mode command e.g. disallow others the permissions of read and write

chmod o-rw file/folder

Find a string in file (in specific folder)

grep -nr yourString* .

-r will enable grep to search in sub folders

Remove files

rm file

remove folder

rmdir foldername

Remove files and folders recursively

rm -rf foldername

 

Setting related_name for django abstract base classes

In Django, ForeignKey and ManyToManyField fields have a corresponding backward relation is automatically created for you.  By default, this relation is given a name automatically, but if you want, you can specify your own using the related_name parameter like: ForeignKey (User, related_name='<name-here>') However, things get a bit different when you use abstract classes.  Not sure if other Django versions have the same behavior, but in 1.6.3, this code breaks things if more than one class inherits from the base abstract class:

class TimeStampedModel(models.Model):
    created = models.DateTimeField(auto_now_add=True)
    updated = models.DateTimeField(auto_now=True)

    class Meta:
        abstract = True

class TimeStampedAuthModel(TimeStampedModel):
    created_by = models.ForeignKey(User)
    updated_by = models.ForeignKey(User)

    class Meta:
        abstract = True

django_abstract_foreign_keys_1

The reason for this can be found in the ever helpful Django docs page

As explained there, you resolve  this as follows:

To work around this problem, when you are using related_name in an abstract base class (only), part of the name should contain '%(app_label)s' and'%(class)s'

So we patch up our declaration to be:

created_by = models.ForeignKey(User, related_name='%(app_label)s_%(class)s_created_by')
updated_by = models.ForeignKey(User, related_name='%(app_label)s_%(class)s_updated_by')

Also, as pointed out in this blog post, any name ending with a ‘+’ also works

Zinnia Customisations Part II

In the first part of this series, we looked at how to create an app that will hold the customisations we want done to zinnia (customised pages for adding and editing blog posts).  Now, it’s time to wrap this app and tie the app into the rest of the site’s structure and enable users to start using the new logic.

To kick things off, you need two template files copied into your root templates folder (For my test project, I’ve also added the zinnia-theme-bootstrap helper theme to bootstrap the template files):

  • base.html : zinnia’s base template
  • entry_detail_base.html : renders blog post info

Note that if you do end up using the extra Bootstrap 3 theme, your extend statements will be a bit different, as well as the overall HTML markup, but the gist of the changes remains the same.  For example the default base.html template is:

{% extends "zinnia/skeleton.html" %}

But after adding the theme, is becomes:

{% extends "zinnia:zinnia/base.html" %}

Note the app namespace template loader syntax used in the extend statement.  This helper app is installed along with zinnia-theme-bootstrap and it enables you to both extend and override a template at the same time.  So essentially what that one line does is extend the default base.html template in zinnia’s template directory, and further override a specific block within that template, in this case the sidebar. Pretty cool stuff.  It’s an improvement over Django’s default template loaders that require you to copy the entire template you want to override, even if you only want to override one small block, and allows you to extend and override a specific block in the extended template.

So you’ll now want to start updating the templates you copied over into your project’s root template folder, starting with base.html.  Look for this line somewhere towards the end of the HTML markup:
<a href="{% url 'admin:zinnia_entry_add' %}" title="{% trans "Post an entry" %}">

Change it to:

<a href="{% url 'custom_add_blog_post' %}" title="{% trans "Post an entry" %}">

Then let’s move on to entry_detail_base.html.  Here, you’ll want to look for this line:

<a href="{% url 'admin:zinnia_entry_change' object.pk %}" title="{% trans "Edit the entry" %}">

Change it to:

<a href="{% url 'custom_edit_blog_post' object.pk %}" title="{% trans "Edit the entry" %}">

And finally, two last steps.  First your URL conf entries by add this in a non-conflicting location:

(r'^summernote/', include('django_summernote.urls')),
url(r'^', include('customisations.urls')),

Also, you’ll need to update settings.py:

First, add your new apps into INSTALLED_APPS:

  1. ‘chosen’,
  2. ‘django_summernote’,
  3. ‘customisations’,

And lastly, some settings needed by django-summernote (full reference here):

SUMMERNOTE_CONFIG = {
# Using SummernoteWidget - iframe mode
# or set False to use SummernoteInplaceWidget - no iframe mode
'iframe': True, 

# Change editor size
'width': '100%',

# Set editor language/locale
'lang': 'en-US',
}

And that wraps up things.  Hope this helps you.  There are plenty of customisations you can do…this is just one of the many.  The only limitation is your creativity with extending the core code I guess.

Zinnia Customisations Part I

I got the pleasure of working on customising a few things in Zinnia a while back, and been meaning to write up about this for a while now.  The customisation was fairly simple…instead of the user being redirected to the admin to create or edit blog posts, they should remain within the same site layout the rest of the website uses.  So, here’s how I did that. I’ve also tweaked things a bit here, so you’ll need to grab a few items from the Cheeseshop on top of zinnia itself. Note that at the time the site I was working on was built, Zinnia was at v0.14, hence the pip version prefix:

pip install django-blog-zinnia==0.14 django-summernote==0.5.5 django-chosen

Next, depending on your setup, you might have to create an app to hold these customisations you are making.  So run: ./manage.py startapp customisations Then let’s get started. First, a few lines for your views file in the just created app.


from django.template.defaultfilters import slugify
from django.contrib.sites.models import Site
from django.views.generic import CreateView, UpdateView
from django.http import HttpResponseRedirect
from django.utils.encoding import smart_str
from django.utils.html import linebreaks
from django.utils.decorators import method_decorator
from django.contrib.auth.decorators import permission_required, login_required

from zinnia.models.entry import Entry
from zinnia.settings import MARKUP_LANGUAGE

from .forms import CustomEntryForm

class CreateBlogEntryView(CreateView):
    form_class = CustomEntryForm
    template_name = 'blog/new_blog_entry.html'

    @method_decorator(login_required)
    @method_decorator(permission_required('zinnia.add_entry'))
    def dispatch(self, *args, **kwargs):
        return super(CreateBlogEntryView, self).dispatch(*args, **kwargs)

    def form_valid(self, form):
        potential_slug = slug = slugify(form.cleaned_data['title'])

        # ensure unique slug
        i = 2
        while True:
            if Entry.objects.filter(slug=potential_slug).count() > 0:
                potential_slug = u'%s-%s' % (slug, i)
                i += 1
            else:
                break

        self.object = form.save(commit=False)
        self.object.content = smart_str(self.htmlize(form.cleaned_data['content']))
        self.object.slug = potential_slug
        self.object.content_template = 'zinnia/_entry_detail.html'
        self.object.detail_template = 'entry_detail.html'
        self.object.tags = form.cleaned_data['tags']
        self.object.save()
        self.object.sites = [Site.objects.get_current().pk]
        self.object.authors = form.cleaned_data['authors']
        self.object.related = form.cleaned_data['related']
        self.object.categories = form.cleaned_data['categories']
        self.object.save()

        return HttpResponseRedirect(self.object.get_absolute_url())


    def htmlize(self, content):
        """
        Convert to HTML the content if the MARKUP_LANGUAGE
        is set to HTML to optimize the rendering and avoid
        ugly effect in WYMEditor.
        """
        if MARKUP_LANGUAGE == 'html':
            return linebreaks(content)
        return content


class UpdateBlogEntry(UpdateView):
    model = Entry
    form_class = CustomEntryForm
    template_name = 'blog/update_blog_entry.html'

    @method_decorator(login_required)
    @method_decorator(permission_required('zinnia.change_entry'))
    def dispatch(self, *args, **kwargs):
        return super(UpdateBlogEntry, self).dispatch(*args, **kwargs)

Then update urls.py to point to these views. Choose appropriate names for your confs


from django.conf.urls import url
from django.conf.urls import patterns

import views

urlpatterns = patterns('',
    url(r'^new-blog-entry/$',views.CreateBlogEntryView.as_view(), name='custom_add_blog_post'),
    url(r'^update-blog-entry/(?P[\w-]+)$', views.UpdateBlogEntry.as_view(), name='custom_edit_blog_post'),
)

You’ll also notice the forms import in views.py, so create that in your app’s forms.py file.

from django import forms

from zinnia.models.entry import Entry
from zinnia.models.category import Category
from zinnia.models.author import Author
from django_summernote.widgets import SummernoteWidget
from chosen import forms as chosenforms

class CustomEntryForm(forms.ModelForm):
    """
    Form for posting a blog entry
    Also reused by the edit view
    """
    authors = chosenforms.ChosenModelMultipleChoiceField(
        queryset=Author.objects.all(), required=False, overlay='Select One or More Authors'
    )
    related = chosenforms.ChosenModelMultipleChoiceField(
        queryset=Entry.objects.all(), required=False, overlay='Select Related Blog Entries'
    )
    categories = chosenforms.ChosenModelMultipleChoiceField(
        queryset=Category.objects.all(), required=False, overlay='Select Blog Categories'
    )

    class Meta:
        model = Entry
        exclude = (
            'slug', 'sites', 'creation_date', 'last_update', 'comment_count', 'pingback_count',
            'trackback_count', 'content_template', 'detail_template'
        )
        widgets = {
            'content': SummernoteWidget(),
        }

Now we have pretty much everything except the templates. I have these in a folder named ‘blog’ within the app’s template folder. Depending on how much of a DRY purist you are, you might split the add/edit template into three, or even more files. For me, three was enough:

  • _form_fields.html: holds the actual form fields used in the form
  • new_blog_entry.html: for when you are creating a new blog post
  • update_blog_entry.html: for when you are editing an existing blog post

The code for these is a bit too long, so to save time, you can get the app as a zip folder from Dropbox.  That’s it for the first part of this post.  In the second and last post, we’ll go through making these changes available for users.  Currently, all this awesomeness is invisible to the user.

Great search engine alternative

I was looking at a new Linux distro, Elementary OS, which is built on Canonical Ubuntu, and noticed that the default search engine was DuckDuckGo.  While the homepage has the minimalistic Google look going on, that’s where the similarity ends I guess.  One thing the engine doesn’t do, is track it’s users, which is the first reason it was built.  There are also a lot of handy little gems built in for users.  They call these goodies, and they are INDEED goodies for they cover a wide range of items like a password generator, md5 conversion, basic calculations, geographic info, various sysadmin helpers, web design helpers etc

Check them out at: https://duckduckgo.com/goodies  If you want to try it out, you can easily do so in a number of ways:

Now, I’m off to Duck something! (that’s their equivalent phrase for Google it)

DuckDuckGo Goodies 2014-05-12 17-24-20DuckDuckGo Goodies 2014-05-12 17-24-00  DuckDuckGo Goodies 2014-05-12 17-24-33 DuckDuckGo Goodies 2014-05-12 17-24-50