Developing on OS X – Part 1

This multipart series will take you through how to setup a Python development environment on OS X Yosemite & El Capitan.  We will setup Python…and the various helpers I use for Django development.  We’ll also install a few other applications I make use of in my day-to-day coding activities.  For this first part, we’ll go through settings up the various helper applications we’ll end up using for our Python setup

Enabling Writing to NTFS

Install Homebrew and Homebrew Cask

For anyone new to OS X, homebrew is a package manager, and it enables you to install command-line applications and libraries that would require manual compilation.  While Homebrew Cask extends Homebrew.

To install homebrew, run this command on your terminal:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

To get Homebrew Cask run this command on your terminal:

brew tap caskroom/cask

We’re now ready to install osxfuse…El Capitan users will want to install the 3.x.x version (3.1.0 or higher would be best) to avoid System integrity Protection issues https://github.com/osxfuse/osxfuse/releases.  For Yosemite, brew will suffice

brew cask install osxfuse

The final step to give you write-access to NTFS will be to install ntfs3g.

brew install homebrew/fuse/ntfs-3g

If using El Capitan, System Integrity Protection’s fs role will first need to be disabled.  This is needed to enable writing to /sbin

  • reboot and hold CMD+R to go into recovery mode
  • Run this on a terminal: $ csrutil disable
  • reboot normally

Create a symlink to mount NTFS

sudo mv /sbin/mount_ntfs /sbin/mount_ntfs.original
sudo ln -s /usr/local/sbin/mount_ntfs /sbin/mount_ntfs

Remember to re-enable System Integrity Protection (and re-lock the system directories) if you’d previously disabled it (same steps as before), only run $ csrutil enable

After that you should now be able to read and write to your NTFS drives.

Xcode

Xcode is an integrated development environment (IDE) containing a suite of software development tools developed by Apple for developing software for OS X and iOS.  While you don’t need the entire suite, you’ll want to install the command line tools.  Run this command then follow the instructions:

xcode-select --install

Terminal

I always like having a good terminal at my disposal…iterm2 is my custom terminal of choice on OS X.  To get iTerm2, download and install it as a normal OS X pkg file

Version Control

It’s always recommended to have a VCS in place when you’re writing your code….this way you can always revert to a previously working version of your code if you break something along the line.  It’s a matter of personal preference which VCS you use, but I like git.  And for anyone new to VCS, this handy guide explains it all.

To install git:

brew install git

 

Next step is to configure git for your setup.  If you haven’t configured your SSH keys, that would be the first thing.  If you get stuck, this Github link will help


ssh-keygen -t rsa -C "<your-email-address>"
git config --global user.name "<your-name>"
git config --global user.email "<your-email-address>"

 

After this add your key to your Github account (same process for Bitbucket or any other code hosting site where you can upload your SSH keys).

PRO TIPS:

  • You can never be too careful with your code so it’s always a good idea to enable two-factor authentication whenever possible.
  • For Android users, if you change phones frequently, instead of the Google Authenticator app, you can check out Authy…it’s slightly better since it has a clud backup feature, and if you’re a Chrome user, you can even register your browser as an MFA device

That’s it for the basics…in the next blog post we’ll get down and dirty with getting Python running

Advertisements

OSGeo Live on virtual box

Osgeolive_wordle

For those of you who still wants to keep windows as their main operating system but wants to try the goodies of Free and Open Source software for Geo (FOSS4Geo), OSGeo has many options like live DVD and virtual box VMs. All you have to do is to install Virtual Box and download OSGeo virtual machine (plus 7zip for extracting zipped file). Follow the instruction below

http://live.osgeo.org/en/quickstart/virtualization_quickstart.html

Download will take time as it is around 3GB (I downlaoded vdkm), extratct using the following command (if you are on windows, use 7zip GUI)

7z x software-backup/osgeo-live-vm-8.0.7z

When you will run command sudo apt-get update, password is require and it is “user” (on desktop, there is a file named passwords.txt, contains all the passwords e.g. OS-user ‘user’ has password ‘user’)

One more thing, to set screen to a reasonable size, at the end of OSGeo blog there are some commands, if they do not work (like in my case) go to  start -> System Tools -> Synaptic Package Manager and install the following packages (read my old blog about it Resize virtual machine’s desktop in VirtualBox)

apt-get install virtualbox-guest-dkms
virtualbox-guest-utils
virtualbox-guest-x11

The first package will install second package as dependencies however you have to install third package. If successful with installation, you should see a desktop like below

OSGeoLive

What it contains

  1. Libraries with paths configured (this is what you need the most e.g. gdal)
  2. Programming languages like python (with packages installed)
  3. QGIS, GRASS GIS, gvSIG, OpenJump, SAGA GIS and uDig
  4. R Statistics
  5. and many more 🙂

Trust me, for new users, the first two are very important as they reduce complexities and frustration (a big thanks to OSGeo of course ).

 

Latitude/Longitude of each pixel using python and gdal

I needed lat/long of each pixel of a GeoTiff file. I searched through internet to find a solution and ,thanks to Stack Overflow, found a piece of code that I modified. As usual, python and gdal were used. Just follow the instructions below.

First get the number of rows and columns of you image by using gdalinfo (although it can be automated as well but lets just go with it)

gdalinfo raster.tif

Now edit the python code below and set rows and columns according to your image (don’t forget to change the image name).

from osgeo import gdal

# Open tif file
ds = gdal.Open('raster.tif')

# GDAL affine transform parameters, According to gdal documentation xoff/yoff are image left corner, a/e are pixel wight/height and b/d is rotation and is zero if image is north up. 
xoff, a, b, yoff, d, e = ds.GetGeoTransform()

def pixel2coord(x, y):
 """Returns global coordinates from pixel x, y coords"""
 xp = a * x + b * y + xoff
 yp = d * x + e * y + yoff
 return(xp, yp)

# get columns and rows of your image from gdalinfo
rows = 36+1
colms = 34+1

if __name__ == "__main__":
 for row in  range(0,rows):
  for col in  range(0,colms): 
   print pixel2coord(col,row)

Finally, execute this python script using following command (provided that you have named your file script.py)

python script.py

Here is the output of my raster.tif

(29.25, 6.0)
(29.5, 6.0)
(29.75, 6.0)
(30.0, 6.0)
(30.25, 6.0)
(30.5, 6.0)
(30.75, 6.0)
(31.0, 6.0)
(31.25, 6.0)
(31.5, 6.0)
.
.
.

In my case, I converted output to csv and opened it in qGIS along with raster.tif (shown below).

for_fb2

and a closer look

for_fb

 

Proj4 and python

Proj4 is a projection library used by many software including GRASS, MapServer and PostGIS (source). For python, one has to install package pyproj.

Few days back, we were trying to georeference some aerial photos (around 7000). The aerial photos were accompanied by an excel sheet with cells like image id, latitude and longitude (image’s center).

But there was an issue, we were georeferencing images using source maps with projection EPSG:3857 (a.k.a the web projection). Every time we georeference an image, lat long to x y conversion was required (geographic to projected conversion).

To solve this, I converted excel sheet into csv (because parsing excel sheets with python is a bit too much for me) and wrote a small python script to transform coordinates. Following is python code.

import csv
import pyproj

# read from csv
cr = csv.reader(open("Aerial_images.csv","rb"))

# writer to csv
csvfile = open('Aerial_images-3857.csv', 'wb') 
csv_writer = csv.writer(csvfile, delimiter='~',quotechar='|', quoting=csv.QUOTE_MINIMAL)

#source 
p1 = pyproj.Proj(proj='latlong',datum='WGS84')

# final projectoin
p2 = pyproj.Proj(init='epsg:3857')

for row in cr: 
  print row[0], row[1], row[2], row[3]
  x1=float(row[2]) 
  y1=float(row[3]) 

  x2, y2 = pyproj.transform(p1,p2,x1,y1)
  #'%9.3f %11.3f' % (x2,y2)

  str_temp = row[0], row[1], row[2], row[3], x2,y2 
  csv_writer.writerow(str_temp)

All it took was few seconds and conversion was done (converted points below).

aerial-images

 

Easily Generate PDFs in Python

A common task in any web application these days is generating files for user reports, the most common being PDFs.  I’ve been building a simple app to track assets and who’s been assigned what asset within a department, and I needed a PDF report of the main page that shows asset allocation.

Naturally, I went with xhtml2pdf since I wanted to take a HTML file, feed it with some context data, and return the output as the PDF.  This is all fine for basic table PDFs, but the moment you add anything fancy like border lines and background colours….things get a bit messed up layout.

After a few minutes of Google search, I came across weasyprint.  It renders the PDFs much better, exactly what is seen in the browser.  For installing this, the recommended way is to go through the CheeseShop as always:

pip install WeasyPrint

But for my Ubuntu 12.04 dev environment, I needed a few more extra plugins to get things installed:

sudo apt-get install libgdk-pixbuf2.0-0 libffi-dev

If these libraries are missing, you’ll get this error:

weasyprint-install-error

So after running through the setup…should take a couple of seconds to download and compile all the dependencies, you can now generate your PDF. I’m developing the system in Django, so here’s what I did:

from django.http import HttpResponse
from django.template import RequestContext, loader
from weasyprint import HTML

context_dict = {
'assets': Item.objects.all(),
}

template = loader.get_template('asset/pdf.html')
html = template.render(RequestContext(request, context_dict))
response = HttpResponse(mimetype='application/pdf')
HTML(string=html).write_pdf(response)

return response

Incase you have inline images using the static templatetag, you might want to make a few edits make a HTTP request on the app since WeasyPrint will have trouble determining the base URL. Note that it might cause a deadlock on a single-threaded server as pointed out in this StackOverflow post:


HTML(string=html, base_url=request.build_absolute_uri()).write_pdf(response)

After that, the result should be a pleasing to look at PDF supporting all sorts of CSS formatting options. For my deployment, I’m using inline CSS in my base template, but you can also feed in more CSS from the included CSS class. More details on this: Python API and StackOverflow

Of course your can always render the HTML directly without an extra library parsing it then return the plain HTML and have the user print the page as a PDF, but this means the user’s browser determines the output….Chrome and Firefox always do a good job here, but you never know what might happen down the line. But if this is OK for you, here’s what you can do:


context_dict = {
'assets': Item.objects.all(),
}
template_name = "asset/pdf.html"
return render_to_response(template_name, context_dict)