Add text to beginning of a file in bash

If you’ve ever worked with bash or the Linux command line, you already know you can add text at the end of a file using the >> operator. ¬†I however found myself in need of adding text to the beginning of the file, and started to wonder what the opposite of >> was (definitely not << ūüôā )

Due to lack of a prepend operator, you can get creative with sed, or even combine echo and cat

  • sed -i “1itext to prepend” myfile.txt: ¬†-i¬†will update the contents of the file and automatically save it, while¬†1i¬†will¬†insert to the 1st line of the file.
  • echo -e “text to prepend\n$(cat myfile.txt)” > myfile.txt:¬†the command inside of $(…) is executed before the file is overwritten with >myfile.txt. ¬†The -e is to ensure any backslashes¬†in myfile.txt are interpreted…otherwise the command will crash and burn.

Of course given the flexibility of some of the Linux tools at our disposal, other solutions are definitely possible.  Personally, I went with sed, but if you use echo and cat regularly, the second format might be more easier to remember

 

Advertisements

Gdal – a swiss knife

572px-GDALLogoColor.svg

I worked with Global Inventory Modeling and Mapping Studies (GIMMS) dataset few weeks back, it is based on AVHRR and goes back to 1981.  For analysis, I had to resample GIMMS dataset from 8 km to 5 km (the version of GIMMS I was working with had 8 km resolution). I was working with approx. 736 images. Since I was performing an other task in R, thus, decided to combine resampling with it. I wrote a bash script to call my R script and passed source image and desired resolution image. The process was very simple as shown below (full script available here).

.......
# target image folder
src1folder="path-to-files"

#desired resolution image
src2folder="folder-for-desired-image"
src2img="desired-image-name.tif" 

for myfile in *.tif
do
  # target path and image name
  target="target-image-folder-path" 
  
  Rscript /media/external/muhammad/myscripts/scale.r $src1folder $myfile $src2folder $src2img $target
  
  # R temp folder is being filled by temp files, we need to clean it as well. 
  rm /tmp/R_raster_geoscilab/*.gri
  rm /tmp/R_raster_geoscilab/*.grd

done
.......

The R script was executed inside the loop for each image in folder “path-to-files”. R script (available here) read desired and source images and performed resampling using projectRaster() method. There were couple of other tasks performed by my R script but resampling was major. ¬†When I tested this approach, ¬†a single image took 12 minutes (5 images per hour). WOW! my script will run for 6 days to finish this task. Yes, scripts worked but 6 days were too much. I found out that it is projectReater() function which is taking almost 11 minutes. After a quick discussion with a friend of mine (R guru and big fan of gdal + Open Source) I tried gdal utilities. Quickly, I wrote a new script utilizing gdalwarp utility (command below, full script is available here).

gdalwarp -tr 0.050000000000000 -0.050000000000000 $outputfile $output_newfile

Notice the values for -tr switch which dictates the output image to be in 5km pixel size.  When I tested this image, WOW it took only 5 seconds for a single image. Still R was required for filtering but now I will not run my scripts for 6 days. It is important to note here that I used the default method for resampling in gdalwarp. Below is a GIMMS 5km resolution image (with nodata for further processing).

GIMMS-2011-dec-a

Moral of story: Try to use other tools rather than focusing on one.

Essential Linux commands, version 1.0

I have horrible memory when it comes to names and numbers, same is true for Linux commands. I use and then quickly forget useful commands or switches. I always promises my self to note down these commands but its not working. This blog entry (and the future versions) will serve as personal manual.

Package Information

Show installed packages (all)

dpkg -l

Search for a specific package e.g. gdal

dpkg -l | grep gdal

What if I want to know if  a package is installed or not.

dpkg-query -l 

for example, package gdal is installed or not (Note you have to provide full package name, its not like grep)

dpkg-query -l libgdal-dev

or just the -S switch

dpkg -S libgdal-dev

and if you want to know dependencies of a package, run the following command

apt-cache showpkg package_name

e.g. 

apt-cache showpkg libgdal-dev

Note: Not only the above command shows package dependencies for libgdal-dev package but will also show packages depending on libgdal-dev,see section Reverse Depends in the output.

Folders

List files, sorted by creation date and time

ls -ltrh

Sort by file size

ls -lhS

List hidden folders

ls -ld .?*

List folder with permissions

ls -ld */

Locate command is dependent on a DB, thus, it is necessary to run the following command from time to time (use sudo)

updatedb

Locate command is perfect but some times I want to find folder under a specific path

find /path -name "name" -type d

e.g.

find /var -name "apache2" -type d

-type d is for directory

and to find out folder size

du -hs

switch h is for human readable format and s is summarized. 

Create multiple folders with one command (the command below will create folder 1990, 1991….2010)

mkdir {1990..2010}

Delete folders (and files in them)

rm -rf foldername
or 
rmdir foldername

-f is for force (you will not be asked again and again for ‘yes’)

Files

File size using the following command

ls -sh

h is for human readable format.

Copy files from one location into other

cp -v file*.html /var/www/website

Note: Use -r switch to copy directories.

copy all files from one folder to an other

cp Folder1/* Folder2/

Move files

mv filename folder_name

Example:

 mv test.png /var/www/images

or for folder 

mv -r /home/user/folder /location/to/cop

Rename files

mv original.txt renamed.txt

sudo might be required

Cat command is one of Linux powerful commands. It sequentially concatenates multiple files and can be redirected into a file or pipe for an other command.

cat *.txt > merged-file.txt

or

cat *.txt > grep "some search string"

In a console, how to create files (no mouse to right click and select “create new file” ūüôā ). Use the touch command

touch newfile.txt

ls command can tell a lot about file and folders e.g. if file/folder permission information is require then use

ls -l

It show  user, group, date and time of creation and type of permission on a file and folder. How to understand permission information e.g

drwxrwxr-x Tony:stark ...............

Lets break it down

directory  owner   group  others
d          rwx     rwx    r-x 

it shows owner, group and others permission information e.g. the above item is a folder and owner “Tony” has read, write and execute permissions. user belong to group “stark” have read, write and execute on this item and all other users have read and execute permissions but they cannot write to this item.

What if a user wants to have permissions to read, write or execute. Here comes change mode command

chmod a=rwx file/foler 

e.g

chmod a=rwx Iron-man.txt

sudo is required.

If item is a folder and contains sub-folders and files, the recursive switch can help to set the same permissions for files/sub-folders inside the parent folder

chmod -R a=rwx file/folder

a=rwx means that all owner, group and others will have read, write and execution permission.

If just one type of user is to be given permissions then use the following versions

Give owner read and write permissions
chmod u+rw file/folder
Give group read and execute permissions
chmod g+rx file/folder
Give others read and execute permissions
chmod o+rw file/folders

or combine user and group in one command

chmod u+r,g+x file/folder

Similarly,  permission can be revoked using change mode command e.g. disallow others the permissions of read and write

chmod o-rw file/folder

Find a string in file (in specific folder)

grep -nr yourString* .

-r will enable grep to search in sub folders

Remove files

rm file

remove folder

rmdir foldername

Remove files and folders recursively

rm -rf foldername