Bash Script 2: Command Line Options

In the previous post, we created a simple bash script to convert a video file to specific encoding options. Today, we’ll look at adding some command line options into the mix. There’s no need to parse all of this on our own. We’ll be using getopts to parse the options. Let’s start with the final script from the previous post.

#!/bin/bash

###############################
#                             #
# convert-video.sh            #
#                             #
# Written By: Barry Gilbert   #
# Written On: Oct 2018        #
#                             #
###############################

# Shows the end user the usage message
showUsage() {
  echo "" >&2;
  echo "Usage: convert-video.sh [video-filename]" >&2;
  echo "  video-filename video file to convert" >&2;
  exit 99;
}

# Check to see if a filename was provided
# $# will contain the number of arguments
# If the number of arguments is not 1
if [ $# -ne 1 ]
then
  echo 'No File Provided';
  showUsage;
fi

# $1 will contain the first argument
# Check to see if $1 is not a file
if [ ! -f "$1" ]
then
  echo 'File Not Found';
  showUsage;
fi

# Convert the video
avconv -i "$1" -c:v libx264 -preset medium -crf 20 \
  -c:a mp3 "$1.mkv";

Using getopts is very easy. For now, let’s just add a -h option to show the usage.

# Loop through the options passed in by the user
while getopts ":h" opt; do
  case $opt in
    h)
      showUsage
      ;;
  esac
done

# Reset the parameter, so we can grab the filename
shift $((OPTIND-1))

Super simple, right? Now, let’s add in support to yell at the user if they pass a bad option.

# Loop through the options passed in by the user
while getopts ":h" opt; do
  case $opt in
    h)
      showUsage
      ;;
    \?)
      # Bad option, YELL AT USER
      echo 'Invalid option: -$OPTARG"
      showUsage
      ;;
  esac
done

# Reset the parameter, so we can grab the filename
shift $((OPTIND-1))

Now, let’s add in one more, -v, for verbosity. We’ll also pass this along to avconv.

VERBOSITY_LEVEL=0

# Loop through the options passed in by the user
while getopts ":h" opt; do
  case $opt in
    h)
      showUsage
      ;;
    v)
      VERBOSITY_LEVEL=$((VERBOSITY_LEVEL+1))
      ;;
    \?)
      # Bad option, YELL AT USER
      echo 'Invalid option: -$OPTARG"
      showUsage
      ;;
  esac
done

# Reset the parameter, so we can grab the filename
shift $((OPTIND-1))

VERBOSITY_OPTION=""
if [ VERBOSITY_LEVEL -eq 1 ]
then
  VERBOSITY_OPTION=" -loglevel=verbose"
elif [ VERBOSITY_LEVEL -eq 2 ]
then
  VERBOSITY_OPTION=" -loglevel=debug"
fi

Then use the VERBOSITY_OPTION when calling avconv.

avconv -i "$1" $VERBOSITY_OPTION -c:v libx264 \
  -preset medium -crf 20 -c:a mp3 "$1.mkv";

The last thing to do is to update the showUsage.

# Shows the end user the usage message
showUsage() {
  echo "" >&2;
  echo "Usage: convert-video.sh [-h] [-v] [video-filename]" >&2;
  echo "  -h  shows this help information" >&2;
  echo "  -v  verbosity for avconv." >&2;
  echo "      Multiple -v options increase the verbosity." >&2;
  echo "      The maximum is 2." >&2;
  echo "  video-filename video file to convert" >&2;
  exit 99;
}

That’s it! Let’s look at the entire updated script:

#!/bin/bash

###############################
#                             #
# convert-video.sh            #
#                             #
# Written By: Barry Gilbert   #
# Written On: Oct 2018        #
#                             #
###############################

# Shows the end user the usage message
showUsage() {
  echo "" >&2;
  echo "Usage: convert-video.sh [-h] [-v] [video-filename]" >&2;
  echo "  -h  shows this help information" >&2;
  echo "  -v  verbosity for avconv." >&2;
  echo "      Multiple -v options increase the verbosity." >&2;
  echo "      The maximum is 2." >&2;
  echo "  video-filename video file to convert" >&2;
  exit 99;
}

VERBOSITY_LEVEL=0

# Loop through the options passed in by the user
while getopts ":h" opt; do
  case $opt in
    h)
      showUsage
      ;;
    v)
      VERBOSITY_LEVEL=$((VERBOSITY_LEVEL+1))
      ;;
    \?)
      # Bad option, YELL AT USER
      echo 'Invalid option: -$OPTARG"
      showUsage
      ;;
  esac
done

# Reset the parameter, so we can grab the filename
shift $((OPTIND-1))

VERBOSITY_OPTION=""
if [ VERBOSITY_LEVEL -eq 1 ]
then
  VERBOSITY_OPTION=" -loglevel=verbose"
elif [ VERBOSITY_LEVEL -eq 2 ]
then
  VERBOSITY_OPTION=" -loglevel=debug"
fi

# Check to see if a filename was provided
# $# will contain the number of arguments
# If the number of arguments is not 1
if [ $# -ne 1 ]
then
  echo 'No File Provided';
  showUsage;
fi

# $1 will contain the first argument
# Check to see if $1 is not a file
if [ ! -f "$1" ]
then
  echo 'File Not Found';
  showUsage;
fi

# Convert the video
avconv -i "$1" $VERBOSITY_OPTION -c:v libx264 \
  -preset medium -crf 20 -c:a mp3 "$1.mkv";

Bash Script 1: Convert Videos

Bash has a very powerful full-featured scripting language. This can do everything from maintenance to match file processing. Over the next few posts, we’ll be looking at crafting some bash scripts. The primary focus will be to convert videos to a standardized video and audio format. With each post, the script will get various updates, making the script a little smarter and more robust with each version.

Creating a simple script

First off, open your favorite text editor and copy/paste this into it:

#!/bin/bash

###############################
#                             #
# convert-video.sh            #
#                             #
# Written By: Barry Gilbert   #
# Written On: Oct 2018        #
#                             #
###############################

# Shows the end user the usage message
showUsage() {
  #TODO: Show program usage
  exit 99;
}

#TODO: Check to see if a filename was provided

#TODO: Convert the video

I usually start out with a simple script like this, one that doesn’t do anything, but has placeholder ‘TODO’ comments. If you run it, the program simply exits with no output.

showUsage()

In the example above, there is a showUsage function that we will call if the user requests help (-h) or the user doesn’t pass in a filename. To build out the message, simply add some echo commands to send text back to the user.

Here is the showUsage function with some echos:

# Shows the end user the usage message
showUsage() {
  echo "";
  echo "Usage: convert-video.sh [video-filename]";
  echo "  video-filename video file to convert";
  exit 99;
}

This is a very basic usage message. However, most of the usage messages are written to stderr, not stdout. This is done because if another program calls this script it may require the output to be something specific. I know, it’s bad practice to require that, but programmers are people too. So, let’s send the usage information to stderr instead:

# Shows the end user the usage message
showUsage() {
  echo "" >&2;
  echo "Usage: convert-video.sh [video-filename]" >&2;
  echo "  video-filename video file to convert" >&2;
  exit 99;
}

Check for a filename

The next TODO in the sample is to check that a filename has been provided. If a valid filename hasn’t been provided, we will just call showUsage. This is done with 2 checks. The first checks to see if anything was provided. The second looks for the file.


# $# will contain the number of arguments
# If the number of arguments is not 1
if [ $# -ne 1 ]
then
  echo 'No File Provided';
  showUsage;
fi

# $1 will contain the first argument
# Check to see if $1 is not a file
if [ ! -f "$1" ]
then
  echo 'File Not Found';
  showUsage;
fi

Here are the 2 if’s. The first checks the number of arguments provided to the script. This is contained in the special variable $#. The first if statement is true if the number of arguments is not equal to 1. The second if statement is true if the file does not exist. The exclamation point denotes not here.

Convert the video

Now is the time to setup the actual conversion:

# Convert the video
avconv -i "$1" -c:v libx264 -preset medium -crf 20 \
  -c:a mp3 "$1.mkv";

That backslash at the end of the line tells bash that the line continues on the next line. You can do this on the command line too!

That is a very basic video conversion to x264/mp3 in a mkv file format. I choose that specific combination because it works well over dlna to my xbox.

And now the finished product:

#!/bin/bash

###############################
#                             #
# convert-video.sh            #
#                             #
# Written By: Barry Gilbert   #
# Written On: Oct 2018        #
#                             #
###############################

# Shows the end user the usage message
showUsage() {
  echo "" >&2;
  echo "Usage: convert-video.sh [video-filename]" >&2;
  echo "  video-filename video file to convert" >&2;
  exit 99;
}

# Check to see if a filename was provided
# $# will contain the number of arguments
# If the number of arguments is not 1
if [ $# -ne 1 ]
then
  echo 'No File Provided';
  showUsage;
fi

# $1 will contain the first argument
# Check to see if $1 is not a file
if [ ! -f "$1" ]
then
  echo 'File Not Found';
  showUsage;
fi

# Convert the video
avconv -i "$1" -c:v libx264 -preset medium -crf 20 \
  -c:a mp3 "$1.mkv";

RAID

How many people have had their hard drive fail? Anyone? Everyone? The truth is, if you’ve owned a computer for the last decade, you’ve likely had a hard disk fail. When this happens, there’s little chance of getting your data back. Because of this common occurrence, some smart engineers figured out how to treat multiple hard disks as a single drive, allowing for data redundancy and recovery if one of the drives fails. This is called RAID.

RAID, or Redundant Array of Independent Disks, is simply that, a group of independent disks that provide redundancy. There are several types of RAID setups, each with pros and cons. Most people run either a mirror RAID (RAID-1) or a parity RAID (RAID-5 or RAID-6). Let’s walk through the various types.

RAID-0

This simply takes 2 or more disks and makes one big drive. There is no redundancy here. All of the drives are basically concatenated to make 1 drive that spans all of the disks.

RAID-1

This is the mirrored RAID. 2 or more disks are used and each disks contains the exact same information. The con here is once you have 2 disks, it makes no sense to add a third unless you know failure is imminent.

RAID-10

This combines RAID-0 with RAID-1. In effect mirroring and concatenating all of the disks. Sometimes this is called RAID-0+1 or RAID-1+0.

RAID-4

This type introduces parity, or a way to calculate missing blocks, should a disk go bad. Basically, 2 or more disks are used to store the data and an extra disk stores the parity information. The cons are that you can’t add more disks easily and writes have to update the parity disk, making them slow.

RAID-5

This is very similar to RAID-4, with the exception of the parity blocks being spread across all disks. Because of this, the write speed of this setup is much faster than RAID-4.

RAID-6

This is an extension of RAID-5, having a second parity block. The write speed is a bit slower than RAID-5.

Setting this up is different for every operating system. There are also hardware RAID setups, mostly RAID-0, RAID-1, and RAID-10.

Always, always, always back up your data before setting up a RAID! I’d even recommend removing your data disk from the system before setting up the RAID, so there’s no chance of an accident.

Desk Job? Exercise!

All day at work, I’m sitting with my eyes glued to a computer screen. There’s plenty of natural light, but being stuck at a desk all day long is never any fun. Not to mention all of the detriments that a desk job entails to your health. That’s why I make it a point to exercise at least 3 times a week and go on frequent walks throughout the work day.

Exercise doesn’t have to involve a lot of exertion and sweat! In fact, simply walking for more than 30 minutes counts as exercise for the day. You can even be productive on a walk. Study a foreign language using Duo Lingo, or check out your trello boards or todo list to see what else I need to get done today. Or you can just look around and enjoy the scenery.

Power Up

When you’re ready to go beyond just walking, include some run intervals! High Intensity Interval Training (HIIT) is the best way to burn fat and boost your metabolism. Just run for a minute then walk some more, then run for a minute then walk some more, and keep repeating it. That’s it!

If running isn’t your thing, or you have some medical condition that prevents you from running, try lifting some weights. It is very simple when you look at it pragmatically. Just focus on a few muscles each day.

Baby Step Adjustments

Have you ever jumped in head first into a computer project, like making a blog, and gave up after a few days? I have done  this all too much. Recently, when I quit smoking, I realized that small steps, or “baby steps” were the ultimate answer for change. This holds for computer habits as well.

Baby steps may sound stupid, but it’s the way everyone’s mind is setup. We are all creatures of habit. What one does on Mondays is mostly all the same from one week to the next. If we change too much too quick, it’s a shock to our system and we regress to our old habits. Change is mostly gradual.

So, if you’re still interested in the blog, start today by writing. Write for 30 minutes everyday. You don’t have to publish it, but you should keep it. Just name the file today’s date, i.e. “10-23.txt” (this alphabetically sorts by date). Once you’re comfortable writing 30 minutes and can draft a decent entry in that amount of time, you’re then ready to spend 15-30 minutes a day for 2 weeks setting up your blog. Yes, there is a lot that goes into setting up a blog.

This applies to just about anything. Make a small change. If you regress, learn from the mini-failure. Ask yourself: Am I pushing too hard? What can I do differently to help success?

git flow

Source control is a must-have for any team working on a project. It provides so many things, including accountability, code reviews, blah blah, and so much more! IMO, the best part is working in branches that can be merged to other branches. In a typical project you have 2 main branches: master and develop. Master should reflect what is in production and develop is the current updates slated to go live at some point in the not too distant future. Beyond those main branches, you may have a hotfix branch with a bug fix that needs to be pushed ASAP, or a feature branch with some new feature. All of those branches can get tangled and complicated. Enter git flow.

Git flow is a command line tool to help with branches and branching for git repos. It’s really simple to setup gives you a few tools that make merging branches to master easier! To get started, install git flow and then run this in your repo root:

git flow init

This init process will ask some simple questions, most can be left as the default unless you have specific naming conventions for branches. Once that is done, you can do any of the following.

Create version 1.0.0

Let’s say you’d like to merge develop into master and tag it as version 1.0.0. Perform the following commands:

#Start Release 1.0.0, merging develop into master
git flow release start 1.0.0

#Update your version file
vim version

#Commit the version file update as the 'Version Bump'
git add version
git commit -m 'Version Bump'

#Finish Release 1.0.0
git flow release finish 1.0.0

#Push it, including tags
git push --follow-tags

Feature Branch

When creating a new feature that may not go to production for a while, you should do the work in a feature branch.

#Create a new feature branch from develop
git flow feature start report-xyz

#Make changes and commit
vim report-xyz
git add report-xyz
git commit -m 'Adding report-xyz'
git push

#Finish the feature, merging it into develop
#This can also be done via a PR for code reviews
git flow feature finish report-xyz

Hotfix Branch

There’s a bug on production! Going to have to fix it, based on the code in master. Once it’s fixed both master and develop need to have that update.

#Create a new hotfix branch from master
git flow hotfix start bad-profile-image

#Make your changes
vim profile
git add profile
git commit -m 'Fix for profile image'
git push

#TODO Verify that all is well

#Once all is good, do a 'Version Bump'
vim version
git add version
git commit -m 'Version Bump'

#Finish the hotfix, merging it into develop and master
#This can also be done via PRs for code reviews
git flow hotfix finish bad-profile-image

That’s all for today. Happy Coding!!

Node Callback Hell

I may be a little late, but I’m hopping aboard the node train! Having a more than a decade under my belt of php and roots in C/C++ helps tremendously in seeing the bigger picture with node. IMO, it’s a great platform! Having said that, there are quite a few caveats that you must address. I’ll be covering a few this month and the first is: Callback Hell!

Callbacks are great, but they can quickly cause your code to be 5 levels deep. Here’s an example:

const verifyAuth = function( token, callback ) {
  var token_query = 'SELECT * FROM users WHERE token = ?';
  var roles_query = 'SELECT * FROM user_roles WHERE user_id = ?';
  mysql.query( token_query, [token], function(err, rows) {
    if ( err ) {
      callback(err);
    } else {
      if ( rows.length == 0 ) {
        callback('User not found');
      } else {
        var user = rows[0];
        mysql.query( roles_query, [user.user_id], function(err, rows) {
          if ( err ) {
            callback(err);
          } else {
            callback( null, user, rows );
          }
        })
      }
    }
  });
};

Here, you see there are 2 separate db queries. An initial solution is to name and then split out the separate functions:

function lookupToken( token, callback ) {
  var token_query = 'SELECT * FROM users WHERE token = ?';
  mysql.query( token_query, [token], callback );
}

function lookupUserRoles( user, callback ) {
  var roles_query = 'SELECT * FROM user_roles WHERE user_id = ?';
  mysql.query( roles_query, [user.user_id], callback );
}

const verifyAuth = function( token, callback ) {
  lookupToken( token, function(err, user) {
    if (err) {
      return callback(err);
    }
    lookupUserRoles( user, callback );
  })
}

That’s a great first step! Now, let’s travel down the rabbit hole, with async. This allows you to load the 2 db calls in sequence:

const async = require('async');

const verifyAuth = function( token, callback ) {
  var token_query = 'SELECT * FROM users WHERE token = ?';
  var roles_query = 'SELECT * FROM user_roles WHERE user_id = ?';
  var user_record;
  var roles_records;

  async.series( function lookupUser(done) {
    mysql.query( token_query, [token], function(err, rows) {
      if ( err ) {
        return done(err);
      }
      if ( rows.length == 0 ) {
        return done('User not found');
      }
      user_record = rows[0];
      done()
    })
  }, function lookupUserRoles(done) {
    mysql.query( roles_query, [user.user_id], function(err, rows) {
      if ( err ) {
        return done(err);
      }
      roles_records = rows;
      done()
  }, function allDone( err ) {
    callback( err, user_record, roles_records );
  })
};

It doesn’t stop there! You can also iterate of an array in parallel. For example, if you want to update many records at once, you can use async.each:

const async = require('async');

const saveRecords = function( items, callback ) {
  var query = 'UPDATE item SET value = ? WHERE id = ?';
  async.each( items, function saveItem(item, done) {
    mysql.query( query, [item.value, item.id], done );
  }, function allDone( err ) {
    callback( err );
  })
};

While you can’t remove callbacks completely, my current rule of thumb is to not have any code with more than 5 indentations, so 2 space = 1 indent means no lines with more than 10 spaces at the beginning.

Re-using custom npm modules

From time to time, there will be a need to include a customized node module across multiple projects. This may be because the original has a bug and you know how to fix it, or simply creating a new re-usable module. Luckily, there’s an easy way to do this using github. If you’re new to github, I’d suggest reading up on it before proceeding.

Step 1: Push your module to github

Prepare your module, as usual. Instead of sending it to npm, just push it up to a new github repo. If you’re working on fixing or updating a module that already exists, you can just fork the module’s github repo and work in your fork!

Step 2: Installing from github

This is really easy:

npm install --save username/repo

Here’s a real example:

npm install --save barrygilbert/mysql-restapi

You can also reference a specific branch:

npm install --save barrygilbert/mysql-restapi#master

Step 3: Do your thing!

Now that you can install that module into multiple projects, go code it up!

Linux Commands Cheatsheet

Here is my linux command cheatsheet. Make sure to see the tips at the end!

#Copy a file
cat file_1 > file_2

#Append a file to another file
cat file_1 >> file_2

#Append to .gitignore:
echo node_modules/ >> .gitignore

#Create a tar with everything under folder "a"
tar -c a/ > tarball.tar

#Create a gzipped tar with all in folder "a"
tar -c a/ | gzip  > tarball.tar.gz
#Create a bzipped tar with all in folder "a"
tar -c a/ | bzip2 -9 > tarball.tar.bz2
# -9 is the highest compression
#Nicely create a tar with everything under folder "a"
nice tar -c a/ > tarball.tar

#Nicely create a bzipped tar with all in folder "a"
nice tar -c a/ | nice bzip2 -9 > tarball.tar.bz2
#Dump the database, with a datestamp, with low priority b/c it's prod
nice mysqldump -u DB_USER -p DB_NAME | nice bzip2 -9 > dump-`date -I`.sql.bz2
#Answer the password prompt

#Copy folder "a" recursively to another computer running ssh and rsync
rsync -azvv --progress a/ user@host:/path/on/host

#Copy just the contents of folder "a" to another computer running ssh
scp a/* user@host:/path/on/host

#Copy folder "a" recursively to another computer running ssh and rsync over port 2222
rsync -azvv -e "ssh -p 2222" --progress a/ user@host:/path/on/host

#Copy just the contents of folder "a" to another computer running ssh over port 2222
scp -P 2222 a/* user@host:/path/on/host

#Backup an sdcard on /dev/sdc
dd if=/dev/sdc of=backup.sdcard.img BS=32M status=progress

#Copy an image to an sdcard on /dev/sdc
dd if=sdcard.img of=/dev/sdc BS=32M status=progress

#Make a command be quiet!
cat some_big_file > /dev/null

Tips

> writes to a file
>> appends to a file
| sends to a command (stdout => stdin)

rsync will only transfer files if they are different, so it is great for manually syncing media files across devices, or even pulling down the latest files to a local build.

nice can run at priorities. Default niceness is 10, and can be adjusted by adding -n 15 (or whatever niceness)

npm-run-all

If you’ve used node.js, you probably have npm jobs that you can run. During development, you may have several of these jobs running at once. Perhaps serving up some pages while also running grunt to minify your front-end javascript. Instead of having 2 terminals open, you can use the module npm-run-all and just have the one.

Using the example above, you have 2 npm jobs that run. These are both defined in package.json as:

...
  "scripts": {
    "grunt": "grunt",
    "serve": "node index.js"
  }
...

Each of these can be started by running:

npm grunt

and

npm serve

Starting them both requires the npm-run-all module:

npm install --save npm-run-all

Then add a new npm job under scripts:

...
  "scripts": {
    "grunt": "grunt",
    "serve": "node index.js",
    "start": "npm-run-all -p -l grunt serve"
  }
...

That’s it! Now kick off both jobs:

npm start

npm-run-all has quite a few command line options. The two I’m using is:

  • -p Run jobs in parallel
  • -l Show the job name with the output (helps with debugging)