Linux Text To Speech with Saved Audio

In my last blog post I described a procedure to find a forgotton PIN for 10 digit mechanical lock boxes where you enter a specific sequence of button presses to efficiently test all the combinations. The generated sequence was supplied in the form of a text file, and although this works, it's a little cumbersome moving your eyes between the buttons and paper all the time. It occurred to me that this would be a lot easier if the numbers were read to me. I then imagined how easy it would be if I had a pair of headphones and the instruction in an audio file on my phone. This seemed like the perfect application for a text to speech application and Linux.

After a little bit of research I decided to use the eSpeak speech synthesizer. It has many options for different voices for different languages and countries and allows quite a bit of customisation of the the way the text is read.

The command below that converts the text in "lockbox.txt" to audio in "lockbox.wav" uses the english voice (-ven), pronounces capital letters in a certain way (-k20), leaves a certain gap between words (-g4), reads back at a certain words per minutes (-s90), and  has a certain pitch (-p29). It's that easy!

espeak -ven -k20 -g4 -s90 -p20 -f lockbox.txt -w lockbox.wav

Before processing the file I made some slight alterations to it by replacing some of the commands in the lock box opening sequence. Originally the commands were zero thought nine, open, and clear. I replaced open with test as it was only one syllable and easier to hear.  It's also important to leave spaces between numbers otherwise it will read 11 as "eleven" instead of "one one".

Here is the instructional WAV file converted to an MP3.  It goes for 30 minutes or so and with a little bit of practice you should be able to follow along at that speed.  If you can't, that's fine, just slow the speed down in your music player.  If you screw up just go back 15 or twenty seconds to catch up.

For the YouTube fans out there, here is another version. It might possibly be the most boring and monotonous video on YouTube. That's my speciality though :-)

To be serious though I'd like to try eSpeak on a Raspberry Pi.  I think it'd be great to read out status updates and events.  Compared to some of the other synthesized voices I've heard it's actually pretty good.

Anatomy of a Code 128 Bar Code

I've been playing around with bar codes recently, specifically the Code-128 variant.  I'm looking at this particular type because it has some interesting capabilities, the one that interests me the most is the ablity to encode all 128 ASCII characters.  This may seem innocuous, but when you think about how most bar code scanners are connected to computers you'll see the problem. We'll get to that later.

Firstly how do you generate a bar code?  Once again Linux comes to the rescue.  A command called barcode is quite powerful (but hard to google, don't use common names for programs).  It can generate a postscript file containing a wide range of bar code formats.  For example, I've encoded a message with the command below.

barcode -b '103 34 100 65 82 0 67 79 68 69 83 0 65 82 69 0 70 85 78 1 98 77 26 13 9' -e 128raw -o blog.ps

Sample Bar Code

I've used the 128raw mode because it gives me more flexibility and allows me to enter special characters.  If we decode that message using the on-line service at zxing.org we can see the text.  Note that there is a carriage return in the bar code, this places the smiley emoticon on the next line.  You can try to decode it using your phone, but you may not be able to get a good image off the screen.  Some bar code apps don't seem to implement the full standard either, so it may not recognise the carriage return.

Decoded bar code message

So what you say.  Hold your horses I'm getting there.  The decimal numbers I used on the command line can be seen in the results above in hexadecimal.  The program also automatically adds a parity symbol and a stop symbol to the bar code, these are the last two numbers, 1e and 6a.  The parity symbol is calculated by multiplying the code for each symbol by it's zero referenced position in the bar code and then adding them together along with the code for the first symbol.  This is then divided by 103 and the remainder is the parity symbol.  For example.

103 + 1x34 + 2x100 + 3x65 + 4x82 + 5x0 + 6x67 + 7x79 + 8x68 + 9x69 + 10x83 + 11x0 + 12x65 + 13x82 + 14x69 + 15x0 + 16x70 + 17x85 + 18x78 + 19x1 + 20x98 + 21x77 + 22x26 + 23x13 + 24x9 = 15274

15274 / 103 = 148 remainder 30.

This means the parity symbol is 30 or 1E in hex.

Code-128 is extensive, it has three code sets to select from.  The start symbol you select decides which set you use.  Later on in the bar code you can switch to other code sets with a special control symbol.  This can be permanent, like using caps lock on a keyboard, or it can be for the next character only, sort of like the shift key.  I've used both methods in my sample bar code above.

In the image below I've broken down the bar code into sections explaining what each part does.  The patterns for each symbol can be found online, they aren't related to the code number, they've been selected using a specific set of guidelines.  So it's basically a look-up table situation.

The bar code starts and ends with a empty area known as the quiet zone.  The rest is pretty self explanatory.  You should probably check out the Wikipedia page for Code 128 bar codes.  It has a great table of all the code sets.

I've put the pdf of the image below here.  It might be a bit easier to read.  If you open, rotate, and enlarge it so it fill most of your screen, you should be able to scan it with your phone.  It works for me.

Anatomy of a Code 128 Bar Code

So why is this interesting?  Does it have to be?  Most bar code readers connected to computers are seen as a simple keyboard input device.  They don't sanitise their input at all, why would they, they're only expecting numbers and the occasional letter right?  Well, with code-128 you have all the ASCII control characters at your disposal.  I've tried this on the Symbol MC3090 (the bane of my existence) and it recognises and executes the escape and carriage return symbols without hesitation.  I haven't tried the other control characters.  I should also mention that most scanners add a carriage return after each scan, kind of like how you enter data into a text field and press enter to go further.  With just these two symbols you can automate processes with a specially crafted bar code, kind of like the USB rubber ducky that +Hak5 sell.  I don't want to use this for nefarious purposes, I just want to use it to automate some mind numbing tasks, but you could imagine situations like below are possible.

XKCD - Exploits of a mum

Generating a Captcha from the Linux Command Line

I've been playing around with OCR software lately, Tesseract, gOCR, and Ocropus.  I'd like to get all the developers together in a room and lock them in until they come out with something awesome.  Each program has features that I'd like to see in combined package, but for now I'll work with what I have.

Anyway, this post is a bit of a tangent to the whole goal of OCR, recognising text.  Thinking about how to make the job of an OCR program harder can lead to a deeper understanding of the recognition process.  The leading technology to beat OCR is the captcha.  Those annoying little blurred words you have to read to gain access to forums and other sites.  They're there to prove you're a human and not a spam bot.  Through a combination of geometric distortions and filters it makes text hard for computers to read but not humans.

Anyway, for the hell of it I thought it would be nice to be able to generate them from the command line.  So here's what I came up with.  You'll need imagemagick installed as well.

I've put everything together in a script located here.
captcha.sh

Plain text is generated first.

convert -background white -fill black -font FreeSerif-Bold -pointsize 36 label:'all work and\nno play\nmakes Grant\na dull boy' test.png

Text

A wave is added to the text.  Ideally the magnitude and wavelength of the wave would be randomised per line, but a simple uniform wave will do a reasonable job as well.

convert test.png -background white -wave 4x55 test2.png

Wave added

A blur is added to text.

convert test2.png -blur 0x1 test3.png

Blur added

A photocopy filter is then added.  This helps to segment some of the letters.
The photocopy filter was found at www.imagemagick.org/discourse-server/viewtopic.php?f=1&t=14441&start=0

convert test3.png -colorspace gray -contrast-stretch 4%x0% \( +clone -blur 0x3 \) +swap -compose divide -composite -blur 0x1 -unsharp 0x20 test4.png

Photocopy effect added

Command Line Mail Merge For Wedding Invitations With Perl

Recently I've been occupied making wedding invitations, so I haven't really done anything too technical, but during this task I did come across a nice little command line trick that could come in handy.

I like to lay out documents like wedding invitations in Libre Impress, it's basically the open source answer to PowerPoint.  You might think it's weird to do layout like this in a presentation program, but it's simple and it allows me to exactly control text and graphics and how they're positioned on the page.  The one drawback this method has is that I can't find a way to import a guest list into a template and generate a final document, basically a mail merge, but there is a pretty easy work around using the command line.

The invitations I want are really basic.  They're text only and approximately one third the size of an A4 page.  They do however need to be personalised, which makes things a little harder but not impossible.  The first step is to complete one invitation and use a generic place holder for the name of the guest.  Use something that won't appear somewhere else in the file.  I'm using the string GuestName. This invitation is then copied to fit 3 onto the page.  You then duplicate this page as many times as you need to so that there are enough invitations in the file.

For this process to work the file needs to be saved in the flat open document format, fodp.  The format is xml based and is easily read, but you need to use the flat version that's uncompressed.  That way the place holder string, GuestName, is in the file in plain text.

Template Invitation

I've mocked up a quick invitation to demonstrate the process.  I've put a few Easter eggs in the invitation mainly just to amuse myself.

Template Invitation

The next thing you need is a guest list.  A simple text file will suffice.  For this I've created a file that contains three people.  Each person is on a separate line.

guests.txt

Alice

Bob

Eve

What's needed now is a way to replace the place holder string GuestName with names from the guests file.  Each time a string GuestName is found it needs to be replaced with a different guest from the guests file.  It turns out that following perl command is ideal for this.

perl -pe 's/GuestName/chomp($r=<STDIN>);$r/ge' Template.fodp < Guests.txt > Invitations.fodp

I'm still learning how it all works but I'll try and explain the command.

the e option allows the command to be entered on the command line

the p option loops over the command and prints the result

the s command is used to replace the string GuestName with chomp($r=<STDIN>);$r

chomp($r=<STDIN>);$r reads a line from the standard input, the Guest.txt file, and removes the newline character at the end.

the g option means do a global search and replace

the e option indicates to evaluate the replace expression

This is the result.  The place holder has been replaced with the name of the guest.

Final Invitation

I can then use the Invitations.fodp file to generate a pdf and then print out the invitations.  There's one thing to point out though.  Just because a GuestName string is first on the page it doesn't mean it's first in the file.  So the input order may not be maintained exactly, it can be done, but you just need to be aware of it.  For me the order doesn't matter so I'm not concerned about it.

Cutting Through the Noise on Twitter

So you've been on Twitter for a while now, over time you've steadily followed more and more people, and now you've reached a point where you're stating to get overwhelmed by the endless barrage of Tweets,  what do you do?  It's time for a clean-up.

Slowly the people you follow change, and your own interests change, so once in a while it doesn't hurt to go through and re-evaluate whether or not you still want to follow someone.  Keeping your timeline manageable and full of relevant content will make your twitter experience more enjoyable, and valuable.  Clearing out the clutter also leaves room to follow new people whose interests are more aligned with your own.

One of the more subtle ways Twitter can become unmanageable is by following people who Tweet a lot of things your not interested in.  We're all guilty of it, sometimes our posts are irrelevant and aren't particularly interesting to others, but that's OK.  That's what I like about Twitter, you get to see the real person and get a better understanding of their personality.  It does however become a problem when people start tweeting 50 or so times a day and fill your timeline with things you don't care about.  I consider 10 tweets a day to be okay, but depending on the quality of the content I don't mind if people post more.

The following is a a quick and easy way to find out if someone is cluttering up your timeline.  Using the Linux command line, or cygwin if windows is your thing, you can find out who the culprits are with the following method to see how many tweets people make.

Open up twitter and scroll all the way to the bottom until you can't load any more tweets.  Select all the text by pressing Control-A.  You should see something like the image below.  Then press Control-C to copy all the text.  Paste the text into a text editor and save it as a file.

Copying Twitter Information

Running the following command on the file you just created will find all the instances of Twitter users in the the timeline.  This basically means any time an at symbol appears followed by an alphanumeric character or an underscore.  The results are then sorted by name, counted, and then sorted by frequency.

cat Tweets.txt | grep -o '@[a-zA-Z0-9\_]*' | sort | uniq -c | sort -n

Looking for Excessive Tweeters

As the data I captured is over a period of about 2 days, you can see the majority of people tweet 10 times or less a day.  I'm not too worried about most of the ones over that limit as they're accounts that I find valuable.  You can however see that a lot of the tweets in my timeline are from 4 news services, and if I can get rid of any of these it will make a big impact to the number of tweets I get.

That should find all the user handles in your timeline, but you still need to find out about retweets, and that can be done with the next command.  It looks for instances of the phrase "Retweeted by" with any text after it and does the same counting and sorting as the last command did.

cat Tweets.txt | grep -o 'Retweeted by.*' | sort | uniq -c | sort -n

Looking for Excessive Retweeters

Because retweets are listed by name and not the user handle, the results will look a bit different.  Once again you can see that the news services are retweeting a lot, and although I find their information valuable, there's quite a bit of duplication between the news stories they report, particularly "7News Brisbane" and "Nine News Brisbane".  So I'm starting there.  I prefer 7 News so I'm dropping 9 News.  I plan to do this slowly over time and not rush into it, cutting only a couple of accounts at a time and see how it goes.  There may also be problems dealing with Unicode names, but I'll leave that as an exercise for the reader to sort out.

Ultimately what I really want is something like this.  Twitter volume controls.

Twitter Volume Mockup

I'd like to be able to go into the list of people I follow and individually change how much of their feed I see.  I might be only interested in their tweets and not retweets or vice versa.  Alternatively I'd also like to be able to reduce the number of tweets I see.  An easy way to do this is to just randomly let through a certain percentage of tweets, but a heuristic method based on my twitter habits would be best.  I don't however want Twitter to decide for me, I want full control to tweak my timeline as I see fit.

This would also benefit Twitter.  They'd be able to get more fine grained feedback on what users think about the quality of a particular users tweets.  It would also improve the value of my timeline to me and I'd be more inclined to use their services.

Logging and Graphing Ping from the Linux Command Line

Lately I've been having a bit of trouble with my ISP.  For the past month or so, between the hours of about 6pm and midnight I can't connect to Twitter.  The start and stop time of the problem varies, and it only occurs intermittently.  Some nights are fine and you think things are fixed, then it starts over again.  I thought it was me at first, but this thread over at Whirlpool shows a growing list of frustrated users with the same problem.  I though TPG would've been able to find the problem straight away, but I was wrong.

The TPG representative on the forum listed some tests that he'd like us to perform to help track down the problem.  Ideally doing tests like ping, traceroute, and nslookup before and after the problem starts.  I don't know much about the infrastructure of the internet or how routing occurs, but I'm familiar with these commands and what they mean.  I'd like to help out, but because the problem was intermittent I kept missing the right time to run the tests, and after a week or so of trying I was fed up.

I had an ancient laptop sitting beside me running an outdated version of Ubuntu that I decided to put to use.  By automating the process of running the tests and logging the output I would be able to capture the exact moment of failure and hopefully pinpoint the problem.  After all, the internet without Twitter just isn't the same.

Recording the result of the tests was easy, a quick and dirty bash script did the job.


logger.sh

#!/bin/bash

DATE=$(date +%Y_%m_%e_%H_%M_%S).log    #get date YYYY_MM_DD_HH_MM_SS

printf "ping -c 5 twitter.com \n\n" > PING_twitter_com_$DATE
printf "%s\n" "$(ping -c 5 twitter.com)" >> PING_twitter_com_$DATE
printf "\n\n\n" >> PING_twitter_com_$DATE

echo $DATE

The script is simple.  It gets the date in a human readable sortable format writes the command to run as a string to a file named PING_twitter_com_YYYY_MM_DD_HH_MM_SS.log.  It then writes the result of the command to the file.  More tests can be added by replicating the above code for each command.  By issuing the following command

while true; do ./logger.sh; sleep 300; done;

logger.sh will be run every 5 minutes (actually it's every 5 minutes plus the time to run the tests) until stopped.  This will generate a log file for each execution containing the data we want.  An example is below.

PING_twitter_com_2013_04_18_20_17_59.log

ping -c 5 twitter.com

PING twitter.com (199.59.148.10) 56(84) bytes of data.
64 bytes from twitter.com (199.59.148.10): icmp_req=1 ttl=51 time=256 ms
64 bytes from twitter.com (199.59.148.10): icmp_req=2 ttl=51 time=256 ms
64 bytes from twitter.com (199.59.148.10): icmp_req=3 ttl=51 time=257 ms
64 bytes from twitter.com (199.59.148.10): icmp_req=4 ttl=51 time=257 ms
64 bytes from twitter.com (199.59.148.10): icmp_req=5 ttl=51 time=257 ms

--- twitter.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 256.601/257.101/257.432/0.464 ms

This is great, but it's a bit hard to see what's going on.  By extracting the average ping value and graphing it over time, a better picture of the problem can be created.  A command line graph can be generated from a directory of log files with the following command.

cat PING_twitter_com*.log | awk -F '[ /]' '/rtt/{ print $8 }' | gnuplot -persist -e "set terminal dumb 121 28; set yrange[200:400]; plot '-' with impulses title 'Ping (ms)';"

At first it seems a little complicated but once you break it down things become clear.  The required files are read with cat.  The files are read in order because the wild card operator used will guarantee that.  This is then piped to awk.  It looks for lines that match a pattern.  rtt is at the start of the lines that contain the average ping so I've set awk to look for that, but you could've used mdev or another string on the line you are looking for.  The field separators for awk are set with the dash F option to be space and forward slash.  This makes extracting the average ping easy.  By breaking the line up at spaces and forward slashes you can see that the average ping is the eighth field.  awk is set to print the eighth field as its output.  This is piped to gnuplot.

gnuplot needs to be set to use the terminal.  The "set terminal dumb" command does this.  Plotting the data from stdin is done with the plot '-' command.  This gives the basic info, but is can be spruced up a bit with some extra formatting, ie, changing the range, using impulses, enlarging the graph, and giving the data a title.  It's up to you to decide what looks best.

A graph of ping results to twitter.com every 5 minutes

As you can see from the graph above, I didn't encounter the problem tonight.  The average ping result remained remarkably stable at around 250 ms.  Who knows, they may have fixed the problem.  But i'll run this for the next couple of nights just to make sure.

Review Your Followers Using the Twitter API and Python

Over time, the list of people that I follow on Twitter has slowly grown and it's at a point where I need to go through and clean house to improve the quality of my time-line.  I know some of the accounts are inactive and some are companies I followed back who have since unfollowed me.  To make this job a bit easier I decided to put together a tool that could show me my relation to users that I'm connected to, and the time that they last updated their status.  There's probably already something out there to do this but by doing it myself I get a chance to sharpen my Python skills.

twitter_scalpal code

Sample output showing the least active users first

The tool is a simple Python script that takes a Twitter screen name of a target user as an input and then retrieves who they follow and who follows them.  For each user found it displays three things.

How the users are related to each other

          If the target user and the found user follow each other 'linked to' is displayed.
          If the target user only follows the found user 'following' is displayed
          If the target user is only followed by the found user 'followed by' is displayed.

The users screen name

The time that the user last updated their status

          The time of the last update in local time
          If the user is protected 'protected' is displayed
          If the user hasn't tweeted 'no tweets' is displayed

The script uses parts of the Twitter API that don't require authentication.  This means you can analyse any user that isn't protected but you will be unable to retrieve data on found users that are protected.  As this is for my own use I'm not too worried about implementing authentication as I only follow one protected user.  It shouldn't be too hard to adapt the script to use authentication.  The code is a little rough around the edges and may fail under certain circumstances, but generally it does the job.

To make the results easier to analyse they are sorted by the time that the user last updated their status, with least active accounts at the top.

As the twitter API is rate limited the script may take a while to complete.  A delay of 24 seconds is added after each request to prevent being blacklisted.  As a rough guess the script should take 1.25 seconds per follower and followee.

The output showing most active, protected, and users without tweets

The script acquiring data

# twitter_scalpal.py
#
# given a Twitter user name, display their
# followers and who they follow along with
# how they are related and the last time
# their status was updated

#! /usr/bin/env python
# -*- coding: utf-8 -*-


import sys
import argparse
import os
import shutil
import urllib2
import json
import math
import time
import email.utils
import datetime


#define constants
time_delay = 24    #time betweet requests to the twitter API
followerURL = "https://api.twitter.com" + \
              "/1/followers/ids.json?cursor=-1&screen_name="
friendURL = "https://api.twitter.com/1/friends/ids.json?cursor=-1&screen_name="
lookupURL = "https://api.twitter.com/1/users/lookup.json?user_id="
entityURL = "&include_entities=true"

#Create command line parser
parser = argparse.ArgumentParser(description='Analyse twitter users')
parser.add_argument('user', type=str, help='twitter user to analyse')
args = parser.parse_args()

userName = args.user


#delete previous data for user and and create a location for new data
shutil.rmtree(args.user,ignore_errors=True)
try:
    os.mkdir(args.user);
except OSError:
    print "System error"
    sys.exit(1)
    

#Get followers
print ""
print "getting followers"

try:
    followers_response = urllib2.urlopen((followerURL + userName)).read()
except urllib2.HTTPError, e:
    print "HTTP Error " + str(e.code) + ". Check if user exists"
    sys.exit(1)
except urllib2.URLError, e:
    print "URL Error " + str(e.args)
    sys.exit(1)

time.sleep(time_delay)


#Parse follower response and print follower count
followdat = json.loads(followers_response);
followers = followdat['ids']
print str(len(followers)) + " found"
print ""


#write followers to file
try:
    with open("./" + args.user + '/followers.json','wb') as f:
        f.write(followers_response)
except IOError:
    print "Couldn't write followers to file"
    sys.exit(1)


#Get friends
print "getting friends"

try:
    friends_response = urllib2.urlopen((friendURL + userName)).read()
except urllib2.HTTPError, e:
    print "HTTP Error " + str(e.code) + ". Check if user exists"
    sys.exit(1)
except urllib2.URLError, e:
    print "URL Error " + str(e.args)
    sys.exit(1)

time.sleep(time_delay)


#Parse follower response and print follower count
frienddat = json.loads(friends_response);
friends = frienddat['ids']
print str(len(friends)) + " found"
print ""


#write friends to file
try:
    with open("./" + args.user + '/friends.json','wb') as f:
        f.write(friends_response)
except IOError:
    print "Couldn't write followers to file"
    sys.exit(1)


#calculate number of unique contacts
contacts = list(set(followers + friends))
num_of_contacts = len(contacts)
print "calculating unique contacts"
print str(num_of_contacts) + " found"
print ""


#initialise variable
all_contacts = []


#calculate the number of contact requests to make (20 at a time)
num_of_contact_requests = (num_of_contacts-1+20)/20


#lookup user information
for i in range(num_of_contact_requests):


    #assemble a URL of 20 contacts to lookup in a single request 
    namerequest = ""
    namerequest = str(contacts[i*20])    #first contact to lookup
    for j in range(1,20):    #add remaining contacts.
        index = i*20 + j
        if(index < num_of_contacts):    #prevent out of bounds access
            namerequest = namerequest + "," + str(contacts[index])
    idurl = lookupURL + namerequest + entityURL


    #lookup contact information    
    print "getting contact details " + str(i+1)
    try:
        contacts_response = urllib2.urlopen((idurl)).read()
    except urllib2.HTTPError, e:
        print "HTTP Error " + str(e.code) + ". Check if user exists"
        sys.exit(1)
    except urllib2.URLError, e:
        print "URL Error " + str(e.args)
        sys.exit(1)

    time.sleep(time_delay)


    #parse contacts and add them to a list of all contacts
    contactdat = json.loads(contacts_response)
    for single_contact in contactdat:
        all_contacts.append(single_contact)


#collect and format data for display
output_list = []
for single_contact in all_contacts:


    #Assemble a string for the last status date.
    #Can be 'no tweets' or 'protected'
    is_contact_protected = single_contact['protected']
    status_count = single_contact['statuses_count']
    if ((is_contact_protected == False) and (status_count > 0)):


        #The following converts the UTC time to local time
        created_at = single_contact['status']['created_at']
        parsed_date = email.utils.parsedate_tz(created_at)
        date_string = datetime.datetime.fromtimestamp(
                      email.utils.mktime_tz(parsed_date)).strftime(
                      '%Y-%m-%d %H:%M:%S')

    elif (is_contact_protected == True):
        date_string = "protected"

    elif (single_contact['statuses_count'] == 0):
        date_string = "no tweets"


    #Create a space padded string of the contacts screen name
    contact_name = single_contact['screen_name']
    contact_name = contact_name + " " * (17-contact_name.__len__())


    #Create a string that describes the contacts relationship
    relationship_string = "following    " 
    if single_contact['id'] in followers:
        relationship_string = "followed by  "
        if single_contact['id'] in friends:
            relationship_string = "linked to    "


    #put the strings in a tuple and add it to a list
    contact_details = (date_string, relationship_string, contact_name)
    output_list.append(contact_details)

     
#sort the output list.  List sorted by first
#element of the tuple, the date string
output_list.sort()


#print the results
print ""
for contact_strings in output_list:
    print contact_strings[1] + contact_strings[2] + contact_strings[0]


#
#output = open("./" + args.user + '/contacts.json','wb')
#output.write(json.dumps(all_contacts))
#output.close()



#write contacts to file
try:
    with open("./" + args.user + '/contacts.json','wb') as f:
        f.write(json.dumps(all_contacts))
except IOError:
    print "Couldn't write contacts to file"
    sys.exit(1)

Sorting a Word List by Syllable with AWK

Download the Word List Files

Recently I've been thinking about changing my domain name to something more technology oriented and less personal.  I never intended to use my name for this site, my first priority was just getting something on-line, then I could worry about choosing the perfect domain name later.  There's only one problem with selecting a new site name, when it comes to things like this, I've got the creativity of a piece of mouldy toast.  So to help me along I thought I could use a bit of computer assisted inspiration in the form of a word list.

Before coming up with a name I already had a bit of an idea of what I wanted.  Preferably I'd like it to be 2 words, with a total of 3 or less syllables.  The last letter of the first word needs to be different from the first letter of the second word.  This prevents having a situation where double letters occur and makes it easier to communicate the site verbally.  For example if I said my site was "hot tech" would you type "hottech" or "hotech".  So basically I want a simple name that is unambiguous to type.  Perfect examples being adafruit, sparkfun, and makerbot.  These names are simple, have impact and hard to type incorrectly.  So that's what I'm trying to achieve.  I'm also trying my hardest not to do some lame rip off of something already out there, for example "arcjoy" instead of "sparkfun".

With these requirements in mind I thought that a good place to start would be a list of words that also indicated how many syllables are in each word.  The closest I could find was a word list from Project Gutenberg.  It's a list of words from the Moby Project with each word split into syllables by a delimiter character.  It's not perfect but it's massive and reasonably easy to work with.  The file does need to be processed slightly to make it more usable.

Unsurprisingly, this task is most efficiently done using the Linux command line and awk, a tool that can read files 1 line at a time and allows you to process them by breaking them up into fields.  So let's get started by downloading the file.  I'm going to demonstrate the commands using a small 10 line snippet from it.  When processing the actual file, commands were redirected to files.  Firstly we'll run a command to display all of the non printable characters.  This will show the line ending and delimiter characters.

cat -v mhyph.txt

display non-printable characters

The first thing you may notice is the ^M ending of the lines.  This is the control code for a carriage return and indicates that the file is formatted for windows.  To process the file with awk it needs to be converted to UNIX format, which only has a newline character at the end of a line, Windows files have a newline and a carriage return character.  If the carriage return isn't removed awk will interpret it to be part of the last field.  Printing this field would then return the cursor to the start of the line, leading to unexpected results.  The sub command of awk can be used to find and replace the carriage return characters at the end of lines (which can also be represented by \r) with nothing.

awk 'sub("\r$", "")' mhyph.txt

removing the carriage return character

We'll now run the cat command again to inspect the delimiters.

cat -v mhyph_unix.txt

Identifying delimiters

The documentation states that the delimiter is represented by the ASCII character 165.  This can be identified in the file as as an M code with the characters M-% representing ASCII character 165.  Calculating an M code can be done by adding 128 to the ASCII value of the character after the dash, in this case the ASCII value of % is 37.  Adding 128 to 37 gives 165, or 0xA5 in hex.

I'd like the format of the final output file to use only readable characters, so I'm using different field separators.  Each line will contain three main fields separated by an underscore character.  The first field will be the number of syllables in each word, the second field will be the actual word, and the last field will be the original word split into syllables using an equals sign as a delimiter.


The next step is where we actually process each line and is reasonably complicated.  In the "begin" part of the command we set the field separator to 0xA5, this lets awk know how to separate the input.  The output record separator is also set to nothing instead of the default new line character.  If this wasn't done a new line would be inserted after every print statement.  The next part of the command prints NF followed by an underscore, this is the number of fields in the input, which is the number of syllables in the word.  A for loop is then run which prints each field in the input. This prints the word, but removes the delimiting characters. An underscore is then inserted.  Next another for loop is run which does basically the same thing, except it prints and equals sign after each field except the the last field.  A new line character is then printed to finish off the line.

awk 'BEGIN{FS="\xA5"; ORS="";}{print ((NF) "_");  for (i=1; i<=NF; i++) print $i; print "_"; for (i=1; i<NF; i++) print ($i "="); print(($NF) "\n"); }' mhyph_unix.txt

Processing the file

Now we have a file that contains a list of words with their syllable count, along with how the word is broken up.  Using the sort command we can sort the file by the number of syllables in the word while preserving order.  The -t option indicates the field separator, -k1,1 tells sort to use only the first field to determine rank, -s makes sure that order is preserved, and -n specifies that when sorting, treat the field as a number.

sort -t _ -k1,1 -s -n mhyph_count.txt

Sorted file

The following command will filter the previous results for words with 2 or less syllables as this is all I'm after for my application.  Using awk, we set the field separator to an underscore character and print the second field if the first field is two or less.

awk -F _ '$1 <= 2 {print $2}' mhyph_count_s.txt

Words with 2 or less syllables

To increase the usefulness of this file and to aid creativity it would be better to randomise the file.  That way similar words will be split up and adjacent words will be dissimilar.  This can be done simply with the sort command and the -R option to randomise the sort order.

sort -R 1n2.txt

Randomised word list

There you go, a randomised list of words with 1 or 2 syllables.  Easy as that.  I don't know if this will help me in my search for a domain name, or inspiration will just hit me and I'll come up with something.  I not even sure that I'll change it yet.  It's a bit of a hassle.  Even if this doesn't help I can still see uses for this list in other places, maybe coming up with pass phrases or even generating a random Haiku.