Q: What skills are important for your job?

I think having a strong foundation in computer science is important. We work across several disciplines and being able to apply computer science concepts to help solve the various problems in a particular research area is important.

Managing expectations. It’s often the case that you work with users who don’t have a firm grasp of the technology and being able to explain the various components of a system in such a way that they understand both the capabilities and limitations is an important skill.

Problem solving. I think as a programmer you’re often confronted with a vast array of problems, whether they be bugs in code, design choices (what implications does a particular design choice have on the overall system?), reverse engineering legacy systems, data processing (how do you efficiently process terabytes of data?). Your ability to adapt to these problems and come up with effective solutions is a very important skill to develop. I’m not sure it’s a skill you ever really master but one that you are constantly fine tuning.

Q: What is your typical process for designing and developing software?

We tend to follow an Agile approach to software development (iterative approach: design, code, test, feedback, repeat). For any given project we first start with a requirements gathering phase. This includes meeting with the various groups involved in the project and getting a solid understanding of the scope. We then meet internally and come up with an initial design and set various milestones for the project. The next phase depends on the main focus of the project, because we are a research focused group we are often times working on a grant application or an already funded grant. In the first case we usually build a prototype of the system which serves as proof of concept should the grant get funded. In the case of an already funded project we follow a more rigorous development cycle involving a more formal design specification and development/staging/production environments for releasing versions of the system.

Q: What makes working for a university unique (as opposed to a private company)?

In a university setting (specifically research) there’s obviously more focus on cutting edge research areas. We’re exploring new topics that aren’t always well defined so there’s quite a bit of room for creativity. We also support researchers across many disciplines and sometimes you work on a project that doesn’t always end up getting funded. So there’s times when you put quite a bit of work into something that never gets off the ground.

In contrast, working for a private company ultimately comes down to being profitable. So design decisions tend to be heavily influenced by customers/investors and what effects the bottom line. In my experience the private sector is much more deadline driven as well. Usually there’s a strict release schedule to meet and a never ending list of features to be implemented.

Q: What do you like most and like least about your job?

I very much enjoy the problem solving aspects of programming. Tackling hard problems is something I enjoy and in this field there’s definitely no shortage of them. I would have to say the thing I like least about my job is having to occasionally work with Microsoft technologies :) I’m a big advocate of free software and I cringe at the thought of having to implement a piece of JavaScript code so that it works in I.E.

Q: What skills have you learned from being a software engineer? What has the job taught you?

I think one of the best skills I’ve learned from being a software engineer is system administration. I’ve always had a passion for tinkering around with different technologies and in doing so spent quite a bit of time building systems so that I could develop on. In the process I’ve learned a lot about system administration which has helped me out tremendously in various projects throughout my career. As a programmer it helps to know more than just how to write code.

Q: Any additional thoughts or comments?

Random advice: One of the most important tools for a programmer is a good editor. There’s a lengthy debate as to which editor is the best (I personally prefer Vim) but whatever one works for you, learn it inside and out.

We’re all good at securing small pieces of paper. I recommend that people write
their passwords down on a small piece of paper, and keep it with their other
valuable small pieces of paper: in their wallet.

I recently read an excellent blog post by John Graham-Cumming in which he presents a elegant system for writing down your passwords using a Tabula Recta. I was inspired by this concept so I created a tool called passtab which aims to provide a light-weight system for managing passwords based on his idea. This post is about the general usage of passtab and presents some of the password management capabilities. This is not your grandmothers password manager so if you’re looking for a nice GUI point and click application that’s easy to use you can stop reading right here. This is for hardcore folks who enjoy looking up their passwords in archaic tablets invented by ancient cryptographers with last names like Trithemius. For the impatient, you can grab a copy of the latest version on github.

Introducing passtab

passtab is a light-weight system for managing passwords using a Tabula Recta. passtab has two main features: 1. generating random Tabula Recta’s in PDF format for printing and storing in your wallet 2. fetching passwords from the Tabula Recta (password managment). These features are independent and you can use passtab to only generate PDFs or optionally make use of the password management features. One unique benefit is the ability to have both an electronic and paper copy of your passwords. You can download the binary release of passtab at github here. Unpack the distribution and run ./bin/passtab --help for a list of options. If the startup shell script doesn’t work you can run java -jar lib/passtab-uber.jar --help. The following sections illustrate some use cases of passtab.

Generate a random Tabula Recta in PDF

passtab can generate random Tabula Recta’s in PDF format.

$ ./bin/passtab --format pdf --output passtab.pdf
Jun 12, 2011 11:16:29 AM org.qnot.passtab.PassTab generate
INFO: Generating a random Tabula Recta (might take a while)...
$ ls *.pdf
passtab.pdf

Here’s an example PDF generated from passtab. You can now print this PDF out and store in your wallet!

How to use the Tabula Recta

Here’s a simple example (taken directly from the README), suppose we have the following Tabula Recta:

    | A B C D E F G H I J K L M N
  --|----------------------------
  A | _ u } I ` } R ) a < L : a A
  B | - o ( : p # O % . _ ; ' j L
  C | w c ( c y 2 h y ~ N O * > w
  D | o : R m L % V , d H r Y B j
  E | 9 , < 0 J p a o ) O w 0 w #
  F | C j i } i z 2 $ O R 5 @ T I
  G | Q - E m 8 N c / + u W Y V >
  H | , y } U Y i j i q w q c - 4
  I | K j W H e ; I ? E 7 H v 2 +
  J | g * 7 4 E } a h Y z < " : w
  K | . _ } I / J k 1 a D ^ ; p K
  L | ` < A L c z } } I P ? 4 y T
  M | F D < 8 < 0 R B t 9 X o B 2
  N | I r O E m o a + Y W w ; : 7

And suppose we want to get our password for logging into webmail at acme.com. We decide to use the first and last letter of the domain name as the start row/column of the password and we want a password 8 characters in length. So we start at the intersection of ‘A’ and ‘E’ and read off 8 characters diagonally resulting in the password: '#h,)RWc

Defining a scheme for selecting the starting row/column for a given password is completely up to the user and can be as simple or as complex as one desires. The direction for reading the password is also up to the user to define (left, right, diagonally, etc.). See John Graham-Cumming’s excellent blog post for more examples.

This method is slightly more complex than just writing down your passwords on a sheet of paper but the added complexity offers some advantages:

1. Can store all your passwords on a single sheet of paper
2. If someone steals this sheet of paper they’ll have a harder time figuring out what your passwords are
3. Allows you to use strong random passwords
4. If you want to change your passwords just re-generate a new Tabula Recta. Your scheme for selecting passwords can stay the same

passtab makes no assumptions about how passwords are read nor does it know anything about your scheme (unless you configure it). Now that you don’t have to remember long random passwords anymore what do you need to remember when using a Tabula Recta? Well first, you need to come up with a method for finding the starting position for a given password. In the example above this can be as simple as using characters from a domain/host name. But the beauty is you can be as creative as you want. A scheme that works for most of your passwords would probably be ideal but you can certainly generate multiple Tabula Recta’s if you like. Once you have a way of coming up with a starting location you need to define a method for reading off the password. In passtab this is called a sequence. In the example above we simply read 8 characters diagonally. But again you can be creative here. You could read 8 characters diagonally skipping every 3rd character, etc. Lastly, you’ll need to remember what to do if you hit the edge of the Tabula Recta before the end of the password. For example, if you start at Z:Z and want to read 8 characters diagonally you can’t because you reached the end of the Tabula Recta. In passtab this is called a collision. In this case we could just continue reading following the edge.

Using the Tabula Recta allows you to make use of long secure random passwords and only have to remember three simple things. You also have all your passwords on a single sheet of paper that fits in your wallet.

Custom Alphabets

In passtab, a Tabula Recta consists of two alphabets. The header alphabet and the data alphabet. The header alphabet is used for the row and column heading of the Tabula Recta and forms the basis for finding the starting location of the passwords. The data alphabet is used to generate the contents of the Tabula Recta and passtab will randomly pick characters from this alphabet using a cryptographically secure random number generator. By default, passtab uses a header alphabet of 0-9A-Z and a data alphabet consisting of all printable ASCII characters. It’s important to keep in mind that the data alphabet directly effects the entropy of your passwords. passtab allows you to customize these alphabets allowing you to generate any kind of Tabula Recta, for example:

$ ./bin/passtab -b A,B,C,D -a 'a,b,c,d,1,2,3,4,!,@,#'
Jun 12, 2011 10:24:26 PM org.qnot.passtab.PassTab generate
INFO: Generating a random Tabula Recta (might take a while)...
  A B C D 
A d 1 @ 4 
B c 4 @ 2 
C b 3 3 ! 
D 1 a @ 4 

Here’s a Tabula Recta using greek symbols as the header alphabet (here’s the example PDF):

$ ./bin/passtab -b 'Σ,Τ,Π,ρ,ϋ,ψ' -a 'a,b,c,d,1,2,3,4,!,@,#'
Jun 12, 2011 11:26:00 PM org.qnot.passtab.PassTab generate
INFO: Generating a random Tabula Recta (might take a while)...
  Σ Τ Π ρ ϋ ψ 
Σ 1 2 1 d d c 
Τ 1 2 b b @ c 
Π 1 # c 3 2 @ 
ρ 4 2 d 2 @ 3 
ϋ 2 3 b 1 ! b 
ψ d @ # c ! a

Password Management

So this is all well and great, but in reality it can be a huge pain to have to look up your webmail password in a Tabula Recta that’s on a sheet of paper in your wallet every time you login. For this reason, passtab has some optional features to help read passwords from the Tabula Recta. This allows you to have both a hard copy of the Tabula Recta in your wallet and an electronic version stored on your hard drive for quick access to your passwords. This obviously comes with some security considerations and care must be taken to protect the passtab database as you would any ssh private key for example. If someone got a hold of the passtab database file they could brute force your Tabula Recta. I ended up creating an encrypted thumb drive and store my passtab configuration and database files on it. You could also use gpg to encrypt it or any other method to protect it from the bad guys. This next section discusses the password management features of passtab.

First some definitions:

• Direction: a direction to move on the Tabula Recta. Valid values are N,S,E,W,NE,NW,SE,SW
• Sequence Item: a sequence item consists of a length and direction. For example, 12:SE would mean move 12 characters in the SE direction (diagonally)
• Sequence: a sequence is a list of sequence items. This allows you to define arbitrary sequences for reading passwords. For example, 4:SE,3:N,1:S would mean read 4 characters SE (diagonally) followed by 3 characters N (up) followed by 1 character S (down)
• Collision: a collision defines what directions to move if we hit the edge of the Tabula Recta before the end of the password. You can define more than one direction and they will be tried in order. For example, N,NE,E,SE,S,SW,W,NW would mean if we hit a wall try those directions in order until we’re able to move again

Generate a Tabula Recta in PDF and save to a passtab database

passtab can generate a Tabula Recta in PDF along with storing it in a passtab database. The passtab database is stored in JSON format and can be easily accessed outside of passtab (any language that can read JSON files). Again, you’ll want to store that JSON file someplace safe. For example:

$ ./bin/passtab --dbsave --name mypasstab
Jun 12, 2011 10:48:33 PM org.qnot.passtab.PassTab generate
INFO: Generating a random Tabula Recta (might take a while)...
$ ls mypasstab.*
mypasstab.json  mypasstab.pdf

Reading passwords from the passtab database

Once we’ve created our passtab database we can now fetch passwords by telling passtab the starting location and the sequence to read. For example, suppose we want to read a password starting at row ‘B’ and column ‘N’ and we want a password 10 characters in length reading diagonally:

$ ./bin/passtab -i mypasstab.json --getpass B:N --sequence 9:SE
o6,ZzH{e$@

Copy the password to the clipboard using xclip:

$ ./bin/passtab -i mypasstab.json --getpass B:N --sequence 9:SE --chomp | xclip

We used 9:SE as our sequence because passtab includes the character at the start location in the password. If we didn’t want to include this character we can optionally skip it like so:

$ ./bin/passtab -i mypasstab.json --getpass B:N --sequence 10:SE --skipstart
6,ZzH{e$@_

Define a list of directions to try in the event of a collision. This will try the directions N,S,E,W in order until we can move again. Here we start at Z:Z and can’t move SE (diagonally) so we try N (up) which works so we move N (up) until we hit another collision:

$ ./bin/passtab -i mypasstab.json --getpass Z:Z --sequence 9:SE --collision N,S,E,W
a((vy&0bV&

Conclusion

This post introduced a new tool called passtab for managing passwords using a Tabula Recta. I’m sure it has plenty of bugs so use at your own risk and if by chance you find it somewhat useful I’d be very interested in any feedback.

If you’d rather skip this epic post and dive right into the code you can browse it all over on github. There’s a README file and a Makefile which details how to run wp2print and includes a simple example.

Background and Assumptions

The idea for this project came about as my family has been writing a private blog that I want to be able to share with my children some day. I’ve often thought about giving them a copy of the blog when their much older and wondered what would be the best way to preserve the content ensuring it’s viewable years down the road. I thought by creating a physical copy of the blog I could have something tangible to pass down through the generations. Using the services provided by companies like Lulu and Blurb printing a book is as simple as uploading a PDF and designing a cover. Having worked in the publishing industry in a former life I had some experience generating PDF books so I looked forward to the challenge.

As my goal was to create a book I needed to convert the content of the blog into an intermediate format which represented the book and could then be used to generate a PDF file. As I had a good amount of experience slinging around DocBook, my general idea was to export the content from WordPress and convert it into DocBook. Then converting DocBook into a PDF is fairly straightforward using the wonderful DocBook stylesheets and Apache FOP.

The first obvious challenge when converting a blog into a book is deciding how to go about organizing the blog posts into chapters and sections. Our family blog was authored in such a way that each post had only one tag (category) and most importantly all posts were tagged chronologically. Meaning that all posts in a given category were sequential. For example, posts 0-5 are all tagged with “tag1″, posts 6-10 are all tagged with “tag2″, and so on. You can probably see where this is going. Having authored the blog in this format allowed me to easily use each tag as a Chapter and each post appeared as a section within that chapter. If your unable to make these assumptions about your blog (and most likely you won’t be able to) just keep this in mind as we delve into the code later on. You would need to modify the script I wrote and add in the appropriate logic to slice your blog posts up into chapters/sections or however you’d like to structure your DocBook file. I experimented with just making each post a chapter (and even a series of DocBook articles) which isn’t a bad option however depending on the number of posts you may want to consider omitting the table of contents.

A few other assumptions I made:

• All posts were written by the same author so I omitted displaying any author information. Easy enough to add in if needed
• All comments were excluded from the book. Comments are an important part of any blog but in this case my blog didn’t have very many comments. I was most interested in the content of the post only and decided to omit any and all comments. These could certainly be added in but some thought would be needed on which DocBook element to use for structuring them within the book.
• There was one page (not a post) with the title “About” that I used as the preface for the book. This can be any post/page or omitted completely if desired.
• Access to the WordPress code that runs the blog. This won’t work if your blog is hosted for example at WordPress.com. You’ll need to export your blog and run it on your own server.

Here’s a brief outline of the entire process. I’ll go over each step in detail in the next section.

1. Convert WordPress content to DocBook – using PHP and some XSLT
2. Convert DocBook to XSL-FO – using DocBook stylesheets
3. Convert XSL-FO to PDF – using Apache FOP
4. Upload PDF file to Lulu and order print book

Convert WordPress content to DocBook

First step was to convert the blog into DocBook. This was by far the most challenging step. My first attempt was to use the Export feature in WordPress which dumps the entire contents of your blog in XML format (WordPress eXtended RSS) and write an XSLT to convert into DocBook. This turned out to be slightly harder than I anticipated because of how the content of each post was formatted in the WordPress XML dump. It appeared to be in the native format WordPress uses to store the post in the database and I didn’t want to have to write a custom WordPress post renderer for DocBook. I decided to instead write a fairly simple PHP script which used the WordPress API to render each post in HTML just like it normally would if someone visited the site, then convert the HTML to DocBook. I found converting the HTML to DocBook was slightly easier than having to parse the native WordPress format. I did this in two steps, first I wrote a PHP script to generate a quasi-DocBook file which uses the WordPress API to embed the HTML content of each post within a <section/>. Then I wrote an XSLT which transforms the quasi-DocBook and embedded HTML into a final valid DocBook file. The main PHP code is here. You’ll need to change the include paths in config.php to point to your WordPress installation (see the Makefile for a complete example). The XSLT is here. It looks for various HTML tags that appear in my blog and converts those to valid DocBook elements. I built up the XSLT by trial and error. I first just rendered the quasi-DocBook generated from the PHP script as PDF. The DocBook stylesheets have a nice feature in that any invalid DocBook elements it encounters are highlighted in red in the resulting PDF. By iterating through the invalid elements I was able to add the correct templates to my XSLT to account for all HTML tags found in my blog posts. You’ll most certainly need to modify this XSLT file to suite your specific needs but should serve as a decent starting point. Here’s an example of the HTML generated by WordPress for an image included in a blog post:

<div id="attachment_155" class="wp-caption aligncenter" style="width: 310px">
  <a href="/wp-content/media/2008/11/image.jpg">
        <img class="size-medium wp-image-155" title="image title" src="/wp-content/media/2008/11/image-300x218.jpg" alt="image alt" width="300" height="218"/>
  </a>
  <p class="wp-caption-text">This is a description of the image</p>
</div>

Which then gets converted to a DocBook mediaobject element:

<para>
  <mediaobject>
    <imageobject>
       <imagedata align="center" fileref="images/2008/11/image.jpg" width="4.0in" depth="3.0in" scalefit="1" format="JPG"/>
    </imageobject>
    <caption><para>This is a description of the image</para></caption>
  </mediaobject>
</para>

A note about images…

Care must be taken to ensure any images you want included in the book are print ready. I ended up having quite a few images in my blog that I wanted to include in the final PDF which required some extra work to get them ready for printing. For best results you’ll want make to be sure the resolution of your images are at least 300ppi (pixels per inch). See this post on Lulu. For example, if your image is 600x600px and you set the resolution to be 300ppi, the printed image will be roughly 2x2in. In my case I was printing a 6×9 book and after factoring in margins/spine/bleed etc. I calculated the maximum print size I wanted each image was 4x3in (as defined in the DocBook XML element <imagedata width="4.0in" height="3.0in"/> in the above example). As most of the images were pictures, this print size ended up being large enough so the photo was still viewable but small enough to allow for 2 images per page. This meant that the minimum size (in pixels) each image had to be was 1200x900px. The problem was when we uploaded pictures to our blog we had WordPress resize them to 500x400px (from their original size of 2816x2112px from the camera). Fortunately, I still had the original image files which I collected and used in the final PDF. Something to keep in mind if you have images (especially photographs) in your blog that you want printed. I ran into another edge case with the images which required a little bit of imagemagick. I had a few important pictures that were taken with photo booth on a mac in which the original size image was a mere 640x480px. I knew the print version of the images would look dreadful so my only option was to resample them to a higher resolution. This can easily be accomplished using imagemagick’s convert command:

$ convert -resample 300x orig.jpg hires.jpg

In summary, be sure your images are high enough resolution for printing. It’s definitely worth the extra work. I had roughly 100 images in my blog and all of them turned out really nice in the final print book. I was quite impressed with the quality of Lulu’s printers.

DocBook –> XSL-FO –> PDF

Converting DocBook to PDF was fairly straightforward using two excellent projects DocBook stylesheets and Apache FOP. I won’t cover how to install them on your platform and refer you to the excellent INSTALL guides at the respective sites. If you happen to be running Ubuntu using the stock packages should work fine. Simply run aptitude install fop docbook-xsl and you should be all set. The basic goal for this step was to use the DocBook XSL FO stylesheets to convert the DocBook created from the previous step into XSL-FO which can be fed into Apache FOP for conversion into PDF. This step required that an XSLT processor be installed such as xsltproc (libXML), Saxon, Xalan, etc. I used xsltproc and can easily be installed on Ubuntu aptitude install xsltproc. After running xsltproc I passed the resulting XSL-FO output into Apache FOP to generate the final PDF. For more details see the Makefile. Here’s the basic commands:

$ xsltproc /path/to/docbook-xsl/fo/docbook.xsl docbook-final.xml > book.fo
$ fop book.fo book.pdf

The DocBook XSL FO stylesheets provide a generous number of parameters for customizing the resulting FO. The default parameter settings produce a very nice looking PDF but if you like to tweak things there’s no shortage of knobs to turn. As I ended up printing my book with Lulu there were a few specific customizations that were required. First I was interested in printing a US Trade 6×9 inch hard cover book so the default page width/height needed to be set accordingly. Some other tweaks I made included adjusting the margins slightly to provide some extra room on the spine edge of the book, customizing the table of contents to only include the chapter/sections, and customizing the indentation of chapters and sections (in this case I didn’t want any indentation). Here’s the resulting xsltproc command with the custom parameter settings:

    xsltproc \
    --stringparam page.width 6in \
    --stringparam page.height 9in \
    --stringparam page.margin.inner 1.0in \
    --stringparam page.margin.outer 0.8in \
    --stringparam body.start.indent 0pt \
    --stringparam body.font.family  Times \
    --stringparam title.font.family Times \
    --stringparam dingbat.font.family Times \
    --stringparam generate.toc 'book toc title' \
    --stringparam hyphenate false \
    /path/to/docbook-xsl/fo/docbook.xsl \
    docbook-final.xml > book.fo

A note about Fonts..

The last and most important configuration I made was with fonts. Lulu requires fonts to be fully embedded which means any font you use in your PDF must be embedded (the font files are included directly in the PDF file) or else they will reject the PDF. Embedding fonts is supported by Apache FOP but requires some custom configuration. First I had to decide which font to use. Fonts can be really tricky and I didn’t want to get too fancy. Using a single font for the entire book was fine with me and I decided to stick with a traditional Times New Roman font. I ended up using the FreeSerif TrueType font from GNU FreeFont. It was already installed on my Ubuntu machine and very easy to embed with Apache FOP. By default these fonts are installed in /usr/share/fonts/truetype/freefont/.There’s lots of other free fonts out there that you could use including the Liberation Fonts and even the Micro$oft True Type Core Fonts which can be installed on Ubuntu by running aptitude install msttcorefonts. To configure Apache FOP to use GNU Free Fonts and embed them into the final PDF I created a file called userconf.xconf with the following lines:

<?xml version="1.0"?>
<fop version="1.0">
<renderers>
   <renderer mime="application/pdf">
      <!-- Full path to truetype fonts to be embedded in PDF file -->
      <fonts>
        <font embed-url="file:///usr/share/fonts/truetype/freefont/FreeSerif.ttf">
          <font-triplet name="Times" style="normal" weight="normal"/>
        </font>
        <font embed-url="file:///usr/share/fonts/truetype/freefont/FreeSerifBold.ttf">
          <font-triplet name="Times" style="normal" weight="bold"/>
        </font>
        <font embed-url="file:///usr/share/fonts/truetype/freefont/FreeSerifItalic.ttf">
          <font-triplet name="Times" style="italic" weight="normal"/>
        </font>
        <font embed-url="file:///usr/share/fonts/truetype/freefont/FreeSerifBoldItalic.ttf">
          <font-triplet name="Times" style="italic" weight="bold"/>
        </font>
      </fonts>
   </renderer>
</renderers>
</fop>

Then ran fop passing the -f option like so: fop -f userconf.xconf book.fo book.pdf. Note the <font-triplet name="Times" /> attribute must match the body.font.family Times XSLT parameter passed to xsltproc command.

Simple Example

All the code described in this post is available on github. I also include a simple example to demonstrate the entire conversion process and provide some sample PDFs to see how final book renders. I created a simple test blog consisting of Shakespeare’s Sonnets I thru X and exported the content in WordPress eXtended RSS so you can then import into a fresh install of WordPress. I tested using the latest version of WordPress at the time of this writing (v3.1). To try it out yourself download the code for wp2print and read thru the README file which outlines all the gory details. The Makefile outlines the general process and should provide a good starting point for experimenting. Here’s some sample PDFs that were rendered from the example Shakespeare blog:

• Book with Chapters and Sections
• Book with all posts as DocBook Articles
• Raw DocBook output

Conclusion

With the help of a few simple scripts it’s possible to create a high quality print ready PDF book from a WordPress blog. Depending on the content of the blog you’ll most certainly need to tailor these scripts to suite your specific requirements. The main challenges are figuring out how you want to organize your blog posts into the framework of a book and then modifying the XSLT templates to convert the WordPress html markup of your blog into valid DocBook elements. The services offered by print on demand publishers such as Lulu provide an easy way to turn the resulting PDF into a high quality paper book.

$ ssh some-host
$ perl script-that-chugs-along.pl
$ Ctrl-Z (suspend)
$ bg (put it in background)
$ disown -h
$ logout

The disown command allows you to remove jobs from the list of active jobs associated with your login shell. Here’s an excerpt from the bash man page:

Without options, each jobspec is removed from the table of active jobs. If the -h option is given, the job is not removed from the table, but is marked so that SIGHUP is not sent to the job if the shell receives a SIGHUP.

My hope was to write about my experience developing applications for the device but it seems as though I was one of the unlucky users who suffered from the infamous buzz. And I don’t mean buzz in a good way :) Apparently this didn’t effect every user and was more prevalent when using certain GSM bands. However random it may have been for some users it was an absolute show stopper for me and rendered the device pretty much unusable as a phone. Anytime I placed a call, the person on the other end heard a horrible loud buzzing sound. After a while Openmoko finally tracked down the source of the buzzing to a hardware problem and published some instructions on how to mod the device (known as the big-C rework). Unfortunately it requires some fairly advanced SMD-soldering skills and is way out of my league. As far as I know, Openmoko never really offered much help in resolving the issue for people who had already purchased the device with the hardware flaw (rev A5/A6) and were unable to preform the necessary surgery.

Despite the lacking phone functionality the Freerunner itself is still pretty cool. I’m sure I’ll be able to think of some other fun projects to use it for especially since it runs Debian.

First Boot

There’s several distributions for the Freerunner which can get quite confusing but the one that comes stock with the Freerunner is referred to as 2007.2. First time booting up the Freerunner you’re presented the home screen for 2007.2. You can also boot into NAND and NOR flash which allows you to update the kernel, root filesystem and the boot loader (U-Boot).

My first mission was to ssh into the device. Followed the instructions on the wiki for setting up USB networking. By default the IP address of the Freerunner is 192.168.0.202. On the desktop side you first have to ifconfig the usb0 interface and setup the correct routes. Here’s the script I run on my desktop after connecting the Freerunner:

#!/bin/bash

/sbin/ifconfig usb0 192.168.0.200 netmask 255.255.255.0
/sbin/route add -host 192.168.0.202/32 dev usb0

One extra step I had to do was configure my firewall to allow connections to/from usb0. I’m running Ubuntu hardy 8.04 and using Firestarter. Open up Firestarter:

• Preferences -> Firewall -> Network Settings
• Set ‘Local network connected device’ to: Unknown device (usb0)
• Check ‘Enable internet connection sharing’

Verified usb0 network connections:

$ ping -I usb0 192.168.0.202
$ ssh root@192.168.0.202

Once connected to the Freerunner next step was to get the date to display on the home screen. To do this I just followed the instructions on the wiki for customizing the today page (run these commands on the Neo):

# dbus-launch gconftool-2 -t boolean -s /desktop/poky/interface/reduced false
# /etc/init.d/xserver-nodm restart

Here’s a screenshot of the home screen:

Upgrade Software

Once I was able to successfully ssh into the Neo and verifed that I could also connect to the internet from the Neo I wanted to upgrade to the latest software release. To do this you use opgk (package management system based on Ipkg). The first time you upgrade from the software release shipped with the Neo you have to first upgrade dropbear (ssh server) from the terminal on the Neo, then you can ssh back into the Neo and upgrade the rest of the software:

# opkg update
On the Neo, open Terminal and run: # opkg install dropbear
Then ssh to neo and run: # opkg upgrade

At this point I rebooted and inserted my T-Mobile sim card and microSD card. Once back at the home screen it showed I was registered to the T-Mobile network and I opened up the dialer app and placed my first call!

Set up Timezone and correct date/time

To fix the timezone run this from the Neo:

# opkg install tzdata tzdata-americas
# ln -sf /usr/share/zoneinfo/America/New_York /etc/localtime
# /etc/init.d/xserver-nodm restart

To set the correct time using ntp run:

# opkg install ntpclient
# ntpclient -s -h pool.ntp.org
# hwclock --systohc

WLAN

Next up was connecting the Neo to my wireless LAN. The wireless interface on the Neo is eth0. First have to make sure WLAN device is turned on which it seemed to be by default when you first boot. You can check this by holding down the power button for a few seconds which should pop up a menu showing the state of the various devices. Here’s the script I use to connect the Neo to my WLAN:

#!/bin/sh

/sbin/ifconfig eth0 down
/sbin/ifconfig eth0 up
/sbin/iwconfig eth0 key restricted 'xxxxx'
/sbin/iwconfig eth0 essid 'xxxx'
/sbin/udhcpc eth0

GPS

tangoGPS rocks. This app is amazing and it worked right out of the box. Followed the directions on the wiki to get it up and running. There was an issue getting a fix with the SD card installed but by the time I tried this out they already had a kernel update which fixed the issue. I had no problem getting a fix and my TTFF was 35s with the SD card in. Here’s some screenshots of tangoGPS in action:

I also installed and ran AGPS Test which is a program for testing out GPS on the Neo. It shows some nice graphs of the various satellites you’re currently connected to and their signal strengths:

Bugs/Issues

Overall I was impressed by how much I was able to get working the first time around however there’s definitely a few issues I came across. The most concerning was the GSM buzzing during phone calls. On the Neo side everything sounds fine but the person on the other end hears a very loud buzzing noise. Here’s the latest update from the hardware list regarding the issue. I tried tweaking the various alsa settings in /usr/share/openmoko/scenarios/gsmhandset.state with some luck but still wasn’t able to find the right balance to completely eliminate the buzzing. Still trying to wrap my head around which alsa settings do what but I found playing with alsamixer during a live call to be helpful. The basic procedure goes something like this:

1. ssh to FreeRunner
2. Make a phone call
3. While call is in progress run alsamixer
4. Tweak settings to minimize buzzing/echo
5. While call is still in progress run: $ alsactl store -f gsmhandset-test1.txt

Now you can diff this new file against the original (/usr/share/openmoko/scenarios/gsmhandset.state) and see which settings were changed. This is really the only thing holding me back from using the Neo as my primary phone so I look forward to a possible fix.

I found using the Terminal on the Neo rather clunky due to the lack of characters available on the keyboard. For example there’s no <TAB> or ‘/’. I’m sure there’s ways to customize the keyboard. Looks like only vi is available by default on the Neo so I plan on seeing if I can find a vim package (.ipk) or figuring out how to compile vim for the Neo.

Here’s a few pics of what the Neo looks like all boxed up..

The Neo came bundled with a USB cable, a nifty stylus + laser pen, AC adapter, battery, and a 512mb microSD card.

The hardware specs for the Neo (GTA02) are impressive, GSM, WiFi, Bluetooth, GPS. All that packed into a full blown Linux distro that fits in the palm of your hand.

I’ll be posting more about my experiences using the Neo as my daily phone. More to come..

First step lets create a test project:

$ mvn archetype:create -DgroupId=org.qnot.example -DartifactId=hello-world
$ cd hello-world

Next add a few dependencies to the project. In this example we’ll add a few libraries from jakarta commons. The <dependencies/> section in the pom.xml should now look like this:

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>commons-cli</groupId>
      <artifactId>commons-cli</artifactId>
      <version>1.1</version>
    </dependency>
    <dependency>
      <groupId>commons-lang</groupId>
      <artifactId>commons-lang</artifactId>
      <version>2.3</version>
    </dependency>
  </dependencies>

Create a META-INF/ directory to store the MANIFEST.MF file which defines the main class in the executable jar.

$ mkdir -p src/main/resources/META-INF/
$ echo 'Main-Class: org.qnot.example.App' > MANIFEST.MF 

Create a src/assemble directory to store the assembly descriptor files

$ mkdir src/assemble

Next we’ll create the actual assembly descriptor file which defines how to package up the jar. Create the file src/assemble/exe.xml with the following xml:

<assembly>
  <id>exe</id>
  <formats>
    <format>jar</format>
  </formats>
  <includeBaseDirectory>false</includeBaseDirectory>
  <dependencySets>
    <dependencySet>
      <outputDirectory></outputDirectory>
      <outputFileNameMapping></outputFileNameMapping>
      <unpack>true</unpack>
      <scope>runtime</scope>
      <includes>
        <include>commons-lang:commons-lang</include>
        <include>commons-cli:commons-cli</include>
      </includes>
    </dependencySet>
  </dependencySets>
  <fileSets>
    <fileSet>
      <directory>target/classes</directory>
      <outputDirectory></outputDirectory>
    </fileSet>
  </fileSets>
</assembly>

Inside the <dependecySets/> is where you can add all the libraries you’d like to include in the uberjar. These must also be defined in your pom.

Finally, add the maven-assembly-plugin to the pom:

  <build>
    <finalName>hello-world</finalName>
    <plugins>
      <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <configuration>
          <descriptors>
            <descriptor>src/assemble/exe.xml</descriptor>
          </descriptors>
          <archive>
            <manifestFile>src/main/resources/META-INF/MANIFEST.MF</manifestFile>
          </archive>
        </configuration>
      </plugin>
    </plugins>
  </build>

To run the assembly and build the executable jar:

$ mvn assembly:assembly
$ java -jar target/hello-world-exe.jar
Hello World! 

I tested the hello-world example using the latest Maven release (2.0.8) and maven-assembly-plugin-2.2-beta-1. If you run into any issues try and update your Maven plugins by running:

$ mvn -U compile

You can download the example hello-world project here.

Dia is a program for creating diagrams and for this exercise we’ll be creating UML diagrams from within Dia. We’re also going to use a perl script called tedia2sql which will transform our Dia files directly to SQL for our target database. What’s also nice about creating database schemas this way is that you can generate SQL for multiple target databases without the maintenance overhead.

First off, install a copy of Dia as well as tedia2sql. You can download tedia2sql here and download Dia here. I’m not going to cover the install in this post but there should be packages available for most Linux distro’s and if your not running Linux now is a great time to start!

Fire up Dia, create a new diagram and save it as “employee.dia”. Select the UML sheet from the drop down list. We’ll be using various UML objects to represent tables and definitions that make up our database. Here’s a quick overview of the main UML objects and their usage:

Class represents a table in the database. A Class has a name which corresponds to the name of the table and attributes which map to the columns of the table. Attributes in classes can have visibility (public, private, protected). Protected attributes are primary keys. More on classes later. Component is a special object that lets you define a list of default values to be inserted into a table. These are equivalent to hard coding “insert into ..” statements in your SQL files. Small Package represents a typemap. Typemaps are used for adding custom SQL types such as MySQL tinyint.

Now lets create our first table. Select the “Class” object icon from the UML sheet and click inside the diagram editor window. This will add the class to your diagram. Now right click on the new class and select “Show Properties”. This will bring up a rather large and complex property window for the UML class you’ve just created. We’re only going to customize a few properties outlined below.

First, under the “Class” tab enter the name of the table (employee) in the “Class Name: ” field. Next click on the “Attributes” tab and enter in the columns of the table as attributes. In the “Name: ” field enter in the column name. In the “Type: ” field enter in the SQL type for the column. If you want to support multiple database platforms try to be generic here and only use ANSI SQL 1992 or else use a typemap. More on typemaps later. In the “Value: ” field enter any default values for the column. For example, this is where you could add in “not null”. If the column your adding is a primary key then don’t put “not null” here and instead select “Protected” for the “Visibility: ” field. Not null will automatically get added to all primary keys on output generation. The screen shot below is an example of the Class property editor window.

Next, repeat the process for as many tables as you need in your database. For this example, create two tables “employee” and “department” with the columns as shown in the screenshot below (note the ‘#’ in front of an attribute indicates that it’s visibility is “Protected” thus making it a Primary Key):

This is a simple example of a database which stores employee data along with the department they belong to. Now suppose we have a default list of departments that we’d like to load into the department table when our database is created. To do this we’ll use the “Component” object from the UML sheet. Select the “Component” object and add it to your diagram editor window. Right click and select “Properties”.

In the “Stereotype: ” field you basically enter in the first part of an “insert into ..” SQL statement. For this example, to insert a default list of departments we’d normally write the following SQL:

insert into department (department_id, name) values (1, 'Marketing');
insert into department (department_id, name) values (2, 'Production');
insert into department (department_id, name) values (3, 'Design');

So in this case, for the “Stereotype: ” field we’d enter in “department (department_id, name)”. Click OK to close the properties dialog. Now in the Component object box enter in the values you’d like to insert, one per line. Here’s what our diagram should look like now:

Now lets add in a typemap. Typemaps are used for when you’d like to configure custom types which are specific to a database platform. A good example of this is MySQL auto_increment. This is a feature specific to MySQL and not supported in all databases. Lets suppose we’d like to have a primary key column which gets auto incremented upon each insert. But we also want to support both MySQL and Apache Derby without having to maintain separate SQL files. To achieve this we’ll create a custom typemap and define the specific SQL for each target db.

Typemaps are created using a Small Package so select the “Small Package” object and add it to your diagram editor window. Right click and select “Properties”.

In the “Stereotype: ” field you enter in the target database. So for example “mysql: typemap”. For a list of supported target databases see tedia2sql –help. Some common ones are: postgres, mysql, sybase, oracle, db2, and innodb. Click OK to close the properties dialog. Now inside the Small package box you can enter custom types one per line. For this example we’ll create a custom type with a name of “identity”. This means that when we generate our SQL files, tedia2sql will replace the column type with our custom type. Here’s a few examples:

identity: int unsigned auto_increment
bigid: bigint unsigned auto_increment

identity: int generated by default as identity
bigid: long generated by default as identity

Repeat the process for Apache Derby using “db2: typemap”. Now that we’ve defined our custom identity typemap we can change the employee_id column from type “int” to type “identity”. We can also do the same for our department table. Now these columns will be auto incrementing columns and depending on which target database we select on output the correct SQL will be generated. Our diagram should now look like this:

At this point we have created our database schema and defined a default list of departments to be inserted. We also created a typemap called “identity” which defines auto incrementing columns for both MySQL and Apache derby. Now lets generate the SQL files to create the database using tedia2sql. Before we run tedia2sql we need to apply a small patch to the tedia2sql script. This fixes a very small formatting issue in tedia2sql related to the typemaps we created. Edit the tedia2sql script (/usr/bin/tedia2sql) and comment out the line inside the parseTypeMap(..) subroutine that looks like this:

$defStr =~ s/s//g; # ignore spaces
     -- change to --
#   $defStr =~ s/s//g; # ignore spaces

or just apply the patch I created here (works for version 1.2.12):

$ patch /usr/bin/tedia2sql < tedia2sql-typemap-1.2.12.patch

Now we can generate the SQL files for our target databases:

$ tedia2sql -i employee.dia -o employee-mysql.sql -t mysql -d
$ tedia2sql -i employee.dia -o employee-derby.sql -t db2 -d

This will generate two SQL files for building the database in both MySQL and Apache derby. If you view these files you can see how tedia2sql handled our typemaps for the auto incrementing columns in each target database:

MySQL

-- employee
create table employee (
  employee_id                int unsigned auto_increment not null,
  department_id             int not null,
  first_name                varchar(255) not null,
  last_name                 varchar(255),
  start_date                date,
  constraint pk_Employee primary key (employee_id)
) ;

Apache Derby

-- employee
create table employee (
  employee_id                int generated by default as identity  not null,
  department_id             int not null,
  first_name                varchar(255) not null,
  last_name                 varchar(255),
  start_date                date,
  constraint pk_Employee primary key (employee_id)
) ;

You can download the files generated as well as the employee.dia file I used in this tutorial here:

• employee.dia
• employee-mysql.sql
• employee-derby.sql

You can test the SQL files generated by tedia2sql and create the employee database as follows:

MySQL

$ mysql -u user -p dbname < employee-mysql.sql

Apache Derby

$ java -cp derby.jar:derbytools.jar \
       -Dderby.system.home=/path/to/dbroot \
       -Dij.protocol=jdbc:derby: \
       -Dij.database='employee;create=true'
    org.apache.derby.tools.ij employee-derby.sql

A quick Makefile will help out a lot in testing out your database schema:

all:
      tedia2sql -i employee.dia -o employee-mysql.sql -t mysql -d
      tedia2sql -i employee.dia -o employee-derby.sql -t db2 -d
clean:
      rm -f *.sql

I’ve found that for small to moderate size projects creating database schemas in Dia and using tedia2sql for SQL generation to be a lifesaver. You can also make use of Dia’s export feature to export your database schema to a number of different formats such as jpg, png, eps, tiff, etc. I really only scratched the surface of what can be done using these great tools. tedia2sql has support for lots of cool features like indexes, foreign key constraints, views and much more. For more information check out using tedia2sql and Dia’s manual.

Happy Diagramming!

I cron a script to download the latest cloud maps every 4 hours or so. The xplanet command I run is as follows:

xplanet -origin sun -north orbit \
        -config xplanet.conf -label \
        -marker_file brightStars \
        -target earth \
        -latitude 22 \
        -longitude -78 \
        -radius 30 \
        -labelpos +30+30

Here’s my xplanet.conf:

[earth]
cloud_map=clouds_2048.jpg
magnify=20

[moon]
magnify=20