Month: January 2014

by

IMAP, I Stab At Thee

I have a love-hate relationship with IMAP (let’s just ignore for the moment how weird it is to have any sort of emotional engagement with a protocol specification). IMAP stands for “Internet Message Access Protocol” and it is a powerful way for client programs to read and manage the messages of an E-mail account. The 107 page definition as laid out in RFC 3501 is not trivial – it is a complex process that can be tricky to get right. This is especially true for client writers as the IMAP server requirements can be very flexible, which in turn requires clients to adapt to a variety of possible implementations. There are also a slew of extensions with their own RFCs, just to keep the water muddy.

My loathing adoration for IMAP started innocently enough. I was getting involved in Open Source software development and was contributing to the Squirrelmail project. Squirrelmail is a webmail program written in PHP, which happens to use IMAP for E-mail account access. My first official submission to this project was on February 20th, 2002, and was completely unrelated to the protocol. However as time wore on, I became more involved in the project. I started digging deeper into the code base. Deeper and deeper. This is where I came face to face with the beast I would wrestle with for the next decade. It was like having my very own virtual Balrog. “BUGS SHALL NOT PASS!“.

PHP has native functions to communicate with IMAP servers, but back in those days they were not always available, and they had a reputation for being unreliable. Squirrelmail had its own client library, using PHP to read and write to the listening socket of the IMAP server. Seems simple, no? Turns out it’s not, but I was fascinated with this aspect of the code. I started spending a lot of time digging into the RFC to ferret out the nuance of one situation or another. I subscribed to esoteric IMAP mailing lists and tried to keep up with the discussion. Soon I had a hard copy of the RFC with me everywhere I went. Not long after that, IMAP commands and responses started streaming through my dreams like ASCII dragons searching for the tender flesh of programmers to feast on.

Leveraging the arrogance of immaturity, I left the Squirrelmail project about a year later, and started my own much less popular webmail project called Hastymail, which I still maintain. I wrote an initial IMAP client implementation in 2003, then again as a part of a complete rewrite to “Hastymail2” in 2008.  Since that time I have just been bolting on features and fixing bugs (in that order), but I have been toying with a new idea recently. All I needed was an opportunity. As luck would have it, I found myself on a long plane trip with some battery life in my laptop, and more importantly that always elusive motivational impulse to jump into something I had not looked at in a long time.

My idea was to decouple the IMAP class from the core Hastymail code so that it could be easily used in any PHP program (hardly breaking new ground here I know, I will get to that). The class was reasonably self-contained, but it did directly use aspects of the Hastymail framework in places, and those areas would need to be marked as deleted then expunged (that’s an IMAP joke). This code was also PHP4 compatible, meaning it was only a “class” in the most basic sense of the word, if at all. I decided to beat it around the head with a PHP5 OOP stick for a while, to see if a better overall structure for the logic could be put in place without ripping too much apart.

I’m genuinely surprised to say it was a pretty successful effort. At some point in the process my mindset evolved from “let’s see if this can work” to “must finish at all costs before thinking about anything else“, and it became a near obsession to reach a particular point of completeness. As to the not breaking new ground issue: I’m well aware of that. I have no illusions that the world needs another PHP IMAP library, or even this one in particular. But it was fun to build, and there is a greater than zero chance that at least one other person on the planet might find it useful – if even as an example of what not to do – so why not share. You can see all the gory details in the newly created github repo.

https://github.com/jasonmunro/hm-imap

Even with all the recent re-factoring there is still plenty of code that has not felt the loving embrace of a developer’s debugger in many a year. I’m going to try to address those lonely lines in the coming months, but I can’t make any promises. A little voice in the back of my head keeps saying this could be the basis of a new generation of the Hastymail project, so we’ll see. On the other hand that same voice tells me that rainbows are government monitoring devices that can only be thwarted by wearing underwear made of pop-tarts, so the jury is still out on whether or not it can be trusted.

by

Green Patrol

I’m a terrible singer. I mean just awful. Because of this, I don’t sing often. I have been known to squeak out a tune to help my children sleep, or to recite crappy 80s lyrics to my wife for comical affect, but aside from that I just don’t sing. I have come to realize my inability to sing is in itself a significant skill. Accomplished singers, such as my wife, have tried to replicate the poor timing and off-key caterwauling my attempts produce to no avail. Try as she might she just can’t sing that badly. In school I was the guy just mouthing the words during a concert while his complexion slowly reddened to the point you thought he might pass out. It was a good arrangement for me since the chorus teacher would willingly let this occur, knowing it had a significant impact on the overall quality of the performance. Alas, there were a few unavoidable solo moments that still stalk my nightmares like post apocalyptic cock-roaches that inexplicably just won’t die. This is the story of one of those times.

It was the summer of 86, and I was newly 14. For a week period during that summer I attended a leadership training camp through the boy scouts. The best part of the week turned out to be the drive up to Maine with the other attendees from my troop. We had never been to anything like this and we were bursting with excitement. Unfortunately every event after that drive went successively down hill for the entire week, culminating in one of the most embarrassing moments of my life.

The training seemed to be organized around the idea that scout masters should be lazy arrogant jerks and scouts should be forced to run around with little direction jumping from one intentionally humiliating situation to the next. I remember thinking at the time that this leadership camp could use some new leadership. I would wager third world prisons had cleaner washroom facilities than this place. One of the rituals we performed each day was a gathering of all the patrols in a large clearing. Each patrol had to sing a verse of the corny camp song as they marched in, substituting their not very cleverly designated “patrol color” in the lyrics. Thankfully several members of the green patrol had no trouble belting out our verse, so my lip syncing went unnoticed. Until the last day.

On the last day the idea that we would actually be freed from this place was taking hold in the scout’s minds. People were smiling as they packed up their gear. Kids were happily running down to the quartermasters shack with armloads of cookware in tow to return. I was doing the same for my patrol when I passed the large clearing where we met everyday. Before I could enjoy the thought that I would never have to endure that inanity again, I came to a horrible realization. A meeting was just starting. The scouts around me all scurried into their respective patrol. The green patrol was nowhere to be found. At this moment I considered flight. I scanned the inner circle of senior scout masters. Had they seen me yet? Could I just run as fast as possible away and not look back? Nope. One had seen me. And the green patrol was up next. My fate was sealed. I swallowed hard and accepted it.

I went from zero to bright red before my first footstep hit the clearing grass. My uneven warbling wafted across the field as I marched through the ranks of scout masters. The reactions ranged from that squinty face people make when they hear an unpleasant sound, to broad smiles, to pitiful glances between facepalms. I don’t know if it was the ever-growing aura of humiliation that I was emitting, or just the need to squelch the cacophony of sound coming out of my mouth, but about halfway to my destination several scout masters broke rank and started singing with me. As I finished the verse and took my place in line I could see that my fellow scouts were frozen in that place between wanting to laugh really hard, and being scared that doing so might turn the spotlight of attention in their direction. The next patrol started marching in and the tension ebbed to normal levels again.

As an adult I can look back at this event with a more objective point of view. Why should I care what a bunch of people who I will probably never see again think? But at age 14 I was ruled by emotional responses, and this was the manifestation of my worst fears. These days I try not to sweat the small stuff, and I think pushing ourselves out of our comfort zone once in a while is a great way to expand our horizons. With that said however, please don’t ask me to sing, we will all regret it.

by

Data Excreta

One of the things I love about computer science is the mathematical certainty of the process. The logical constructs of an algorithm define a structure that is rigidly adhered to. If assembled correctly the result of a function is one hundred percent predictable within the bounds of the system. I realize that is a bit simplistic, but there is a beauty in that idea that I find inspiring. For me the comfortable confines of a predefined world help balance out the worrisome chaos of the real world.

Unfortunately complex software systems don’t adhere to this lovely fantasy-land of predictable outcomes I maintain for myself. The ones that survive multiple iterations tend to build up quirks in the trail of data they leave behind. Time will batter and bruise even the most perfectly conceived schema. Unless the software is frozen, somewhere along the line something will trigger a change in the meaning of some data. Maybe a bug, maybe a feature, but either way it leaves a little turd in the record — A small crack in the data’s integrity.

It’s probably no surprise that I’m a big fan of the relational database model. If done properly, it provides a structure to data without which powerful and diverse queries and manipulations would not be possible. Who wouldn’t love that? Once again the enemy of all that is good, “time”, doesn’t love it. Nor does his annoying side kick “complexity”. Bring in the whole crew with “large scale” and you have achieved the trifecta of data turd production. To top it off, the constant improvements of a nimble development team unintentionally make this problem worse.

Don’t get me wrong, I’m not suggesting these are badly designed systems, I’m thinking specifically about ones built by engineers far more experienced and knowledgeable than yours truly. Sadly it’s impossible to plan for every future eventuality, and even if it was, it would undoubtedly be sub-optimal. So just like in real life, turds are unavoidable. One could argue that a different data model like no-sql or multi-value databases don’t suffer from the same types of issues as relational designs. I would argue that’s just silly because nobody cares about those.

Over the last ten years I have been involved in a number of projects for different employers that directly or indirectly deal with dirty data. On the other side of the fence I have been responsible for depositing a bit of fecal matter in the data stream now and then (I mean, shit happens amiright?). Anyway I’m not bragging when I say I have a decent amount of experience with this. Really I’m not, since what I am actually doing is equating myself with the digital equivalent of a pooper-scooper.

Trying to ferret out cause and effect for years old records in a constantly changing system is often a lost cause. This is one of the few times understanding why an inconsistency occurs is less important than just fixing it and moving on. If I am right (with regard to my propensity for being right I will state that in no way is that statistically relevant to the above conclusions, and in fact revealing that metric might negatively bias the reader), and I’m pretty sure I am, dirty data is something that will need to be dealt with sooner or later. Common sense and patience are your best tools. Lots and lots of patience. I do have some tips from the trenches that might help tilt the scales in your favor:

  • Before modifying any data have a plan in place to revert the changes. Not just a vague idea in the back of your head, like a real plan that you actually test before hand.
  • Review your SQL until it’s solid enough to crush diamonds. Remember the first time you accidentally took down a production server because you forgot a “WHERE” clause on an “UPDATE” statement? Everybody else does too, so cut it out.
  • Transactions are manna from heaven, use them wisely:

    Know when to commit them.
    Know when to roll them back.
    Know when to walk away.
    Know when to run.
    You never count your updates,
    when you’re sitting at the command prompt.
    There will be time enough for counting,
    when the transaction’s done.

  • Be ready to be wrong. The comforts of logical outcomes are long gone, now it’s time to juggle probability nightmares in a recursive cycle of alternating head-desks and face-palms. If you are a drinker establish physical obstacles between alcohol sources and your work proximity.
  • Beware of the law of diminishing returns. The closer you get to wiping out all of the anomalies in your data, the longer it will take to make progress on the remaining problems.
  • Are you replicating data? if so throttle for it. Most of the replication technologies I have worked with are like delicate flowers that completely fall apart on what appears to be the slightest whim.

If we work together we can make the world of data a cleaner place. So roll up your sleeves, plug your nose, and get to it. And remember, a node in a B-Tree index is a terrible thing to waste.

by

Running WordPress 3.8 on a Nexus 7

I had the good fortune recently to come into possession of an Asus Nexus 7 Android tablet. After a couple of months playing games on it, a task it excels at I might add, I decided it might be neat to see if I could setup a super-portable self-contained mini-development environment for WordPress. Turns out this is surprisingly easy to put together. With all the pieces in place I can both run and hack WordPress code on the device, even without an active network connection. There are surely many ways to accomplish this, but here is what worked for me.

  • Keyboard
    Trying to code with a onscreen keyboard is a show stopper for me, so in order to make this work I need an external keyboard. It just so happens that my employer, Automattic, included an exceptional Logitech Bluetooth keyboard in our very generous holiday gift packages. It pairs with the device out of the box, but unfortunately the media keys are not supported. To fix that, and to create some useful application shortcuts, I’m using External Keyboard Helper Pro (https://play.google.com/store/apps/details?id=com.apedroid.hwkeyboardhelper&hl=en). It’s not the most intuitive interface (quite possibly one of the worst) but it is powerful with a lot of advanced options.
  • PHP enabled web and DB server
    I expected this to be the most pain in the ass part of the process, but it turned out to be really easy. I chose the Bit Web Server (https://play.google.com/store/apps/details?id=com.andi.serverweb&hl=en) app which is amazingly simple to use and does not require root privileges. It had a small problem with some default paths on first install, but that only took a minute to fix. Web and DB services are accessible via the localhost interface so you can use them without any external network connection. It even comes with phpmyadmin for database management.
  • Vim
    I use Vim for software development and there is a very nice port for Android called Vim Touch (https://play.google.com/store/apps/details?id=net.momodalo.app.vimtouch&hl=en). It’s really a full featured build, and for the most part compatible with my standard configuration.
  • WordPress
    Download the latest WordPress zip file, move it to the document root of the web server, unzip it, run the browser based install. I expected this to be easy, and I was not disappointed: It worked flawlessly.
Vim FTW!

Vim FTW!

I'm blogging!

I’m blogging!

So far it seems to all fit together nicely. I can switch between code and a browser with just the keyboard, and the fact that it’s entirely self contained over localhost is handy for traveling. I have some LONG flights coming up in the next month or two, so I’m hoping I can take advantage of the time and work on some plugins. I have to admit I’m strangely disappointed at how easy this was to setup. I expected a long drawn out battle with obtuse error conditions and undocumented issues. Maybe I should see if I can dual boot Debian and Android …