IMAP

by

The Itch I Can’t Stop Scratching

My first official contribution to an Open Source software project was way back in 2002.  I was solving a problem for my employer, and ended up becoming a developer for the venerable Squirrelmail project. It was an exciting time. The community was vibrant, active, and surprisingly welcoming to a near-complete novice willing to get their hands dirty. Looking back at the code I wrote lo those many years ago makes me want to gouge my eyes out with red-hot sporks, but I can’t deny the impact contributing to that project had on both my mindset and career path. Since then my involvement in Open Source has waxed and waned, but has always remained. That seemingly innocent interaction sparked a lifelong interest in webmail applications, and I have been tinkering with them ever since.

After a brief 5-year stint writing mostly Python and C++ , I started working with PHP full-time again last May when I joined Automattic. I realized pretty soon after starting that my skills were rusty. Like PHP4 rusty. I needed to experiment with the latest-greatest the language had to offer, but in a safe way, and on my own terms. For the third time in my life, I decided to unleash yet another Open Source webmail client on the world. That surge of excitement you are not feeling at this point is totally understandable. Especially considering the code I wrote the first two times would best be stowed away in the “how not to write complex software” file.

I set out with a newly provisioned github repo, the enthusiasm of someone half my age, and some lofty goals:

  1. Build a client with combined views from multiple E-mail accounts, able to speak both IMAP and POP3, and flexible enough to merge other data sources
  2. Turn security up to 11. Perhaps 12
  3. Make it fast, compact, and compliant
  4. Utilize a modular system that all components outside the bare bones framework use. Like an uber-plugin system the whole app runs on
  5. Do all this while pushing myself to learn what great features new versions of PHP have to offer

To get started, I ferreted out and cleaned up the core IMAP, POP3, and SMTP routines from my last webmail project. While I was at it, I modernized the IMAP library to support some useful protocol extensions, and even built some unit tests *gasp*. These libs have been battle-tested against real world server idiosyncrasies for over a decade, so while they may not be ideal from a code design standpoint, they have an established record of compatibility. This is important when dealing with complex protocols that have a myriad of server implementations. I’m looking at you IMAP.

Next I set out to create a simple request and response processing framework – one that uses “modules” to do the real work of building the resulting page. The framework is lightweight (request processing uses on average 2MB of server memory), and leverages some nifty code features. With a framework in place, the next step was to start cranking out module sets for specific functionality. I started with core requirements like laying out the page content and logging in and out. Next I dove into IMAP, since it would be the primary protocol for E-mail access, and easily the most complicated data source to implement.

9 months later I am happy to say I have a pleasant to use E-mail and RSS reader including preliminary SMTP support for outbound mail (very preliminary). It’s easy on the server and the browser, and has some interesting features for combined content views. It is still very much a work in progress, but here are some highlights:

  • Super small pages with minimal server requests. A single page load only requires 3 HTTP requests with a combined response size of about 30KB (gzipped). Email and Feed data are populated via one parallel AJAX call per source, with response sizes of ~1KB. All interface icons are served inline with data urls to keep request count low.
  • Oodles of security features: TLS/STARTTLS support for all protocols; forced HTTPS for browser requests; secure HTTP-only session level cookies; AES compatible encryption for session and persistent data using unique keys; white-listed and typed user input; built-in HTTP POST nonce enforcement; HTTP header fingerprinting; easy output escaping; a two factor authentication module; probably more I’m forgetting.
  • Modules for IMAP, POP3, SMTP, RSS, and several other app components with more on the way. Modules can be enabled or disabled independently. The module system is super flexible and lends itself to some interesting customization options. It might even turn out to be too flexible.
  • Easy-to-extend session management including stock PHP session support and custom DB sessions. The DB session support is not a registered PHP session handler – it is a completely independent implementation.
  • Authentication is also easy to extend and already supports authenticating via IMAP, POP3, or an included PBKDF2 compliant database schema.
  • Database access is not required (unless used for authentication), but can be leveraged for session and persistent data storage with any PDO supported DB. Table definitions are included for Mysql and Postgresql.
  • Validated HTML5 output, including responsive views for mobile devices and HTLM5 local session storage for caching.
  • Lots of other boring technical details really neat stuff!

I could ramble on about this forever, better stop now before I get carried away. No post about half-done probably soon-to-be obsolete software is complete without at least one screenshot. Here is a look at the interface with a combined view of 9 different RSS feeds.

hm3_feedsIt’s not only been a great learning experience to work on this code, it’s been a lot of fun too. The repository is at http://github.com/jasonmunro/hm3/ for anyone who wants to take a look. Documentation is scarce and things are changing quickly, so if you do check it out, use caution :).

by

IMAP, I Stab At Thee

I have a love-hate relationship with IMAP (let’s just ignore for the moment how weird it is to have any sort of emotional engagement with a protocol specification). IMAP stands for “Internet Message Access Protocol” and it is a powerful way for client programs to read and manage the messages of an E-mail account. The 107 page definition as laid out in RFC 3501 is not trivial – it is a complex process that can be tricky to get right. This is especially true for client writers as the IMAP server requirements can be very flexible, which in turn requires clients to adapt to a variety of possible implementations. There are also a slew of extensions with their own RFCs, just to keep the water muddy.

My loathing adoration for IMAP started innocently enough. I was getting involved in Open Source software development and was contributing to the Squirrelmail project. Squirrelmail is a webmail program written in PHP, which happens to use IMAP for E-mail account access. My first official submission to this project was on February 20th, 2002, and was completely unrelated to the protocol. However as time wore on, I became more involved in the project. I started digging deeper into the code base. Deeper and deeper. This is where I came face to face with the beast I would wrestle with for the next decade. It was like having my very own virtual Balrog. “BUGS SHALL NOT PASS!“.

PHP has native functions to communicate with IMAP servers, but back in those days they were not always available, and they had a reputation for being unreliable. Squirrelmail had its own client library, using PHP to read and write to the listening socket of the IMAP server. Seems simple, no? Turns out it’s not, but I was fascinated with this aspect of the code. I started spending a lot of time digging into the RFC to ferret out the nuance of one situation or another. I subscribed to esoteric IMAP mailing lists and tried to keep up with the discussion. Soon I had a hard copy of the RFC with me everywhere I went. Not long after that, IMAP commands and responses started streaming through my dreams like ASCII dragons searching for the tender flesh of programmers to feast on.

Leveraging the arrogance of immaturity, I left the Squirrelmail project about a year later, and started my own much less popular webmail project called Hastymail, which I still maintain. I wrote an initial IMAP client implementation in 2003, then again as a part of a complete rewrite to “Hastymail2” in 2008.  Since that time I have just been bolting on features and fixing bugs (in that order), but I have been toying with a new idea recently. All I needed was an opportunity. As luck would have it, I found myself on a long plane trip with some battery life in my laptop, and more importantly that always elusive motivational impulse to jump into something I had not looked at in a long time.

My idea was to decouple the IMAP class from the core Hastymail code so that it could be easily used in any PHP program (hardly breaking new ground here I know, I will get to that). The class was reasonably self-contained, but it did directly use aspects of the Hastymail framework in places, and those areas would need to be marked as deleted then expunged (that’s an IMAP joke). This code was also PHP4 compatible, meaning it was only a “class” in the most basic sense of the word, if at all. I decided to beat it around the head with a PHP5 OOP stick for a while, to see if a better overall structure for the logic could be put in place without ripping too much apart.

I’m genuinely surprised to say it was a pretty successful effort. At some point in the process my mindset evolved from “let’s see if this can work” to “must finish at all costs before thinking about anything else“, and it became a near obsession to reach a particular point of completeness. As to the not breaking new ground issue: I’m well aware of that. I have no illusions that the world needs another PHP IMAP library, or even this one in particular. But it was fun to build, and there is a greater than zero chance that at least one other person on the planet might find it useful – if even as an example of what not to do – so why not share. You can see all the gory details in the newly created github repo.

https://github.com/jasonmunro/hm-imap

Even with all the recent re-factoring there is still plenty of code that has not felt the loving embrace of a developer’s debugger in many a year. I’m going to try to address those lonely lines in the coming months, but I can’t make any promises. A little voice in the back of my head keeps saying this could be the basis of a new generation of the Hastymail project, so we’ll see. On the other hand that same voice tells me that rainbows are government monitoring devices that can only be thwarted by wearing underwear made of pop-tarts, so the jury is still out on whether or not it can be trusted.