IMAP, I Stab At Thee

I have a love-hate relationship with IMAP (let’s just ignore for the moment how weird it is to have any sort of emotional engagement with a protocol specification). IMAP stands for “Internet Message Access Protocol” and it is a powerful way for client programs to read and manage the messages of an E-mail account. The 107 page definition as laid out in RFC 3501 is not trivial – it is a complex process that can be tricky to get right. This is especially true for client writers as the IMAP server requirements can be very flexible, which in turn requires clients to adapt to a variety of possible implementations. There are also a slew of extensions with their own RFCs, just to keep the water muddy.

My loathing adoration for IMAP started innocently enough. I was getting involved in Open Source software development and was contributing to the Squirrelmail project. Squirrelmail is a webmail program written in PHP, which happens to use IMAP for E-mail account access. My first official submission to this project was on February 20th, 2002, and was completely unrelated to the protocol. However as time wore on, I became more involved in the project. I started digging deeper into the code base. Deeper and deeper. This is where I came face to face with the beast I would wrestle with for the next decade. It was like having my very own virtual Balrog. “BUGS SHALL NOT PASS!“.

PHP has native functions to communicate with IMAP servers, but back in those days they were not always available, and they had a reputation for being unreliable. Squirrelmail had its own client library, using PHP to read and write to the listening socket of the IMAP server. Seems simple, no? Turns out it’s not, but I was fascinated with this aspect of the code. I started spending a lot of time digging into the RFC to ferret out the nuance of one situation or another. I subscribed to esoteric IMAP mailing lists and tried to keep up with the discussion. Soon I had a hard copy of the RFC with me everywhere I went. Not long after that, IMAP commands and responses started streaming through my dreams like ASCII dragons searching for the tender flesh of programmers to feast on.

Leveraging the arrogance of immaturity, I left the Squirrelmail project about a year later, and started my own much less popular webmail project called Hastymail, which I still maintain. I wrote an initial IMAP client implementation in 2003, then again as a part of a complete rewrite to “Hastymail2” in 2008.  Since that time I have just been bolting on features and fixing bugs (in that order), but I have been toying with a new idea recently. All I needed was an opportunity. As luck would have it, I found myself on a long plane trip with some battery life in my laptop, and more importantly that always elusive motivational impulse to jump into something I had not looked at in a long time.

My idea was to decouple the IMAP class from the core Hastymail code so that it could be easily used in any PHP program (hardly breaking new ground here I know, I will get to that). The class was reasonably self-contained, but it did directly use aspects of the Hastymail framework in places, and those areas would need to be marked as deleted then expunged (that’s an IMAP joke). This code was also PHP4 compatible, meaning it was only a “class” in the most basic sense of the word, if at all. I decided to beat it around the head with a PHP5 OOP stick for a while, to see if a better overall structure for the logic could be put in place without ripping too much apart.

I’m genuinely surprised to say it was a pretty successful effort. At some point in the process my mindset evolved from “let’s see if this can work” to “must finish at all costs before thinking about anything else“, and it became a near obsession to reach a particular point of completeness. As to the not breaking new ground issue: I’m well aware of that. I have no illusions that the world needs another PHP IMAP library, or even this one in particular. But it was fun to build, and there is a greater than zero chance that at least one other person on the planet might find it useful – if even as an example of what not to do – so why not share. You can see all the gory details in the newly created github repo.

https://github.com/jasonmunro/hm-imap

Even with all the recent re-factoring there is still plenty of code that has not felt the loving embrace of a developer’s debugger in many a year. I’m going to try to address those lonely lines in the coming months, but I can’t make any promises. A little voice in the back of my head keeps saying this could be the basis of a new generation of the Hastymail project, so we’ll see. On the other hand that same voice tells me that rainbows are government monitoring devices that can only be thwarted by wearing underwear made of pop-tarts, so the jury is still out on whether or not it can be trusted.

10 thoughts on “IMAP, I Stab At Thee

  1. One of my friends had to write an IMAP interface for a project he was working on. Every time I saw him, he seemed more gaunt, and more stressed.

  2. Looks like hm-imap is something I could use. I downloaded it from GitHub but the example.php generates some syntax errors on PHP 5.4. I hacked away to get rid of the syntax errors in examples.php and now it is complaning about errors in hm-imap.php (syntax error, unexpected ‘[‘ on line 1988.

    I don’t want to mess with hm-imap.php since it is the core and not an example. Anyone know what the issue is, or does anyone have it running?

    • Just wanted to leave an update. Looks like the code uses array dereferencing, which is not supported in PHP 5.3. I was trying to run the code on a PHP 5.3 based system, not a 5.4 based system.

      I tried it on a 5.4 based system and it seems to at least be running without syntax errors.

      • Wow, you are the first person I know of who has tried this code, thanks! I would be happy to fix the array dereferencing to make this work on more PHP versions. Also it’s worth noting I have made a few additional updates after moving this code to a different repo located here: https://github.com/jasonmunro/hm3 (the files are in lib/). Thanks again for the feedback!

      • I have spent a bit of time with it just trying the included examples. Tried it against two servers. One server works (exim based) and another server does not work. The one that fails is zoho.com.

        I think it bombs in hm-imap when it checks if is_supported(‘SORT’). I assume the remote server does not have the support for that so it can’t pull back any mail?

      • I’m using the library with Gmail which does not support IMAP SORT, so I don’t think that is the issue. The library has a client side sort ability (much slower than IMAP SORT), that fetches headers and sorts the UIDS locally, which should work. I have never used zoho, but will sign up (if it’s free) and give it a whirl. Thanks again for the great feedback!

    • I did, and so far the only problem I noticed is that message headers are missing the full CRLF line terminator. I committed a work around this morning that handles that. I’m using the get_mailbox_page method: $imap->get_mailbox_page('INBOX', 'ARRIVAL', true, 'ALL', 0, 20) to get a partial message list for a folder. You are correct that zoho does not support the IMAP SORT extension, but the get_mailbox_page method should fall back to client based sorting using good old IMAP FETCH. If you want a sorted list for all UIDs in a folder you might want to use $imap->sort_by_fetch(). Also I pushed a handful of tweaks from the hm3 repo over to the hm-imap github so it’s now current with what I am using. Hope that helps!

  3. Pingback: The Itch I Can’t Stop Scratching | Unencumbered by Facts

  4. Pingback: The IMAP BODYSTRUCTURE command, and a bug in the Gmail IMAP service | Unencumbered by Facts

Leave a comment