email client

by

Cypht 1.0.0 Released

After more than 3 years of work, over 3,300 commits, 8 release candidates, 126 resolved issues, and 35,000+ lines of code, I’m pleased to announce the first official stable release of the Cypht webmail program is now available! As anyone who has worked in creating releases for software knows, it’s hard to draw a line in the sand. There is nothing worse than creating a release only to find out the next day you forgot something critical or missed an important bug fix. At the same time, creating releases is a crucial part of getting your software into the hands of users.

I created the release branch 2 months ago with the hope that it would only take a week or two to work out the kinks. After eight release candidates, we finally hit the “it’s good enough, let’s do this thing” point. The way I’m structuring releases in git is to create a release branch from the master branch, then porting applicable bug fixes from the master branch to the release branch while putting out pre-release candidates. Point releases will come from the same branch, but primary development continues on the master, until the next major release, which starts the process over again. I first learned this style of releasing from the Squirrelmail project lo these many years ago. In those days we used diff and patch to port fixes from trunk. With “git cherry-pick” this process is a LOT easier.

The downside to this approach is that over time the master branch diverges from the release branch, and it can get harder and harder to port fixes. The solution is to release often, effectively “dead-ending” the prior release branches as new ones are created. This is a good thing since it encourages frequent releasing. Enough has changed in the master branch in the last 2 months, I’m already eyeballing a 1.1 feature release.

I want to thank everyone who contributed code, filled out a bug report, sent me an E-mail inquiry, requested a feature, donated a translation, or told me they love/hate it. The primary force behind Cypht development is what I want a webmail client to do, but feedback is super important to broaden our user base. I greatly appreciate everyone’s feedback and support for the project.

If you are looking for a secure, lightweight self-hosted webmail that provides access to all your E-mail accounts from one place, give Cypht a try and let me know what you think!

https://github.com/jasonmunro/cypht/releases/tag/v1.0.0

by

The IMAP BODYSTRUCTURE command, and a bug in the Gmail IMAP service

Now that is one catchy post title. Who DOESN’T like to discuss the nuances of the IMAP BODYSTRUCTURE command? I guess if there are other “E-mail geeks” out there like me – toiling away in the evening’s waning hours writing E-mail software for no good reason – they might find it mildly interesting. At best. The only saving grace to this entire post is the fact that I found what appears to be a legitimate bug in the Gmail IMAP service. Of course this means I have bested all of Google’s E-mail engineers at their own game. I expect penance in the form of exhalation of my prowess, and perhaps a competitive job offer. I doubt either will materialize, but I am prepared to wait.

In the meantime let’s get on with the boring. It’s no secret I have a love-hate relationship with the IMAP protocol, and one of its painfully wonderful features is the BODYSTRUCTURE command. This command returns a string representation of the structure of a MIME formatted message. What makes this command wonderful is that it provides access to all the message parts in a bandwidth limited fashion. What makes it painful is the fact that it’s a disaster to parse, like most IMAP responses, though this one really takes the cake. The BODYSTRUCTURE command gives a mail client enough information to determine the “message part ids” needed to access particular sections of the message. Unlike more simplistic protocols like POP3, we can use this information to selectively choose the message part we want to display, and only fetch that content without having to download the entire thing. Take that simplistic protocols like POP3!

MIME message structure can get really complicated, since message parts can be contained inside other message parts that are inside other message parts (etc). It is critical for a client to accurately represent the structure, and to properly assign the “message part ids” so they can be individually viewed or downloaded. This is the point at which I stumbled on a problem with the Gmail IMAP service. The message in question is a digest E-mail from the Bugtraq mailing list. This message is formatted as a “MESSAGE/DIGEST” MIME type. Digests generally have a text part summary of the included messages, then a list of RFC822 parts containing the original E-mails sent to the list (an RFC822 part is like a container for an entire E-mail message). The Bugtraq digest E-mail follows this pattern. The BODYSTRUCTURE response from Gmail’s IMAP interface for these types of messages appears correct, however the “message part ids” derived from it do not work – they are rejected as invalid.

At first I assumed I was doing something wrong, usually a solid assumption when you are working on an IMAP client. Or you are me. The BODYSTRUCTURE response parsing code in my client was ungainly as sin, so I took this opportunity to re-factor it into something slightly less ugly. Even with the improved code, the message part ids continued to fail, so I decided to copy the message to a local IMAP account using Dovecot to compare the results. Surprisingly, it appears that Gmail’s IMAP server is doing it wrong. The BODYSTRUCTURE response from both IMAP servers is identical. The way my client is parsing the responses and determining the message part ids is also identical. Attempting to access the individual message parts using these ids fails in Gmail, but works in Dovecot. Dun-dun-dun!

Combined with some deep diving in the form of casually skimming the IMAP and MIME RFC’s, I’m convinced this is a bug in Gmail’s IMAP service. Interestingly the Gmail web interface displays digest messages as one big text blob and dumps all the parts out in a row. This might be simpler for users, but wading through raw message text is cumbersome for large digests. For the I’m-sticking-it-to-Gmail-in-this-post record, it also violates the RFC recommendations for clients displaying complex message structure.

Honestly though, I love Gmail, and it’s great that they allow IMAP access. They have also added some neat extensions to the IMAP protocol, such as a Google-like search command that kicks the crap out of the default IMAP search. Since the BODYSTRUCTURE response from Gmail’s IMAP service is correct, I suspect the problem with message part ids not working is a relatively simple fix to their IMAP implementation. True to form, I suspect this without any clue as to the inner workings of the Gmail IMAP implementation.

by

IMAP, I Stab At Thee

I have a love-hate relationship with IMAP (let’s just ignore for the moment how weird it is to have any sort of emotional engagement with a protocol specification). IMAP stands for “Internet Message Access Protocol” and it is a powerful way for client programs to read and manage the messages of an E-mail account. The 107 page definition as laid out in RFC 3501 is not trivial – it is a complex process that can be tricky to get right. This is especially true for client writers as the IMAP server requirements can be very flexible, which in turn requires clients to adapt to a variety of possible implementations. There are also a slew of extensions with their own RFCs, just to keep the water muddy.

My loathing adoration for IMAP started innocently enough. I was getting involved in Open Source software development and was contributing to the Squirrelmail project. Squirrelmail is a webmail program written in PHP, which happens to use IMAP for E-mail account access. My first official submission to this project was on February 20th, 2002, and was completely unrelated to the protocol. However as time wore on, I became more involved in the project. I started digging deeper into the code base. Deeper and deeper. This is where I came face to face with the beast I would wrestle with for the next decade. It was like having my very own virtual Balrog. “BUGS SHALL NOT PASS!“.

PHP has native functions to communicate with IMAP servers, but back in those days they were not always available, and they had a reputation for being unreliable. Squirrelmail had its own client library, using PHP to read and write to the listening socket of the IMAP server. Seems simple, no? Turns out it’s not, but I was fascinated with this aspect of the code. I started spending a lot of time digging into the RFC to ferret out the nuance of one situation or another. I subscribed to esoteric IMAP mailing lists and tried to keep up with the discussion. Soon I had a hard copy of the RFC with me everywhere I went. Not long after that, IMAP commands and responses started streaming through my dreams like ASCII dragons searching for the tender flesh of programmers to feast on.

Leveraging the arrogance of immaturity, I left the Squirrelmail project about a year later, and started my own much less popular webmail project called Hastymail, which I still maintain. I wrote an initial IMAP client implementation in 2003, then again as a part of a complete rewrite to “Hastymail2” in 2008.  Since that time I have just been bolting on features and fixing bugs (in that order), but I have been toying with a new idea recently. All I needed was an opportunity. As luck would have it, I found myself on a long plane trip with some battery life in my laptop, and more importantly that always elusive motivational impulse to jump into something I had not looked at in a long time.

My idea was to decouple the IMAP class from the core Hastymail code so that it could be easily used in any PHP program (hardly breaking new ground here I know, I will get to that). The class was reasonably self-contained, but it did directly use aspects of the Hastymail framework in places, and those areas would need to be marked as deleted then expunged (that’s an IMAP joke). This code was also PHP4 compatible, meaning it was only a “class” in the most basic sense of the word, if at all. I decided to beat it around the head with a PHP5 OOP stick for a while, to see if a better overall structure for the logic could be put in place without ripping too much apart.

I’m genuinely surprised to say it was a pretty successful effort. At some point in the process my mindset evolved from “let’s see if this can work” to “must finish at all costs before thinking about anything else“, and it became a near obsession to reach a particular point of completeness. As to the not breaking new ground issue: I’m well aware of that. I have no illusions that the world needs another PHP IMAP library, or even this one in particular. But it was fun to build, and there is a greater than zero chance that at least one other person on the planet might find it useful – if even as an example of what not to do – so why not share. You can see all the gory details in the newly created github repo.

https://github.com/jasonmunro/hm-imap

Even with all the recent re-factoring there is still plenty of code that has not felt the loving embrace of a developer’s debugger in many a year. I’m going to try to address those lonely lines in the coming months, but I can’t make any promises. A little voice in the back of my head keeps saying this could be the basis of a new generation of the Hastymail project, so we’ll see. On the other hand that same voice tells me that rainbows are government monitoring devices that can only be thwarted by wearing underwear made of pop-tarts, so the jury is still out on whether or not it can be trusted.