E-mail

by

5 Cool Cypht Webmail Features

Cypht is the Open Source webmail program I have been toiling away on for the last few years. It stands out from the competition because of a few unique options, not that it doesn’t have its own warts. But let’s focus on the positive, and not talk about things like the painful installation process, outstanding bugs, or unfinished features. I wouldn’t be doing a very good job converting this blog into a propaganda platform for the project with a title like “5 Shitty Cypht Webmail Features“. Also, I’m a dork. What I think qualifies as cool is known to be subjective.

1. Stand Alone Authentication
Pretty boring opener, but bear with me. Typically webmail programs are designed to point to an E-mail source, like an IMAP or POP3 server, for authentication. They pass the username and password you give them on to the E-mail server, which then tells the webmail program if you are legit or bogus. Cypht boldly (not really) breaks this paradigm by splitting authentication from your E-mail sources.

We support using LDAP or a database to authenticate users, as well as the old-school method of using a pre-configured IMAP or POP3 server. Adding new authentication mechanisms is designed to be easy (relatively), so any source that can verify your username and password can be coded up. We even support a dynamic login process that lets you pick from common E-mail service providers, and can auto-discover E-mail services for a domain (sometimes).

2. Combined Views
Cypht provides combined lists of E-mail messages from all your accounts. One of the reasons I started working on yet another webmail program, was because I wanted this feature. I spend 95% of my time using these views for my E-mail needs, and very little time browsing folders and pages like it’s 1999 (though Cypht supports this as well)

Show me the latest 20 unread messages from each of my accounts over the last 2 weeks in one list? Done. Search for the boarding pass I misplaced even though I forgot which E-mail account I used to make the reservation, and the plane takes off in 5 minutes? Done (this actually happened to me). If you have more than one E-mail account, combined views quickly become the bee’s knees. If you only have one E-mail address, you probably should have skipped this section.

3. Module Sets
Plugins are cool. Module sets are cooler. First of all, they sound cooler. Secondly, module sets are not just a way to bolt on features. Cypht is entirely built of module sets, and a framework to run them. Only one is required, the “core” set. It does things like basic page layout and login/logout. Everything else (IMAP, SMTP, POP3, RSS, contacts, profiles, the list goes on) is its own module set, and can be enabled or disabled independently.

As if that wasn’t the ultimate in coolness, there’s more! Module sets can override each other. Don’t like the default behavior of a core module? You can change it without hacking a single line of Cypht code by creating your own module set that overrides it. I need a sweater it’s getting so cool in here! There is even some poorly written documentation about module sets for aspiring developers.

4. Focus on Security
Security is serious business for a webmail program. So many attack vectors! From filtering out nasties, to TLS everywhere, to encrypting data at rest – Cypht goes the extra mile to try to cover all the bases. Cypht was built with security and privacy as core design principles.

Securing a complex web application is a process, and we welcome feedback and suggestions to continue to improve. For the gory details, check out our security page with a list of impressive sounding technical stuff.

5. Production and Debug Mode
Cypht has two modes of operation. “Debug” mode is what you use when troubleshooting issues or doing development. “Production” mode is what you use when … in production. Debug mode fire-hoses your PHP log with information about each request, enables all errors and warnings, and activates new modules as you create them for a quick write-then-test cycle. Production mode uses combined and minfied assets, silences warnings, and pre-calculates module dependencies.

If you are looking for a different kind of webmail, one that is lightweight, secure, and has a complicated install process -check out Cypht. Or don’t. It’s cool.

by

Be Kind, Send Text

There are still some of us out there, an ever dwindling group I’m sure, who don’t want images or borders or colors or bold fonts in an E-mail. I know that using HTML markup to send fancy-pants E-mail is popular with the kids these days, I even begrudgingly support it in my own webmail client. But for the love of all that is holy, if you are going to send HTML formatted E-mails (or any other rich text format), please do it correctly – INCLUDE A TEXT VERSION.

The MIME standard, which defines the structure of “modern” E-mail messages, already supports exactly what you need: Just use “multipart/alternative”, and send a text only version along with the wiz-bang HTML copy you spent a week styling (most of which any sane client will strip out anyway). it’s not rocket science, and even though these standards are decades old, it’s surprising how many companies and services still get it wrong. In the last few weeks alone, I have run across the following:

No text part at all

This is the most common. You didn’t even try to send a text version of the content. Keep in mind, it’s not just curmudgeons like myself who won’t read your message, but people with accessibility requirements (think blind people using screen readers) are also going to have a hard time finding the content hidden in 50 nested divs. Why do you hate blind people?

HTML markup in the text part

You tried, I will give you credit for that. But you failed miserably, because now I get a text message with a bunch of raw HTML tags and useless style rules with some content buried deep within. Thanks, but no thanks.

A text part without the same content

You included a text part. It’s not raw HTML thank goodness. Sadly, it’s just a short message to click a link and read the HTML version online. I’m not going to click that link. Another common one is just an empty text part with no content at all. I appreciate your brevity! My favorite offender in this category is a major bank I do business with. They periodically send me E-mail with an HTML part and text part, yay! unfortunately, the text part is always the same. It simply reads:

TEXT VERSION

Very helpful! I know LARGE BANKS have low standards for software development, but this is ridiculous.

Incorrect or missing character set

I can almost forgive you for this. You sent a text part. It has the same content as the HTML version and isn’t markup. But you used non-ASCII characters and did not include the correct character set header. The result is highly likely to render incorrectly. And you were so close to getting it right!

In conclusion, please be kind to aging oldsters like myself, and send a properly formatted text version of whatever you want me to read. Then I can write happy fun-time blog posts about rainbows instead of ranting about E-mail formats. Eh, I will probably still rant about E-mail formats, but I do like rainbows.

by

The Future of E-mail

I think about E-mail a lot. I’m weird that way. I think of E-mail as an annoying neighbor who wanders over to your yard on a regular basis to chit-chat when you are obviously busy – It just won’t go away. It’s like a realization had in the shower in your early 40’s about your hairstyle choice – It hasn’t really changed in 20 years. And (last simile, I promise), like an untreated rash – It’s not going to just fade away. Trust me on that last one.

Every so often somebody declares that “X software” will kill E-mail, usually referring to some sort of instant message program like Slack. I will admit that while I worked at Automattic we used Slack to a much greater degree than E-mail, but they served different purposes. If E-mail is the electronic version of mail, glorified IRC Services like Slack are the electronic version of a phone call. I don’t recall the US Post Office going out of business when the telephone was invented.

After the “E-mail killers” comes the revolutionaries, changing the way you use E-mail forever. It’s new! It’s shiny! Never mind the fact it’s built on the same old plumbing, don’t worry about that! It has smart-<feature>! It solves all the problems with E-mail you didn’t even know you had until we told you! The world is saved! I like exclamation points!

As a webmail developer, I’m not trying to revolutionize anything. I’m certainly not interested in killing off E-mail. Channeling my inner curmudgeon, I actually like E-mail. But I also think there is room for improvement. With my latest Open Source webmail project, called Cypht (which this blog is designed to shill for), I’m trying to address those areas of concern. Unfortunately for you, I’m going to elaborate.

I have lots of E-mail addresses, mostly because I’m a dork, but I bet that if you have any E-mail account at all, you probably have more than one. Work account. Gmail. Maybe your ISP crams one down your throat like mine does. A throw-away account. Some other free E-mail service you signed up for that one time for some secretive reason you don’t want to talk about.

I want a webmail solution that I can host, that gives me direct access to all my accounts. I don’t want to forward everything to the Google content-mining advertisement delivery interface. I don’t want to POP messages from one account to another in a chain of complicated hops that somehow results in two copies of everything. I want an E-mail delivered to account X to stay in account X, I just want to check it at the same time I check Y and Z.

In my not so humble opinion, this is the most useful feature of Cypht – aggregated views from multiple accounts. You can still go old school and browse folder hierarchies like it’s 1999, but you can also see all your unread messages from all your accounts in a single view. Or search them all at once. Even though this software is not even alpha quality yet, the search feature has already saved my butt more than once.

Well shoot, I got so focused on pimping my project that I forgot to say anything meaningful about the future of E-mail. So here are some half-assed predictions. I predict that E-mail will persist for at least another 20 years as one of the main underpinnings of the internet, and that for the most part it will go unchanged. IMAP servers will still be IMAPing, and SMTP servers will keep SMTPing. Or maybe we will all be using Slack to E-mail and back to Slack gateways, so we never have to login to an E-mail client again. Heck if I know, I’m terrible at predicting things. Either way I need to go pick up my rash ointment.

by

The IMAP BODYSTRUCTURE command, and a bug in the Gmail IMAP service

Now that is one catchy post title. Who DOESN’T like to discuss the nuances of the IMAP BODYSTRUCTURE command? I guess if there are other “E-mail geeks” out there like me – toiling away in the evening’s waning hours writing E-mail software for no good reason – they might find it mildly interesting. At best. The only saving grace to this entire post is the fact that I found what appears to be a legitimate bug in the Gmail IMAP service. Of course this means I have bested all of Google’s E-mail engineers at their own game. I expect penance in the form of exhalation of my prowess, and perhaps a competitive job offer. I doubt either will materialize, but I am prepared to wait.

In the meantime let’s get on with the boring. It’s no secret I have a love-hate relationship with the IMAP protocol, and one of its painfully wonderful features is the BODYSTRUCTURE command. This command returns a string representation of the structure of a MIME formatted message. What makes this command wonderful is that it provides access to all the message parts in a bandwidth limited fashion. What makes it painful is the fact that it’s a disaster to parse, like most IMAP responses, though this one really takes the cake. The BODYSTRUCTURE command gives a mail client enough information to determine the “message part ids” needed to access particular sections of the message. Unlike more simplistic protocols like POP3, we can use this information to selectively choose the message part we want to display, and only fetch that content without having to download the entire thing. Take that simplistic protocols like POP3!

MIME message structure can get really complicated, since message parts can be contained inside other message parts that are inside other message parts (etc). It is critical for a client to accurately represent the structure, and to properly assign the “message part ids” so they can be individually viewed or downloaded. This is the point at which I stumbled on a problem with the Gmail IMAP service. The message in question is a digest E-mail from the Bugtraq mailing list. This message is formatted as a “MESSAGE/DIGEST” MIME type. Digests generally have a text part summary of the included messages, then a list of RFC822 parts containing the original E-mails sent to the list (an RFC822 part is like a container for an entire E-mail message). The Bugtraq digest E-mail follows this pattern. The BODYSTRUCTURE response from Gmail’s IMAP interface for these types of messages appears correct, however the “message part ids” derived from it do not work – they are rejected as invalid.

At first I assumed I was doing something wrong, usually a solid assumption when you are working on an IMAP client. Or you are me. The BODYSTRUCTURE response parsing code in my client was ungainly as sin, so I took this opportunity to re-factor it into something slightly less ugly. Even with the improved code, the message part ids continued to fail, so I decided to copy the message to a local IMAP account using Dovecot to compare the results. Surprisingly, it appears that Gmail’s IMAP server is doing it wrong. The BODYSTRUCTURE response from both IMAP servers is identical. The way my client is parsing the responses and determining the message part ids is also identical. Attempting to access the individual message parts using these ids fails in Gmail, but works in Dovecot. Dun-dun-dun!

Combined with some deep diving in the form of casually skimming the IMAP and MIME RFC’s, I’m convinced this is a bug in Gmail’s IMAP service. Interestingly the Gmail web interface displays digest messages as one big text blob and dumps all the parts out in a row. This might be simpler for users, but wading through raw message text is cumbersome for large digests. For the I’m-sticking-it-to-Gmail-in-this-post record, it also violates the RFC recommendations for clients displaying complex message structure.

Honestly though, I love Gmail, and it’s great that they allow IMAP access. They have also added some neat extensions to the IMAP protocol, such as a Google-like search command that kicks the crap out of the default IMAP search. Since the BODYSTRUCTURE response from Gmail’s IMAP service is correct, I suspect the problem with message part ids not working is a relatively simple fix to their IMAP implementation. True to form, I suspect this without any clue as to the inner workings of the Gmail IMAP implementation.

by

IMAP, I Stab At Thee

I have a love-hate relationship with IMAP (let’s just ignore for the moment how weird it is to have any sort of emotional engagement with a protocol specification). IMAP stands for “Internet Message Access Protocol” and it is a powerful way for client programs to read and manage the messages of an E-mail account. The 107 page definition as laid out in RFC 3501 is not trivial – it is a complex process that can be tricky to get right. This is especially true for client writers as the IMAP server requirements can be very flexible, which in turn requires clients to adapt to a variety of possible implementations. There are also a slew of extensions with their own RFCs, just to keep the water muddy.

My loathing adoration for IMAP started innocently enough. I was getting involved in Open Source software development and was contributing to the Squirrelmail project. Squirrelmail is a webmail program written in PHP, which happens to use IMAP for E-mail account access. My first official submission to this project was on February 20th, 2002, and was completely unrelated to the protocol. However as time wore on, I became more involved in the project. I started digging deeper into the code base. Deeper and deeper. This is where I came face to face with the beast I would wrestle with for the next decade. It was like having my very own virtual Balrog. “BUGS SHALL NOT PASS!“.

PHP has native functions to communicate with IMAP servers, but back in those days they were not always available, and they had a reputation for being unreliable. Squirrelmail had its own client library, using PHP to read and write to the listening socket of the IMAP server. Seems simple, no? Turns out it’s not, but I was fascinated with this aspect of the code. I started spending a lot of time digging into the RFC to ferret out the nuance of one situation or another. I subscribed to esoteric IMAP mailing lists and tried to keep up with the discussion. Soon I had a hard copy of the RFC with me everywhere I went. Not long after that, IMAP commands and responses started streaming through my dreams like ASCII dragons searching for the tender flesh of programmers to feast on.

Leveraging the arrogance of immaturity, I left the Squirrelmail project about a year later, and started my own much less popular webmail project called Hastymail, which I still maintain. I wrote an initial IMAP client implementation in 2003, then again as a part of a complete rewrite to “Hastymail2” in 2008.  Since that time I have just been bolting on features and fixing bugs (in that order), but I have been toying with a new idea recently. All I needed was an opportunity. As luck would have it, I found myself on a long plane trip with some battery life in my laptop, and more importantly that always elusive motivational impulse to jump into something I had not looked at in a long time.

My idea was to decouple the IMAP class from the core Hastymail code so that it could be easily used in any PHP program (hardly breaking new ground here I know, I will get to that). The class was reasonably self-contained, but it did directly use aspects of the Hastymail framework in places, and those areas would need to be marked as deleted then expunged (that’s an IMAP joke). This code was also PHP4 compatible, meaning it was only a “class” in the most basic sense of the word, if at all. I decided to beat it around the head with a PHP5 OOP stick for a while, to see if a better overall structure for the logic could be put in place without ripping too much apart.

I’m genuinely surprised to say it was a pretty successful effort. At some point in the process my mindset evolved from “let’s see if this can work” to “must finish at all costs before thinking about anything else“, and it became a near obsession to reach a particular point of completeness. As to the not breaking new ground issue: I’m well aware of that. I have no illusions that the world needs another PHP IMAP library, or even this one in particular. But it was fun to build, and there is a greater than zero chance that at least one other person on the planet might find it useful – if even as an example of what not to do – so why not share. You can see all the gory details in the newly created github repo.

https://github.com/jasonmunro/hm-imap

Even with all the recent re-factoring there is still plenty of code that has not felt the loving embrace of a developer’s debugger in many a year. I’m going to try to address those lonely lines in the coming months, but I can’t make any promises. A little voice in the back of my head keeps saying this could be the basis of a new generation of the Hastymail project, so we’ll see. On the other hand that same voice tells me that rainbows are government monitoring devices that can only be thwarted by wearing underwear made of pop-tarts, so the jury is still out on whether or not it can be trusted.