IMAP

by

Cypht Webmail Screen Shots – Mobile Version

I had a request on the “Cypht Webmail Screen Shots” post to share some mobile/responsive views of the program. Since I always aim to please, and since the percentage of people with even a shred of interest in this project is so very very small, here it is!

Overall, my design skills fall somewhere between terrible and total shit. My design premise for the Cypht UI has been: “Don’t pretend you know what colors go well together. Don’t fool yourself into thinking you can make something slick. JUST KEEP IT SIMPLE DUMMY“.

I’m pleasantly surprised with the result. It’s pretty standard webmail fare, super simple, and reasonably consistent across browsers.

cypht_mobile10

The main menu. Available by swiping right on any page, or with the awkward looking triangle in the upper left corner of a page.

cypht_mobile9

I pretty much never write E-mail on my phone. But I could!

cypht_mobile8

Yay, only 1 unread message! Also, if you have not checked out Humble Bundle, you should.

cypht_mobile6

The site settings page

cypht_mobile5

General site settings. Kind of squeezed in there, but not too bad!

cypht_mobile3

Search all the things again!

cypht_mobile4

Read a search result. The message view page is tricky on mobile. This one looks pretty good, but others not so much. It’s a work in progress.

cypht_mobile2

Just a teeny tiny calendar. Nothing to see here.

cypht_mobile1

RSS news feed for my favorite Linux news site.

cypht_mobile15

Server management page

cypht_mobile13

“Quick add” dialog on the server page. So cool.

cypht_mobile14

Keyboard shortcut management.

cypht_mobile16

IMAP folder management.

cypht_mobile11

Save settings page.

There are some noticeable omissions to this list, specifically the contacts management interface and the profiles page. I left profiles out because it looks like dog poop, and I left contacts out because I realized the screenshot was of my actual contacts and that would be dumb to post. It looked pretty good though! All in all, Cypht is pretty responsive. Writing this post helped me identify some areas we need to work on, so thanks to the person who requested it!

by

5 Cool Cypht Webmail Features

Cypht is the Open Source webmail program I have been toiling away on for the last few years. It stands out from the competition because of a few unique options, not that it doesn’t have its own warts. But let’s focus on the positive, and not talk about things like the painful installation process, outstanding bugs, or unfinished features. I wouldn’t be doing a very good job converting this blog into a propaganda platform for the project with a title like “5 Shitty Cypht Webmail Features“. Also, I’m a dork. What I think qualifies as cool is known to be subjective.

1. Stand Alone Authentication
Pretty boring opener, but bear with me. Typically webmail programs are designed to point to an E-mail source, like an IMAP or POP3 server, for authentication. They pass the username and password you give them on to the E-mail server, which then tells the webmail program if you are legit or bogus. Cypht boldly (not really) breaks this paradigm by splitting authentication from your E-mail sources.

We support using LDAP or a database to authenticate users, as well as the old-school method of using a pre-configured IMAP or POP3 server. Adding new authentication mechanisms is designed to be easy (relatively), so any source that can verify your username and password can be coded up. We even support a dynamic login process that lets you pick from common E-mail service providers, and can auto-discover E-mail services for a domain (sometimes).

2. Combined Views
Cypht provides combined lists of E-mail messages from all your accounts. One of the reasons I started working on yet another webmail program, was because I wanted this feature. I spend 95% of my time using these views for my E-mail needs, and very little time browsing folders and pages like it’s 1999 (though Cypht supports this as well)

Show me the latest 20 unread messages from each of my accounts over the last 2 weeks in one list? Done. Search for the boarding pass I misplaced even though I forgot which E-mail account I used to make the reservation, and the plane takes off in 5 minutes? Done (this actually happened to me). If you have more than one E-mail account, combined views quickly become the bee’s knees. If you only have one E-mail address, you probably should have skipped this section.

3. Module Sets
Plugins are cool. Module sets are cooler. First of all, they sound cooler. Secondly, module sets are not just a way to bolt on features. Cypht is entirely built of module sets, and a framework to run them. Only one is required, the “core” set. It does things like basic page layout and login/logout. Everything else (IMAP, SMTP, POP3, RSS, contacts, profiles, the list goes on) is its own module set, and can be enabled or disabled independently.

As if that wasn’t the ultimate in coolness, there’s more! Module sets can override each other. Don’t like the default behavior of a core module? You can change it without hacking a single line of Cypht code by creating your own module set that overrides it. I need a sweater it’s getting so cool in here! There is even some poorly written documentation about module sets for aspiring developers.

4. Focus on Security
Security is serious business for a webmail program. So many attack vectors! From filtering out nasties, to TLS everywhere, to encrypting data at rest – Cypht goes the extra mile to try to cover all the bases. Cypht was built with security and privacy as core design principles.

Securing a complex web application is a process, and we welcome feedback and suggestions to continue to improve. For the gory details, check out our security page with a list of impressive sounding technical stuff.

5. Production and Debug Mode
Cypht has two modes of operation. “Debug” mode is what you use when troubleshooting issues or doing development. “Production” mode is what you use when … in production. Debug mode fire-hoses your PHP log with information about each request, enables all errors and warnings, and activates new modules as you create them for a quick write-then-test cycle. Production mode uses combined and minfied assets, silences warnings, and pre-calculates module dependencies.

If you are looking for a different kind of webmail, one that is lightweight, secure, and has a complicated install process -check out Cypht. Or don’t. It’s cool.

by

Cypht: New Open Source Webmail

I suck at product announcements, probably because I have no perceptible marketing skills. Usually I just whip out a list of technical details that even I can’t edit because I fall asleep by the third bullet point – A list that significantly limits the number of people who care one iota about what I’m blathering on about. What was I blathering on about? That’s right, sucky product announcements. Let’s pile another one on the heap, shall we?

This is the official wiz-bang super exciting announcement for a NEW shiny NEW Open Source webmail program called “Cypht” (pronounced “sift”). Did I mention it was NEW? New does not necessarily mean better you say? Well, I had not thought about that before deciding to emphasize it. Seriously however, in the world of Open Source webmail, new is actually exciting (and subjective I guess since I have been working on this for like a year already).

Will this software disrupt the E-mail paradigm while streamlining workflow to maximize interpersonal communication channels using a ground breaking application stack and development process? Nope. From a user perspective it’s like a lot of web-based E-mail programs you are already familiar with. From a developer perspective the code is experimental, with a focus on a smart overall design that offsets some of the downfalls of building a complex program in PHP.

Wait, don’t go just because I said PHP. It’s not like the other PHP based webmail programs I have worked on, but it’s also not just another vanilla web-based E-mail client. One of the core ideas and most interesting features is the concept of viewing aggregated E-mail from multiple sources, without actually forwarding the messages to a central account. Let me esplain:

I have a few Gmail accounts. An old Yahoo E-mail. Throw in a few other domains I own and I have about 5 or so accounts I want to keep up with (to varying degrees). I don’t want to auto-forward them all to a single account, I like keeping my work E-mail separate from my personal messages. So I said to myself,

“Self, why not build a webmail application that does that? One that could give self fast access to a list of all the unread messages in all self’s inboxes from the last 2 days. Or something like that. Oh and search and stuff.”

And so it came to pass, that said webmail was created and continues to be tinkered with. I have droned on endlessly about this software on this blog in the past, so if you are one of the people who came here for the “Linux on a Macbook Pro” post (which is why most people come here) and clicked on the wrong link, this probably sounds familiar.

Since this is not really new, not even new to this blog, why the awful product announcement? Is it just an excuse to rehash old post material until someone pays attention to your stupid program? Maybe, but I think we are getting a little off-track, and for the record its my blog so I will be the one asking the questions around here partner.

Did I mention I like lists of technical details? Here comes one!

  • Really small page sizes, like the entire page + JS + CSS + Ajax requests is less than the Google home page. With normal browser caching and HTML5 local session storage the data transferred is reduced to a stupifyingly small amount.
  • An extra emphasis on security and privacy throughout the application. This is an Open Source program meant to be safe and useful, not an ad generating machine mining your E-mail.
  • It’s not your dad’s webmail! Unless you are one of my daughters in which case it is your dad’s webmail.
  • Valid HTML5 pages with accessibility friendly markup and mobile views.
  • A module system that is like a plugin system on crank. It’s all friggen modules!
  • At this point in development, Cypht is a decent E-mail and news reader with limited outbound message support (in this context limited means don’t use it).
  • A huge following on Github of 1, combined with a large developer base of 1. This thing is taking off I tell you!
  • A website with tons of impressive sounding acronyms like IMAP, POP3, SMTP, HHVM, RSS, SSL, PHP, 2FA, PBKDF2, AES, and more!
  • Other stuff!

Still here? I’m as surprised as you are. This concludes today’s craptacular product announcement. This blog will now return to its irregular schedule of posts about video games or goats or whatever.

by

The IMAP BODYSTRUCTURE command, and a bug in the Gmail IMAP service

Now that is one catchy post title. Who DOESN’T like to discuss the nuances of the IMAP BODYSTRUCTURE command? I guess if there are other “E-mail geeks” out there like me – toiling away in the evening’s waning hours writing E-mail software for no good reason – they might find it mildly interesting. At best. The only saving grace to this entire post is the fact that I found what appears to be a legitimate bug in the Gmail IMAP service. Of course this means I have bested all of Google’s E-mail engineers at their own game. I expect penance in the form of exhalation of my prowess, and perhaps a competitive job offer. I doubt either will materialize, but I am prepared to wait.

In the meantime let’s get on with the boring. It’s no secret I have a love-hate relationship with the IMAP protocol, and one of its painfully wonderful features is the BODYSTRUCTURE command. This command returns a string representation of the structure of a MIME formatted message. What makes this command wonderful is that it provides access to all the message parts in a bandwidth limited fashion. What makes it painful is the fact that it’s a disaster to parse, like most IMAP responses, though this one really takes the cake. The BODYSTRUCTURE command gives a mail client enough information to determine the “message part ids” needed to access particular sections of the message. Unlike more simplistic protocols like POP3, we can use this information to selectively choose the message part we want to display, and only fetch that content without having to download the entire thing. Take that simplistic protocols like POP3!

MIME message structure can get really complicated, since message parts can be contained inside other message parts that are inside other message parts (etc). It is critical for a client to accurately represent the structure, and to properly assign the “message part ids” so they can be individually viewed or downloaded. This is the point at which I stumbled on a problem with the Gmail IMAP service. The message in question is a digest E-mail from the Bugtraq mailing list. This message is formatted as a “MESSAGE/DIGEST” MIME type. Digests generally have a text part summary of the included messages, then a list of RFC822 parts containing the original E-mails sent to the list (an RFC822 part is like a container for an entire E-mail message). The Bugtraq digest E-mail follows this pattern. The BODYSTRUCTURE response from Gmail’s IMAP interface for these types of messages appears correct, however the “message part ids” derived from it do not work – they are rejected as invalid.

At first I assumed I was doing something wrong, usually a solid assumption when you are working on an IMAP client. Or you are me. The BODYSTRUCTURE response parsing code in my client was ungainly as sin, so I took this opportunity to re-factor it into something slightly less ugly. Even with the improved code, the message part ids continued to fail, so I decided to copy the message to a local IMAP account using Dovecot to compare the results. Surprisingly, it appears that Gmail’s IMAP server is doing it wrong. The BODYSTRUCTURE response from both IMAP servers is identical. The way my client is parsing the responses and determining the message part ids is also identical. Attempting to access the individual message parts using these ids fails in Gmail, but works in Dovecot. Dun-dun-dun!

Combined with some deep diving in the form of casually skimming the IMAP and MIME RFC’s, I’m convinced this is a bug in Gmail’s IMAP service. Interestingly the Gmail web interface displays digest messages as one big text blob and dumps all the parts out in a row. This might be simpler for users, but wading through raw message text is cumbersome for large digests. For the I’m-sticking-it-to-Gmail-in-this-post record, it also violates the RFC recommendations for clients displaying complex message structure.

Honestly though, I love Gmail, and it’s great that they allow IMAP access. They have also added some neat extensions to the IMAP protocol, such as a Google-like search command that kicks the crap out of the default IMAP search. Since the BODYSTRUCTURE response from Gmail’s IMAP service is correct, I suspect the problem with message part ids not working is a relatively simple fix to their IMAP implementation. True to form, I suspect this without any clue as to the inner workings of the Gmail IMAP implementation.

by

The Itch I Can’t Stop Scratching

My first official contribution to an Open Source software project was way back in 2002.  I was solving a problem for my employer, and ended up becoming a developer for the venerable Squirrelmail project. It was an exciting time. The community was vibrant, active, and surprisingly welcoming to a near-complete novice willing to get their hands dirty. Looking back at the code I wrote lo those many years ago makes me want to gouge my eyes out with red-hot sporks, but I can’t deny the impact contributing to that project had on both my mindset and career path. Since then my involvement in Open Source has waxed and waned, but has always remained. That seemingly innocent interaction sparked a lifelong interest in webmail applications, and I have been tinkering with them ever since.

After a brief 5-year stint writing mostly Python and C++ , I started working with PHP full-time again last May when I joined Automattic. I realized pretty soon after starting that my skills were rusty. Like PHP4 rusty. I needed to experiment with the latest-greatest the language had to offer, but in a safe way, and on my own terms. For the third time in my life, I decided to unleash yet another Open Source webmail client on the world. That surge of excitement you are not feeling at this point is totally understandable. Especially considering the code I wrote the first two times would best be stowed away in the “how not to write complex software” file.

I set out with a newly provisioned github repo, the enthusiasm of someone half my age, and some lofty goals:

  1. Build a client with combined views from multiple E-mail accounts, able to speak both IMAP and POP3, and flexible enough to merge other data sources
  2. Turn security up to 11. Perhaps 12
  3. Make it fast, compact, and compliant
  4. Utilize a modular system that all components outside the bare bones framework use. Like an uber-plugin system the whole app runs on
  5. Do all this while pushing myself to learn what great features new versions of PHP have to offer

To get started, I ferreted out and cleaned up the core IMAP, POP3, and SMTP routines from my last webmail project. While I was at it, I modernized the IMAP library to support some useful protocol extensions, and even built some unit tests *gasp*. These libs have been battle-tested against real world server idiosyncrasies for over a decade, so while they may not be ideal from a code design standpoint, they have an established record of compatibility. This is important when dealing with complex protocols that have a myriad of server implementations. I’m looking at you IMAP.

Next I set out to create a simple request and response processing framework – one that uses “modules” to do the real work of building the resulting page. The framework is lightweight (request processing uses on average 2MB of server memory), and leverages some nifty code features. With a framework in place, the next step was to start cranking out module sets for specific functionality. I started with core requirements like laying out the page content and logging in and out. Next I dove into IMAP, since it would be the primary protocol for E-mail access, and easily the most complicated data source to implement.

9 months later I am happy to say I have a pleasant to use E-mail and RSS reader including preliminary SMTP support for outbound mail (very preliminary). It’s easy on the server and the browser, and has some interesting features for combined content views. It is still very much a work in progress, but here are some highlights:

  • Super small pages with minimal server requests. A single page load only requires 3 HTTP requests with a combined response size of about 30KB (gzipped). Email and Feed data are populated via one parallel AJAX call per source, with response sizes of ~1KB. All interface icons are served inline with data urls to keep request count low.
  • Oodles of security features: TLS/STARTTLS support for all protocols; forced HTTPS for browser requests; secure HTTP-only session level cookies; AES compatible encryption for session and persistent data using unique keys; white-listed and typed user input; built-in HTTP POST nonce enforcement; HTTP header fingerprinting; easy output escaping; a two factor authentication module; probably more I’m forgetting.
  • Modules for IMAP, POP3, SMTP, RSS, and several other app components with more on the way. Modules can be enabled or disabled independently. The module system is super flexible and lends itself to some interesting customization options. It might even turn out to be too flexible.
  • Easy-to-extend session management including stock PHP session support and custom DB sessions. The DB session support is not a registered PHP session handler – it is a completely independent implementation.
  • Authentication is also easy to extend and already supports authenticating via IMAP, POP3, or an included PBKDF2 compliant database schema.
  • Database access is not required (unless used for authentication), but can be leveraged for session and persistent data storage with any PDO supported DB. Table definitions are included for Mysql and Postgresql.
  • Validated HTML5 output, including responsive views for mobile devices and HTLM5 local session storage for caching.
  • Lots of other boring technical details really neat stuff!

I could ramble on about this forever, better stop now before I get carried away. No post about half-done probably soon-to-be obsolete software is complete without at least one screenshot. Here is a look at the interface with a combined view of 9 different RSS feeds.

hm3_feedsIt’s not only been a great learning experience to work on this code, it’s been a lot of fun too. The repository is at http://github.com/jasonmunro/hm3/ for anyone who wants to take a look. Documentation is scarce and things are changing quickly, so if you do check it out, use caution :).