The Future of E-mail

I think about E-mail a lot. I’m weird that way. I think of E-mail as an annoying neighbor who wanders over to your yard on a regular basis to chit-chat when you are obviously busy – It just won’t go away. It’s like a realization had in the shower in your early 40’s about your hairstyle choice – It hasn’t really changed in 20 years. And (last simile, I promise), like an untreated rash – It’s not going to just fade away. Trust me on that last one.

Every so often somebody declares that “X software” will kill E-mail, usually referring to some sort of instant message program like Slack. I will admit that while I worked at Automattic we used Slack to a much greater degree than E-mail, but they served different purposes. If E-mail is the electronic version of mail, glorified IRC Services like Slack are the electronic version of a phone call. I don’t recall the US Post Office going out of business when the telephone was invented.

After the “E-mail killers” comes the revolutionaries, changing the way you use E-mail forever. It’s new! It’s shiny! Never mind the fact it’s built on the same old plumbing, don’t worry about that! It has smart-<feature>! It solves all the problems with E-mail you didn’t even know you had until we told you! The world is saved! I like exclamation points!

As a webmail developer, I’m not trying to revolutionize anything. I’m certainly not interested in killing off E-mail. Channeling my inner curmudgeon, I actually like E-mail. But I also think there is room for improvement. With my latest Open Source webmail project, called Cypht (which this blog is designed to shill for), I’m trying to address those areas of concern. Unfortunately for you, I’m going to elaborate.

I have lots of E-mail addresses, mostly because I’m a dork, but I bet that if you have any E-mail account at all, you probably have more than one. Work account. Gmail. Maybe your ISP crams one down your throat like mine does. A throw-away account. Some other free E-mail service you signed up for that one time for some secretive reason you don’t want to talk about.

I want a webmail solution that I can host, that gives me direct access to all my accounts. I don’t want to forward everything to the Google content-mining advertisement delivery interface. I don’t want to POP messages from one account to another in a chain of complicated hops that somehow results in two copies of everything. I want an E-mail delivered to account X to stay in account X, I just want to check it at the same time I check Y and Z.

In my not so humble opinion, this is the most useful feature of Cypht – aggregated views from multiple accounts. You can still go old school and browse folder hierarchies like it’s 1999, but you can also see all your unread messages from all your accounts in a single view. Or search them all at once. Even though this software is not even alpha quality yet, the search feature has already saved my butt more than once.

Well shoot, I got so focused on pimping my project that I forgot to say anything meaningful about the future of E-mail. So here are some half-assed predictions. I predict that E-mail will persist for at least another 20 years as one of the main underpinnings of the internet, and that for the most part it will go unchanged. IMAP servers will still be IMAPing, and SMTP servers will keep SMTPing. Or maybe we will all be using Slack to E-mail and back to Slack gateways, so we never have to login to an E-mail client again. Heck if I know, I’m terrible at predicting things. Either way I need to go pick up my rash ointment.

Size Matters

Remember when 3G was the latest greatest mobile network upgrade? I do. It was like a bazillion times faster than my current “data service”. It wasn’t even that long ago (otherwise it would have already been flushed from my aging brain cells like so many other memories). Now, 3G seems closer to no internet at all. I can’t help but cringe every time the LTE icon on my phone is replaced with the dreaded 3G symbol. Even the rotating circle of the icon seems to move at a snail’s pace. What the heck happened? Glad you asked!

The size and complexity of websites and applications is spiraling out of control. It’s a silent epidemic in my not so humble opinion. Probably because the ones it really affects can’t get a contact form to load so they can complain. Seriously people, the amount of junk packed into a single webpage request is off the charts, as demonstrated below:

20150807082113

If that very convincing metric I just made up and spent less than 2 minutes slapping into a chart is not enough proof for you, let’s try some real world tests. First, let’s check 3 websites the kids all love and break ’em down using the chromium web developer console to take a peek under the sheets. Then let’s compare that to Cypht, since everything I write on this blog is really just a pathetic attempt to shill for my Open Source project.

size_matters_google

251 KB transferred with 11 requests. I guess that is “good” by today’s standards. It’s Google so it has to be good, right?

I don’t Facebook (thank god), and I die a little inside when I use the word Facebook as a verb, but I know all the kids are staring at it when they impolitely tread across my lawn, so let’s check it out! The un-cached and logged out homepage is something to behold:

size_matters_facebook

902 KB, 31 requests. Ouch! Almost 1 MB of data to display a LOGGED OUT HOMEPAGE? I need to get into the web-hosting biz STAT!

It’s my understanding kids today are not taught how to read in school anymore, so they are only capable of using a crippling subset of standard language skills to find the only type of media they are prepared to consume – short videos. From what I have observed, The Youtubes is popular with these creatures. Wonder how it fares?

size_matter_youtube

OK, I can cut Youtube some slack, it’s a video site after all, but holy Bos Taurus: 3.6 MB and 136 requests is pretty insanely high for … pretty much anything.

Finally, let’s take a look at the lovingly handcrafted pages of Cypht, my Open Source webmail project. I will even tilt the scales against it by using a LOGGED IN page since the logged out homepage of the application, and the cypht.org site, are so incredibly minimal that comparing them to the behemoths above is at a scale difficult for human intelligence to grasp.

size_matters_cypht

Hmm…. 30.2 KB transferred with 3 requests. Contrary to what I have been told throughout my life *ehem*, maybe size does matter?

* Don’t even get me started on the inability of websites to adhere to standards. I mean seriously don’t, I’m saving that for another post

** Even this venerable WordPress.com blog could use some alone time with a thigh-master – it’s weighing in at 53 requests and 727 KB transferred for the logged out “about” page. Sadface!

Plugins all the way down

When I set out to build a new Open Source webmail program last year, one of the main things I wanted to accomplish was a robust plugin system. In the past, I designed functionality not unlike the way WordPress plugins work (which is not unlike the way Squirrelmail plugins work, my first exposure to the concept). I think the strength of a plugin system really helps grow an Open Source project. Would WordPress be such a dominant CMS if the plugin ecosystem was not so vast? During its heyday, Squirrelmail had a vibrant add-on community, and a lot of new users were drawn to the project because of it. Many core developers of that project started out writing plugins, including me.

I decided to take a different approach this time around. What if instead of having a core flow of execution that plugins can add to or alter, everything was a plugin? If all the work needed to process a request was modular, it would provide a way for sites to customize almost anything about the application without having to hack a core file or implement a crazy workaround. It would also make it possible to customize the app simply by including the plugins you want. I decided to build the application this way, and while it turned out a teeny-tiny bit complicated, the results are pretty cool if I do say so myself. And I do.

The system is composed of three distinct components. The “framework” which processes a page request and provides high level constructs for things like session management and page routing. “modules sets” that provide specific types of functionality, such as an IMAP module set that let’s you access IMAP E-mail accounts, and individual “modules” inside of sets, that do a small piece of work. The framework provides an execution environment for module sets. Module sets assign modules to request identifiers and define allowed input and output types. Finally, individual modules do the work of processing page input and building page output. Let’s pretend that was not complicated, because there’s more!

Module sets contain two types of modules, “handler” modules, and “output” modules. Handler modules have access to the session, site configuration, user configuration, and white-listed/sanitized input values. Output modules only have access to the data produced by handler modules (and other output modules). A module set combines handler modules that process and take action on user input, with output modules that format the response to be sent to the browser. The result is a one way data flow through module execution, with default (though configurable) immutability. It’s kind of like the principles of React, without the icky JavaScript part.

Module sets can override each other and replace module assignments done by other sets, or insert a module before or after a module assigned by another set. There is only one required set, the “core” modules. Sites can override core functionality in this set, but it must be included for the others to work. Of course, some or all of it could be replaced with something completely site specific, which means you could change the program into virtually any type of web application you want, and still get the benefit of the lightweight framework.

One of the best things about this system is it requires module sets to white-list and type-cast user input. Modules cannot access standard PHP super globals – they must use the provided data framework, because those super globals are mercilessly unset before modules execute. Modules also have easy output escaping and string translating functionality built-in, so there are no complicated procedures to safely output untrusted data. Modules are designed to have a single purpose, so they end up being concise, which makes auditing and testing easier. Modules use inheritance to access data they need, there is no need to resort to global scope (the application framework and all the included module sets don’t use any PHP globals).

Developing modules can be tricky, so there is a built-in “debug mode” that makes it a lot easier to catch common mistakes and see immediate results. In debug mode, module assignments are dynamically determined – you don’t have to rebuild the configuration file for a newly created module to fire. Assets like CSS and JavaScript includes are un-minified and separately requested – so troubleshooting problems from a browser developer console is a lot easier. Debug mode also enables a constant stream of information to the PHP error log about each page request. Of course there is also a “hello world” module set included in the source, with loads of comments and examples.

To date, I have built more than 15 module sets using this system. So far, I think it’s pretty neat. It’s complicated, but it provides a structured way to modify the program with a safer-than-most API while maintaining a concise separation of concerns. It’s definitely more complex than a lot of plugin systems out there, but also more powerful with an eye on overall performance. At the very least, I think we can all agree that “module set” sounds way cooler than “plugin”, so that’s a plus.

Grunter

Editor’s Note: This is a slightly revised version of a post written in 2012 for a now defunct website. By using the “editor’s note” prefix and talking in third person, it is obvious I am not the author of this blog, but it’s editor. In no way are we the same person.

grunterThe Overview
Grunter is more than a new social network communication API web 3.0 paradigm template. Grunter is the new. The sun is setting on Twitter, and Grunter is the force of gravity directing that sun to crash into it. Grunter changes what it means to change – then it changes. Grunter redefines language, communication, society, civilization, and redefinitions. Forever. Like a glorious raging phoenix violently re-birthing from the ashes of Twitter’s cremation, Grunter soars through the digital skies inspiring awe. The world is a hollow and bleak reflection of itself without Grunter. Grunter simply is.

The Question
Everyone agrees Twitter is great, but the messages can be SO long to type and read and they have all kinds of words and stuff. Why can’t it be simpler to share my inane thoughts?”

The Answer
Now it is.

The Basics
Using any one of our soon-to-be-award-winning apps you can send a “grunt” to your “trough”. Grunts can be public or directed to specific users. For convenience you can subscribe to other troughs, in the Grunter parlance this is called “bellying-up”. This allows you to see all the grunts you care about in one combi-trough. You can control who can belly-up to your trough and limit access to specific grunts. Grunted messages can be up to ten characters long, with no spaces.

The Technical Sounding Shit
Grunter leverages distributed systems running on scalable enterprise technologies to provide an unparalleled application framework platform in the cloud. We use a ton of cloud, seriously. Our software ecosystem lays a comprehensive groundwork for a myriad of rich user interface interaction vectors. By mashing up cutting edge engineering methodologies into a single test driven development pattern, we have invented a new process for creating software that combines the best aspects of waterfall, agile, and scrum approaches. We call it Scragifall (TM) and using it guarantees rapidly produced bug-free code even when working with untrained monkeys. Grunter is written in Haskell, then compiled to bytecode, de-compiled to Ruby, and finally run in a VM built in node.js.

The Proof
The best way to convince your frontal lobe that Grunter is superior to all social networking available today is to feed it some examples. Here are some common tweets converted to grunts to give you the idea:

tweet: I’m an idiot who thinks you care about what I put on my bagel
grunt: unggg??!

tweet: I am so going to kick the shit out of you
grunt: Grrrrr!

tweet: I forgot my bib and drooled on my new shirt
grunt: ehhrrm…

tweet: OMG that video was like so cool LOL
grunt: drrrrrpp

The Pudding

“I have never even heard of twit her or whatever, but I do make a lot of guttural noises throughout the day. Grunter was a natural fit for me and it makes communicating online comfortable and fun.”

“I loved using Twitter and the little blue bird was cute, but everybody kept tweeting WAY too much text at a time, I’m talking like information overload! One time I even missed the first act on a new episode of Idol because I was reading some long-ass tweet. Now that I’m on Grunter all that has changed. I can still keep in touch with random strangers AND I never miss a minute of my favorite show! Grrrr!!!”

“I’m not a speed reader, so it can take me a whole day to get through four or five tweets. By the time I finish reading what’s new in the morning I have twice as many stacked up to go through in the afternoon! With Grunter I can quickly read my messages and get on with my day. Thanks to Grunter I have a life again.”

Cypht: New Open Source Webmail

I suck at product announcements, probably because I have no perceptible marketing skills. Usually I just whip out a list of technical details that even I can’t edit because I fall asleep by the third bullet point – A list that significantly limits the number of people who care one iota about what I’m blathering on about. What was I blathering on about? That’s right, sucky product announcements. Let’s pile another one on the heap, shall we?

This is the official wiz-bang super exciting announcement for a NEW shiny NEW Open Source webmail program called “Cypht” (pronounced “sift”). Did I mention it was NEW? New does not necessarily mean better you say? Well, I had not thought about that before deciding to emphasize it. Seriously however, in the world of Open Source webmail, new is actually exciting (and subjective I guess since I have been working on this for like a year already).

Will this software disrupt the E-mail paradigm while streamlining workflow to maximize interpersonal communication channels using a ground breaking application stack and development process? Nope. From a user perspective it’s like a lot of web-based E-mail programs you are already familiar with. From a developer perspective the code is experimental, with a focus on a smart overall design that offsets some of the downfalls of building a complex program in PHP.

Wait, don’t go just because I said PHP. It’s not like the other PHP based webmail programs I have worked on, but it’s also not just another vanilla web-based E-mail client. One of the core ideas and most interesting features is the concept of viewing aggregated E-mail from multiple sources, without actually forwarding the messages to a central account. Let me esplain:

I have a few Gmail accounts. An old Yahoo E-mail. Throw in a few other domains I own and I have about 5 or so accounts I want to keep up with (to varying degrees). I don’t want to auto-forward them all to a single account, I like keeping my work E-mail separate from my personal messages. So I said to myself,

“Self, why not build a webmail application that does that? One that could give self fast access to a list of all the unread messages in all self’s inboxes from the last 2 days. Or something like that. Oh and search and stuff.”

And so it came to pass, that said webmail was created and continues to be tinkered with. I have droned on endlessly about this software on this blog in the past, so if you are one of the people who came here for the “Linux on a Macbook Pro” post (which is why most people come here) and clicked on the wrong link, this probably sounds familiar.

Since this is not really new, not even new to this blog, why the awful product announcement? Is it just an excuse to rehash old post material until someone pays attention to your stupid program? Maybe, but I think we are getting a little off-track, and for the record its my blog so I will be the one asking the questions around here partner.

Did I mention I like lists of technical details? Here comes one!

  • Really small page sizes, like the entire page + JS + CSS + Ajax requests is less than the Google home page. With normal browser caching and HTML5 local session storage the data transferred is reduced to a stupifyingly small amount.
  • An extra emphasis on security and privacy throughout the application. This is an Open Source program meant to be safe and useful, not an ad generating machine mining your E-mail.
  • It’s not your dad’s webmail! Unless you are one of my daughters in which case it is your dad’s webmail.
  • Valid HTML5 pages with accessibility friendly markup and mobile views.
  • A module system that is like a plugin system on crank. It’s all friggen modules!
  • At this point in development, Cypht is a decent E-mail and news reader with limited outbound message support (in this context limited means don’t use it).
  • A huge following on Github of 1, combined with a large developer base of 1. This thing is taking off I tell you!
  • A website with tons of impressive sounding acronyms like IMAP, POP3, SMTP, HHVM, RSS, SSL, PHP, 2FA, PBKDF2, AES, and more!
  • Other stuff!

Still here? I’m as surprised as you are. This concludes today’s craptacular product announcement. This blog will now return to its irregular schedule of posts about video games or goats or whatever.