Be Kind, Send Text

There are still some of us out there, an ever dwindling group I’m sure, who don’t want images or borders or colors or bold fonts in an E-mail. I know that using HTML markup to send fancy-pants E-mail is popular with the kids these days, I even begrudgingly support it in my own webmail client. But for the love of all that is holy, if you are going to send HTML formatted E-mails (or any other rich text format), please do it correctly – INCLUDE A TEXT VERSION.

The MIME standard, which defines the structure of “modern” E-mail messages, already supports exactly what you need: Just use “multipart/alternative”, and send a text only version along with the wiz-bang HTML copy you spent a week styling (most of which any sane client will strip out anyway). it’s not rocket science, and even though these standards are decades old, it’s surprising how many companies and services still get it wrong. In the last few weeks alone, I have run across the following:

No text part at all

This is the most common. You didn’t even try to send a text version of the content. Keep in mind, it’s not just curmudgeons like myself who won’t read your message, but people with accessibility requirements (think blind people using screen readers) are also going to have a hard time finding the content hidden in 50 nested divs. Why do you hate blind people?

HTML markup in the text part

You tried, I will give you credit for that. But you failed miserably, because now I get a text message with a bunch of raw HTML tags and useless style rules with some content buried deep within. Thanks, but no thanks.

A text part without the same content

You included a text part. It’s not raw HTML thank goodness. Sadly, it’s just a short message to click a link and read the HTML version online. I’m not going to click that link. Another common one is just an empty text part with no content at all. I appreciate your brevity! My favorite offender in this category is a major bank I do business with. They periodically send me E-mail with an HTML part and text part, yay! unfortunately, the text part is always the same. It simply reads:

TEXT VERSION

Very helpful! I know LARGE BANKS have low standards for software development, but this is ridiculous.

Incorrect or missing character set

I can almost forgive you for this. You sent a text part. It has the same content as the HTML version and isn’t markup. But you used non-ASCII characters and did not include the correct character set header. The result is highly likely to render incorrectly. And you were so close to getting it right!

In conclusion, please be kind to aging oldsters like myself, and send a properly formatted text version of whatever you want me to read. Then I can write happy fun-time blog posts about rainbows instead of ranting about E-mail formats. Eh, I will probably still rant about E-mail formats, but I do like rainbows.

Psql + Vim = Happy Face

I love Vim. I use it to develop software, write blog posts, analyze and manipulate raw data, even compose E-mail. If I’m typing something and it requires more than 4 words, I do it in Vim. I was going to write a post about all the Vim tips and tricks I have consumed over the years, but that information lives almost exclusively in muscle memory, which requires too much effort to extract. Instead, I’m writing about how you can spice up your psql environment with Vim, since I just did that last week, and kind of sort of remember how I did it.

Psql is the command line client for Postgres. It’s a powerful, if somewhat boring, interface to the DB. Type an SQL statement, get an ASCII formatted result. In this way it’s similar to the mysql command line interface for that other RDMS you may have heard of. Psql has some interesting options we can use to bring the Vim goodness. That other client might have similar features, but thankfully I don’t work with that anymore and have all but forgotten it existed at all.

Vi Style Key Bindings

This is like “set -o vi” in bash, but for psql. Support for vi key-binding is provided by the readline or libedit library, so enabling this in psql can be done using the .inputrc file in your home directory. Just create the file if it does not exist, and add the following:

set editing-mode vi
set keymap vi-command

Now you can navigate the Psql command line with some Vim-like shortcuts via a limited command mode accessed with ESC. So cool!

Editing Queries With Vim

Being able to jump around the CLI is nice, but when you are building a complicated query, it’s better to have a full featured editing environment. Psql provides a command, “\e”, that will start an editor with the last query pre-loaded. When exiting the editor, the updated query will be re-run. You can control the editor with the EDITOR environment variable. Just set that to “vim” before starting psql. I define this in my .bashrc file so ANY program respecting the EDITOR variable will use Vim by default, because of course I do. The only other trick here is to make sure Vim knows that you are editing a query so we get fancy-pants syntax highlighting. Adding this to your .vimrc file should do the trick (presuming psql is using /tmp like it does on Debian):

syntax on
au BufRead /tmp/psql.edit.* set syntax=sql

Now I can seamlessly use Vim from psql to develop complex SQL statements. SO GOOD.

Query Results in Vim

In the course of developing said complex SQL statements, I’m running oodles of queries and analyzing the results. This is yet another place we can dollop Vim awesome-sauce into psql. Like the EDITOR environment variable, psql respects the PAGER variable. This defines what program will be used to display results that don’t fit in the available terminal screen real-estate. There are 3 steps to gluing this all together: 1. Setup a custom Vim syntax file to make the results all pretty; 2. create a custom Vim config to be used when displaying results; and 3. Set the correct PAGER value before starting psql.

First up, the syntax highlighting. Create a file called “dbout.vim” and put it in ~/.vim/syntax. If those folders don’t exist in your home directory, create them. Put the following super hacky syntax highlight definitions in that file:

" Vim syntax file
" " Language: db output
" " Maintainer: Jason Munro

syn region Heading start=/^ \l/ end=/[-+]\+$/
syn match Border "|"
syn match IntVal " \d\+\(\n\| \)"
syn match NullVal " NULL\(\n\| \)"
syn match NegVal " -\d\+\(\n\| \)"
syn match FloatVal " \d\+\.\d\+\(\n\| \)"
syn match NegFloatVal " -\d\+\.\d\+\(\\n\| \)"
syn match DateTime "\d\{4}-\d\{2}-\d\{2} \d\{2}:\d\{2}:\d\{2}\(\.\d\{1,}\|\)"
syn match TrueVal " t\(\n\| \)"
syn match FalseVal " f\(\n\| \)"

hi def Heading ctermfg=246
hi def IntVal ctermfg=229
hi def FalseVal ctermfg=88
hi def NullVal ctermfg=242
hi def Border ctermfg=240
hi def NegFloatVal ctermfg=160
hi def FloatVal ctermfg=230
hi def NegVal ctermfg=160
hi def DateTime ctermfg=111
hi def TrueVal ctermfg=64

The color definitions I’m using here expect a 256 color capable terminal, those with less will have to fiddle with the ctermfg values. Next, create a custom Vim configuration for the results that uses the syntax highlighting. Just copy your existing .vimrc to .vimrcpg, and add the following lines:

syntax on
set syntax=dbout

Finally, set the PAGER environment variable before running psql:

export PAGER='vim -R -u ~/.vimrcpg -'

And, voila! Check out the super cool color highlighting from the obviously very useful “test” table I just created!
psqlvim
In conclusion, psql is awesome, but psql with Vim integration is awesome-er.

The Future of E-mail

I think about E-mail a lot. I’m weird that way. I think of E-mail as an annoying neighbor who wanders over to your yard on a regular basis to chit-chat when you are obviously busy – It just won’t go away. It’s like a realization had in the shower in your early 40’s about your hairstyle choice – It hasn’t really changed in 20 years. And (last simile, I promise), like an untreated rash – It’s not going to just fade away. Trust me on that last one.

Every so often somebody declares that “X software” will kill E-mail, usually referring to some sort of instant message program like Slack. I will admit that while I worked at Automattic we used Slack to a much greater degree than E-mail, but they served different purposes. If E-mail is the electronic version of mail, glorified IRC Services like Slack are the electronic version of a phone call. I don’t recall the US Post Office going out of business when the telephone was invented.

After the “E-mail killers” comes the revolutionaries, changing the way you use E-mail forever. It’s new! It’s shiny! Never mind the fact it’s built on the same old plumbing, don’t worry about that! It has smart-<feature>! It solves all the problems with E-mail you didn’t even know you had until we told you! The world is saved! I like exclamation points!

As a webmail developer, I’m not trying to revolutionize anything. I’m certainly not interested in killing off E-mail. Channeling my inner curmudgeon, I actually like E-mail. But I also think there is room for improvement. With my latest Open Source webmail project, called Cypht (which this blog is designed to shill for), I’m trying to address those areas of concern. Unfortunately for you, I’m going to elaborate.

I have lots of E-mail addresses, mostly because I’m a dork, but I bet that if you have any E-mail account at all, you probably have more than one. Work account. Gmail. Maybe your ISP crams one down your throat like mine does. A throw-away account. Some other free E-mail service you signed up for that one time for some secretive reason you don’t want to talk about.

I want a webmail solution that I can host, that gives me direct access to all my accounts. I don’t want to forward everything to the Google content-mining advertisement delivery interface. I don’t want to POP messages from one account to another in a chain of complicated hops that somehow results in two copies of everything. I want an E-mail delivered to account X to stay in account X, I just want to check it at the same time I check Y and Z.

In my not so humble opinion, this is the most useful feature of Cypht – aggregated views from multiple accounts. You can still go old school and browse folder hierarchies like it’s 1999, but you can also see all your unread messages from all your accounts in a single view. Or search them all at once. Even though this software is not even alpha quality yet, the search feature has already saved my butt more than once.

Well shoot, I got so focused on pimping my project that I forgot to say anything meaningful about the future of E-mail. So here are some half-assed predictions. I predict that E-mail will persist for at least another 20 years as one of the main underpinnings of the internet, and that for the most part it will go unchanged. IMAP servers will still be IMAPing, and SMTP servers will keep SMTPing. Or maybe we will all be using Slack to E-mail and back to Slack gateways, so we never have to login to an E-mail client again. Heck if I know, I’m terrible at predicting things. Either way I need to go pick up my rash ointment.

Size Matters

Remember when 3G was the latest greatest mobile network upgrade? I do. It was like a bazillion times faster than my current “data service”. It wasn’t even that long ago (otherwise it would have already been flushed from my aging brain cells like so many other memories). Now, 3G seems closer to no internet at all. I can’t help but cringe every time the LTE icon on my phone is replaced with the dreaded 3G symbol. Even the rotating circle of the icon seems to move at a snail’s pace. What the heck happened? Glad you asked!

The size and complexity of websites and applications is spiraling out of control. It’s a silent epidemic in my not so humble opinion. Probably because the ones it really affects can’t get a contact form to load so they can complain. Seriously people, the amount of junk packed into a single webpage request is off the charts, as demonstrated below:

20150807082113

If that very convincing metric I just made up and spent less than 2 minutes slapping into a chart is not enough proof for you, let’s try some real world tests. First, let’s check 3 websites the kids all love and break ’em down using the chromium web developer console to take a peek under the sheets. Then let’s compare that to Cypht, since everything I write on this blog is really just a pathetic attempt to shill for my Open Source project.

size_matters_google

251 KB transferred with 11 requests. I guess that is “good” by today’s standards. It’s Google so it has to be good, right?

I don’t Facebook (thank god), and I die a little inside when I use the word Facebook as a verb, but I know all the kids are staring at it when they impolitely tread across my lawn, so let’s check it out! The un-cached and logged out homepage is something to behold:

size_matters_facebook

902 KB, 31 requests. Ouch! Almost 1 MB of data to display a LOGGED OUT HOMEPAGE? I need to get into the web-hosting biz STAT!

It’s my understanding kids today are not taught how to read in school anymore, so they are only capable of using a crippling subset of standard language skills to find the only type of media they are prepared to consume – short videos. From what I have observed, The Youtubes is popular with these creatures. Wonder how it fares?

size_matter_youtube

OK, I can cut Youtube some slack, it’s a video site after all, but holy Bos Taurus: 3.6 MB and 136 requests is pretty insanely high for … pretty much anything.

Finally, let’s take a look at the lovingly handcrafted pages of Cypht, my Open Source webmail project. I will even tilt the scales against it by using a LOGGED IN page since the logged out homepage of the application, and the cypht.org site, are so incredibly minimal that comparing them to the behemoths above is at a scale difficult for human intelligence to grasp.

size_matters_cypht

Hmm…. 30.2 KB transferred with 3 requests. Contrary to what I have been told throughout my life *ehem*, maybe size does matter?

* Don’t even get me started on the inability of websites to adhere to standards. I mean seriously don’t, I’m saving that for another post

** Even this venerable WordPress.com blog could use some alone time with a thigh-master – it’s weighing in at 53 requests and 727 KB transferred for the logged out “about” page. Sadface!

Plugins all the way down

When I set out to build a new Open Source webmail program last year, one of the main things I wanted to accomplish was a robust plugin system. In the past, I designed functionality not unlike the way WordPress plugins work (which is not unlike the way Squirrelmail plugins work, my first exposure to the concept). I think the strength of a plugin system really helps grow an Open Source project. Would WordPress be such a dominant CMS if the plugin ecosystem was not so vast? During its heyday, Squirrelmail had a vibrant add-on community, and a lot of new users were drawn to the project because of it. Many core developers of that project started out writing plugins, including me.

I decided to take a different approach this time around. What if instead of having a core flow of execution that plugins can add to or alter, everything was a plugin? If all the work needed to process a request was modular, it would provide a way for sites to customize almost anything about the application without having to hack a core file or implement a crazy workaround. It would also make it possible to customize the app simply by including the plugins you want. I decided to build the application this way, and while it turned out a teeny-tiny bit complicated, the results are pretty cool if I do say so myself. And I do.

The system is composed of three distinct components. The “framework” which processes a page request and provides high level constructs for things like session management and page routing. “modules sets” that provide specific types of functionality, such as an IMAP module set that let’s you access IMAP E-mail accounts, and individual “modules” inside of sets, that do a small piece of work. The framework provides an execution environment for module sets. Module sets assign modules to request identifiers and define allowed input and output types. Finally, individual modules do the work of processing page input and building page output. Let’s pretend that was not complicated, because there’s more!

Module sets contain two types of modules, “handler” modules, and “output” modules. Handler modules have access to the session, site configuration, user configuration, and white-listed/sanitized input values. Output modules only have access to the data produced by handler modules (and other output modules). A module set combines handler modules that process and take action on user input, with output modules that format the response to be sent to the browser. The result is a one way data flow through module execution, with default (though configurable) immutability. It’s kind of like the principles of React, without the icky JavaScript part.

Module sets can override each other and replace module assignments done by other sets, or insert a module before or after a module assigned by another set. There is only one required set, the “core” modules. Sites can override core functionality in this set, but it must be included for the others to work. Of course, some or all of it could be replaced with something completely site specific, which means you could change the program into virtually any type of web application you want, and still get the benefit of the lightweight framework.

One of the best things about this system is it requires module sets to white-list and type-cast user input. Modules cannot access standard PHP super globals – they must use the provided data framework, because those super globals are mercilessly unset before modules execute. Modules also have easy output escaping and string translating functionality built-in, so there are no complicated procedures to safely output untrusted data. Modules are designed to have a single purpose, so they end up being concise, which makes auditing and testing easier. Modules use inheritance to access data they need, there is no need to resort to global scope (the application framework and all the included module sets don’t use any PHP globals).

Developing modules can be tricky, so there is a built-in “debug mode” that makes it a lot easier to catch common mistakes and see immediate results. In debug mode, module assignments are dynamically determined – you don’t have to rebuild the configuration file for a newly created module to fire. Assets like CSS and JavaScript includes are un-minified and separately requested – so troubleshooting problems from a browser developer console is a lot easier. Debug mode also enables a constant stream of information to the PHP error log about each page request. Of course there is also a “hello world” module set included in the source, with loads of comments and examples.

To date, I have built more than 15 module sets using this system. So far, I think it’s pretty neat. It’s complicated, but it provides a structured way to modify the program with a safer-than-most API while maintaining a concise separation of concerns. It’s definitely more complex than a lot of plugin systems out there, but also more powerful with an eye on overall performance. At the very least, I think we can all agree that “module set” sounds way cooler than “plugin”, so that’s a plus.