E-mail

by

JMAP: It’s like IMAP But Not Really

With the new year upon us, I decided it was time to do something ambitious for my Open Source project. This is where I normally quip about never mentioning my project before, even though I literally always do. A new year means new jokes so I’m moving on, and really you should too. Cypht is not just Open Source webmail, it’s like the [COOL THING OR PERSON] of Open Source webmail. And it’s NEW. Well I have been working on it for 5 years but NEW in a relative sense, since time is relative, and really what is time? Anyway, ambitiousness.

There has been a “Support JMAP to replace IMAP/SMTP” request lingering on our Github for almost a year. It’s in the “I’m still interested but effin busy bucket“. Recently the poster of said issue mentioned the JMAP specification will soon be finalized and it might be a good time to give it another look. What exactly is JMAP? Let me verbally circle around it a bit more and I promise I will get there.

I reread the high level bits about JMAP. Then I started digging in, created a new module set, shit-canned that approach because it was stupid, took a different angle, got obsessed, stayed up way too late on weeknights, permanently pinned the docs in browser tabs, filed a couple of Github issues with the Cyrus IMAP project (both of which were mis-configurations on my end), and as of this week – hit the milestone of initial IMAP/JMAP compatibility.

Again, what the hell is JMAP? As we all know there are two widely adopted protocols for getting your E-mail to your eyeballs: POP3 and IMAP. POP3 was designed to work with systems that are not always online because that really used to be a thing. IMAP is awesome unless you write code for it in which case it will ruin you. Regardless IMAP is the common standard these days. It’s complicated and old and hard. Why can’t there be a modern REST like API that can do everything IMAP does but better? Astute readers may have added this up by now, but just to clarify: JMAP is a modern REST like API that can do everything IMAP does but better.

The driving force behind JMAP from what I can tell is Bron Gondwana, the CEO of FastMail. As a subscriber to the IMAP protocol mailing list for the last 15+ years (also an actual thing), I recognize Bron from many informative replies to befuddled posters about the IMAP protocol, or the Open Source Cyrus IMAP server he contributes to.

I would like to take my remaining time and open up the floor to questions, so fire away.

Internet: So what is the JMAP API like?

Me: I’m not going to sugarcoat it, JMAP is complicated. But it is really (really) well designed. JMAP is a REST API so it uses HTTP requests and responses to issue commands and get the results. Almost all requests in JMAP are to the same URL using an HTTP POST to submit a JSON body of “methods”. A method is an action or query you want to perform like “give me the 20 newest messages in this mailbox” or “flag this important message from a Nigerian prince”. Also they have excellent docs. Sometimes you have to jump around to piece together what you are looking for but it’s pretty comprehensive.

Internet: How is JMAP better than IMAP?

Me: My top 5:

  1. JMAP is sane
  2. JMAP is not designed around a persistent network socket, so it’s perfect for webmail clients that connect, do stuff, then disconnect (which is exactly NOT how IMAP is supposed to be used)
  3. JMAP finally brings pagination support into the picture. This alone is a huge performance boost
  4. JMAP allows you to chain methods together with back-references to earlier methods. This allows you to combine queries and actions into a single API request. I have not really used this yet because my initial implementation is mimicking the more inefficient patterns of IMAP, but I think this is the single coolest part of the JMAP API design.
  5. Uids in JMAP are globally unique. I don’t need to select a mailbox then fetch the content for a uid in that folder – I can just fetch the content for a uid as it’s unique across folders.

Internet: Do you think JMAP will really take off?

Me: JMAP is an open, smart, modern, and powerful E-mail protocol, so probably not. Short of the ground breaking (not really) Cypht webmail program, I would say JMAP is the best thing to happen to E-mail in a LONG time. JMAP also supports calendars and todos and contacts and sending outbound E-mail and push notification and state management and deltas and other things I can’t remember. Right now development versions of the Cyrus IMAP server support JMAP, and FastMail is using it in production for some of their users. I hope it takes off because as an E-mail client writer it’s been an absolute pleasure to work with.

So sorry to say, we are out of time! To conclude I want to thank the Cyrus IMAP developers for prompt and helpful replies to my uninformed questions. JMAP may not be the future of E-mail, but it should be. It’s like IMAP but not really. It’s better.

by

It’s that time again – Cypht release forthcoming

Oh Yeah! It’s that time again! Time for another Cypht update! Calm down everybody, I know you are excited. Shit I am too! Let’s dance! OK. Seriously, I’m happy to report we have updates worth reporting.

The big news is preliminary EXPERIMENTAL probably buggy PGP support. This module set provides the ability to sign/encrypt outbound mail (plain text only), and decrypt (some) PGP message parts. We are using openpgpjs on the client, and the Gnugpg PECL extension on the server. Private keys never leave your machine and passphrases must be re-entered anytime you want to use a private key. Nobody ever said PGP was user-friendly. This module set does not change that fact. However it does make it possible to use PGP from a webmail client – which is a dicey proposition no matter how you slice it.

Another neat new feature is the ability to compose messages in Markdown format. We are not using a WYSIWYG editor for this, but instead a very cool Markdown editor that has the ability to do a quick preview and some handy buttons for inserting common formatting options. Kids love Markdown don’t ya know. Since Markdown is not a commonly supported MIME format, we are converting the outbound message to HTML (sadface!) to preserve the formatting before sending it on its way.

Probably the best integration test for a webmail client is to have it send a message, await its arrival, then open it. It would be really great to have this happen every time changes are pushed to the repo, and even better with multiple versions of PHP. Well the future is now because we are doing this thanks to Travis CI, Browserstack, and a crap-ton of work to prep a VM with a functional (albeit local) web-server and mail system.

One of the great things about this release is the much improved testing infrastructure. We now have Travis/Browserstack/Scrutinizer/Coveralls running and analyzing PHPUnit test, Selenium tests, and static code on pushes to the repo. Unit testing used to be limited to the framework, a crucial but small amount of the code base. We now have 100% coverage for the core module set as well, and the second most important set, IMAP, is at about 70% coverage (though some tests are lacking proper assertions). There is still a LOT of code un-unit tested, but we are making progress.

Currently Cypht is on a 1.6 year release cycle – this is not good. I just started working on candidates for a long overdue 1.1.0 release. Git is telling me there have been about 600 commits since 1.0.0 was let loose. Unencumbered by facts that is 60% testing updates, 30% bug fixes, 10% new features, and 100% awesome. I’m really excited to get this release out the door because doing releases is a pain and I’m lazy. With some luck I will increase release frequency and thus be forced to automate away my laziness (UPDATE: I already started).

Lets conclude with some fun stats:

First commit:

jason [ ~/cypht ]$ git log | tail -5

commit 04952b3121e162e072712149126253f0adcf08b8
Author: jasonmunro 
Date:   Fri Feb 14 08:45:00 2014 -0800

 Initial commit

Total commits to master:

jason [ ~/cypht ]$ git rev-list --count master

4007

Total lines of PHP (not including third party stuff):

jason [ ~/cypht ]$ find . -name "*.php" -print | \
    grep -v asset | \
    grep -v third_party | \
    grep -v site | \
    xargs wc -l | \
    tail -1

46915 total

Total lines of JavaScript (not including third party stuff):

jason [ ~/cypht ]$ find . -name "site.js" -print | \
    xargs wc -l | \
    tail -1

4892 total

Total lines of CSS (same):

jason [ ~/cypht ]$ find . -name "site.css" -print | \
    xargs wc -l | \
    tail -1

616 total

Total unit tests:

jason [ ~/cypht ]$ find tests/phpunit/ -name "*.php" | \
    xargs grep 'function test_' | \
    wc -l

407

Total UI tests

jason [ ~/cypht ]$ find tests/selenium/ -name "*.py" | \
    egrep -v 'base|creds|runner' | \
    xargs grep 'def ' | \
    grep -v __init__ | \
    wc -l

84

Total lines of Travis CI configuration:

jason [ ~/cypht ]$ find .travis/ .travis.yml -type f -print | \
    xargs wc -l | \
    tail -1

1606 total

Average memory use in KB of a Cypht request using DEBUG mode for 250K logged requests:

jason [ ~/cypht ]$ grep 'Peak Memory' php_errors.log | \
    cut -f 2 -d ':' | \
    awk '{ SUM += $1; n++ } END { print SUM/n }'

2882.96
by

Cypht Development Update

There have been 350+ commits since Cypht 1.0.0 was released, and in this post I’m going to talk about every single one. Kidding of course, but I do want to share some of the super-cool things that have happened since. As always, I want to thank everyone who has written me an E-mail, filed a bug report, submitted a pull request, joined our IRC channel, looked us up on Google, accidentally stumbled across our website, turned me down for grant money, or even thought about Open Source webmail. You guys and gals are the best!

Libsodium
Just before cutting the release, I merged libsodium support. It was a bigger-ish change than I wanted, but I felt adding the ability to leverage well written crypto was worth it. And of course users without libsodium still use our OpenSSL based encryption. Since that time, the PHP maintainers smartly decided to add libsodium as a core extension instead of a PECL package. They have a different calling convention, but I’m happy to say Cypht already supports both.

Travis CI
Travis CI is a freaking awesome service to run unit tests across different system configurations. And like all services this awesome, it’s free for Open Source projects. I have written about it in the past, and improved how we use it since. Now we have 18 different build combinations, from PHP 5.4 to PHP nightly, with 3 DBs, running 5 up, finishing in under 15 minutes. If that didn’t make any sense to you, it’s OK. Know it’s cool, because it is.

Unit tests
Did somebody mention unit test? Oh yeah, I did! Since early on we have had 100% unit test coverage of the Cypht framework. This is good, because it is the environment Cypht modules run in. But it’s not great, because modules are where the action is. Over the last week I have expanded our unit tests to be able to include modules, and have covered 100% of the only required modules, the “core” set. Since then I have come up with what I think is a novel way to use PHP stream wrappers to fake an external network service (like IMAP) to help expand unit tests to other module sets. I’m looking forward to many hours and sore fingers writing tests for all the module sets. Really I am!

Forward compatible
Thanks to Travis CI, Cypht is working flawlessly with PHP 7, 7.1, and nightly builds (eventually PHP 7.2). Man I love that service!

Integration options
Cypht does things differently than most apps. By design. This can make using it to “add webmail to my dynamic site” a bit tricky. Thanks to some great feedback and testing from supporters, we have really advanced this aspect of the program. With our API login module set, you can integrate SSO (Single-Sign-On) for Cypht with any programming language that can make an HTTP API request and build a dynamic form. We also have some PHP integration options, as well as the ability to code your own session and authentication classes without hacking any Cypht internals.

New profiles
In Cypht 1.0.0, profiles are tied to IMAP accounts, and only 1 per account is supported. Since then they have been rewritten, and now support as many profiles as you want. Profiles allow you to correlate an IMAP/POP3 account with an SMTP account, a signature, reply-to, display name and from address. The code is backwards compatible so existing profiles will be converted into the new format the first time you edit them.

Scrutinizer
Last but not least, I want to give a shout-out to Scrutinizer CI, a very cool static analyzer with a free for Open Source service. Static analysis is imperfect, but a great addition to our development process. Aside from just code quality inspection, Scrutinizer runs 16 security related checks. Cypht only fails 15! Another joke, it passes all of them.

It’s not all unicorn farting rainbows since the release. I really wanted to knock out the PGP module set by this time. The proof of concept is there, I just need to CRUD it up. While we have not added a lot of new features over the last 4 months, we have squashed a TON of wiggly little bugs. I smell another official version coming, and for the most part, it does smell like unicorn rainbow farts.

by

Cypht 1.0.0 Released

After more than 3 years of work, over 3,300 commits, 8 release candidates, 126 resolved issues, and 35,000+ lines of code, I’m pleased to announce the first official stable release of the Cypht webmail program is now available! As anyone who has worked in creating releases for software knows, it’s hard to draw a line in the sand. There is nothing worse than creating a release only to find out the next day you forgot something critical or missed an important bug fix. At the same time, creating releases is a crucial part of getting your software into the hands of users.

I created the release branch 2 months ago with the hope that it would only take a week or two to work out the kinks. After eight release candidates, we finally hit the “it’s good enough, let’s do this thing” point. The way I’m structuring releases in git is to create a release branch from the master branch, then porting applicable bug fixes from the master branch to the release branch while putting out pre-release candidates. Point releases will come from the same branch, but primary development continues on the master, until the next major release, which starts the process over again. I first learned this style of releasing from the Squirrelmail project lo these many years ago. In those days we used diff and patch to port fixes from trunk. With “git cherry-pick” this process is a LOT easier.

The downside to this approach is that over time the master branch diverges from the release branch, and it can get harder and harder to port fixes. The solution is to release often, effectively “dead-ending” the prior release branches as new ones are created. This is a good thing since it encourages frequent releasing. Enough has changed in the master branch in the last 2 months, I’m already eyeballing a 1.1 feature release.

I want to thank everyone who contributed code, filled out a bug report, sent me an E-mail inquiry, requested a feature, donated a translation, or told me they love/hate it. The primary force behind Cypht development is what I want a webmail client to do, but feedback is super important to broaden our user base. I greatly appreciate everyone’s feedback and support for the project.

If you are looking for a secure, lightweight self-hosted webmail that provides access to all your E-mail accounts from one place, give Cypht a try and let me know what you think!

https://github.com/jasonmunro/cypht/releases/tag/v1.0.0

by

Continuous testing for Cypht with Travis CI and BrowserStack

I randomly happened upon Travis CI a few weeks ago. Travis is a “continuous integration” platform that can be tied to a Github account. Every time a change is pushed to the Github repository, Travis can run all your unit tests, and it can connect to a Selenium grid provider like BrowserStack or Sauce Labs to run Selenium tests. All 3 (Travis, BrowserStack and Sauce Labs) provide free versions of their services for Open Source projects. “This sounds really cool!” I thought. And it is. But it took a wee bit of work to get it all running. By wee bit I mean a veritable shit-ton. Hopefully this post will save someone out there the hours of Travis config tweaking it took me to get everything ship-shape.

Travis has a lot of good online documentation, definitely a useful resource to get you started. Basically what Tavis does is spin up virtual machines, that you can control using its setup script. Then it will run whatever commands you want to execute your tests (in my case PHPUnit tests and Selenium tests written in python). By using it’s ability to create a “build matrix”, you can generate different server configurations to run your tests on.

As I write this I am running a build with 75 combinations (!). 5 versions of PHP x 3 different databases x 5 different web browsers. This build is not very practical since it will take about 6 hours to complete, but I had to try it once because of course I did. My standard build is 15 different server configurations (PHP versions x database types) with 5 different browsers (3 server-side combinations per browser).

In no particular order here are some tips and tricks for various parts of the configuration that took some time to figure out.

PHP versions
Setting up multiple PHP versions is a snap. You just list the ones you want in your .travis.yml file (the main Travis configuration file), and BOOM – it creates new instances for each version. The only problem I ran into with this is that PHP 7 versions do not have the LDAP extension enabled, while PHP 5 versions do. You can work around this by adding the extensions during your setup process to the travis.ini file that will eventually become the php.ini file.

PHP version setup

Fix for missing LDAP extentions in PHP 7

PHPUnit versions
PHPUnit has 2 major versions that cover PHP 5.4 through 7.1, so you will need to make sure you have the right version installed in each instance. The easiest way to do this is to wget, chmod, and mv the correct phar file based on the PHP version. Travis makes the version available as an environment variable during the setup process, so by looking at that you can grab the correct PHPUnit file.

Set up different versions of PHPUnit

Services
If you want a PHP enabled web server for your UI tests, and I assume you do since you read this far into the post, you need to install and configure that yourself. I cobbled together a couple of examples from the Travis docs and some various other blog posts to make this work. The example from the Travis docs works fine for PHP 5, however you have to work around an issue with a missing default pool configuration for FPM using PHP 7. I also wanted an IMAP account Cypht could connect to for a more real world experience, so my setup creates a system user to test with, installs Dovecot IMAP, tweaks the configuration slightly, and starts the IMAP service.

Set up Apache with PHP FPM (and the Python Selenium driver)

Default config file for Apache used in the setup

Default FPM pool file used in the setup that fixes PHP 7

Set up a system user with a known password

Set up Dovecot IMAP

Databases
Databases are as easy as PHP version, just list the ones you want in the main Travis configuration file. However they are not configured in a way applications normally use them. The current database for an instance is in an environment variable, so you can use that to determine which database to bootstrap with whatever tables or users you need. Cypht runs tests across Mysql, Postgresql, and Sqlite3.

Database setup (note this is for the massive 75 instance build. You only need one row without the BROWSER part for each database you want to test)

Bootstrap databases for the Cypht unit tests

Browsers
To run selenium tests you need to connect your Travis instance to a selenium grid, like Sauce Labs or BrowserStack. I prefer BrowserStack, but both are great. The online docs for this are pretty comprehensive, and it took a lot less time than I thought to get this working. Then I tried to use different browsers and ran into a serious problem. Chrome, Firefox, and Safari all worked fine, but Edge and Internet Explorer always failed.

By using the replay feature in BrowserStack, I could see that logins to Cypht failed with those 2 browsers. After much head scratching and keyboard bashing, I realized the issue was that these two browsers will not set cookies when accessing a site with the host name of “localhost”. Thankfully there is a work-around for this. You can force the tests to run locally, but also give them a real host name instead of localhost.

Config entry to use a different host name (the important bits are “hosts” and “forcelocal”)

Limitations for Open Source accounts
Travis will allow Open Source accounts to run 5 parallel instances, however both BrowserStack and Sauce Labs only allow 2 parallel connections to their service. In the Travis dashboard you will want to limit your parallel instances to 2 to match the Selenium provider maximum, otherwise those builds will break.

Return value
After the setup completes, Travis runs your “script command”. The return value of this command or commands will tell Travis if your tests were successful or not. You must be sure to return a non-zero value on failure, otherwise Travis won’t know something went wrong with your tests. You can string commands together with “&&” to build a set of commands to run, which will exit immediately if a non-zero value is returned by any command in the list.

Script command Cypht uses

In conclusion, Travis CI rocks for Open Source integration testing and I highly recommend it. Now I have no excuse to not write more tests!