Cypht Webmail Screen Shots – Mobile Version

I had a request on the “Cypht Webmail Screen Shots” post to share some mobile/responsive views of the program. Since I always aim to please, and since the percentage of people with even a shred of interest in this project is so very very small, here it is!

Overall, my design skills fall somewhere between terrible and total shit. My design premise for the Cypht UI has been: “Don’t pretend you know what colors go well together. Don’t fool yourself into thinking you can make something slick. JUST KEEP IT SIMPLE DUMMY“.

I’m pleasantly surprised with the result. It’s pretty standard webmail fare, super simple, and reasonably consistent across browsers.

cypht_mobile10

The main menu. Available by swiping right on any page, or with the awkward looking triangle in the upper left corner of a page.

cypht_mobile9

I pretty much never write E-mail on my phone. But I could!

cypht_mobile8

Yay, only 1 unread message! Also, if you have not checked out Humble Bundle, you should.

cypht_mobile6

The site settings page

cypht_mobile5

General site settings. Kind of squeezed in there, but not too bad!

cypht_mobile3

Search all the things again!

cypht_mobile4

Read a search result. The message view page is tricky on mobile. This one looks pretty good, but others not so much. It’s a work in progress.

cypht_mobile2

Just a teeny tiny calendar. Nothing to see here.

cypht_mobile1

RSS news feed for my favorite Linux news site.

cypht_mobile15

Server management page

cypht_mobile13

“Quick add” dialog on the server page. So cool.

cypht_mobile14

Keyboard shortcut management.

cypht_mobile16

IMAP folder management.

cypht_mobile11

Save settings page.

There are some noticeable omissions to this list, specifically the contacts management interface and the profiles page. I left profiles out because it looks like dog poop, and I left contacts out because I realized the screenshot was of my actual contacts and that would be dumb to post. It looked pretty good though! All in all, Cypht is pretty responsive. Writing this post helped me identify some areas we need to work on, so thanks to the person who requested it!

5 Cool Cypht Webmail Features

Cypht is the Open Source webmail program I have been toiling away on for the last few years. It stands out from the competition because of a few unique options, not that it doesn’t have its own warts. But let’s focus on the positive, and not talk about things like the painful installation process, outstanding bugs, or unfinished features. I wouldn’t be doing a very good job converting this blog into a propaganda platform for the project with a title like “5 Shitty Cypht Webmail Features“. Also, I’m a dork. What I think qualifies as cool is known to be subjective.

1. Stand Alone Authentication
Pretty boring opener, but bear with me. Typically webmail programs are designed to point to an E-mail source, like an IMAP or POP3 server, for authentication. They pass the username and password you give them on to the E-mail server, which then tells the webmail program if you are legit or bogus. Cypht boldly (not really) breaks this paradigm by splitting authentication from your E-mail sources.

We support using LDAP or a database to authenticate users, as well as the old-school method of using a pre-configured IMAP or POP3 server. Adding new authentication mechanisms is designed to be easy (relatively), so any source that can verify your username and password can be coded up. We even support a dynamic login process that lets you pick from common E-mail service providers, and can auto-discover E-mail services for a domain (sometimes).

2. Combined Views
Cypht provides combined lists of E-mail messages from all your accounts. One of the reasons I started working on yet another webmail program, was because I wanted this feature. I spend 95% of my time using these views for my E-mail needs, and very little time browsing folders and pages like it’s 1999 (though Cypht supports this as well)

Show me the latest 20 unread messages from each of my accounts over the last 2 weeks in one list? Done. Search for the boarding pass I misplaced even though I forgot which E-mail account I used to make the reservation, and the plane takes off in 5 minutes? Done (this actually happened to me). If you have more than one E-mail account, combined views quickly become the bee’s knees. If you only have one E-mail address, you probably should have skipped this section.

3. Module Sets
Plugins are cool. Module sets are cooler. First of all, they sound cooler. Secondly, module sets are not just a way to bolt on features. Cypht is entirely built of module sets, and a framework to run them. Only one is required, the “core” set. It does things like basic page layout and login/logout. Everything else (IMAP, SMTP, POP3, RSS, contacts, profiles, the list goes on) is its own module set, and can be enabled or disabled independently.

As if that wasn’t the ultimate in coolness, there’s more! Module sets can override each other. Don’t like the default behavior of a core module? You can change it without hacking a single line of Cypht code by creating your own module set that overrides it. I need a sweater it’s getting so cool in here! There is even some poorly written documentation about module sets for aspiring developers.

4. Focus on Security
Security is serious business for a webmail program. So many attack vectors! From filtering out nasties, to TLS everywhere, to encrypting data at rest – Cypht goes the extra mile to try to cover all the bases. Cypht was built with security and privacy as core design principles.

Securing a complex web application is a process, and we welcome feedback and suggestions to continue to improve. For the gory details, check out our security page with a list of impressive sounding technical stuff.

5. Production and Debug Mode
Cypht has two modes of operation. “Debug” mode is what you use when troubleshooting issues or doing development. “Production” mode is what you use when … in production. Debug mode fire-hoses your PHP log with information about each request, enables all errors and warnings, and activates new modules as you create them for a quick write-then-test cycle. Production mode uses combined and minfied assets, silences warnings, and pre-calculates module dependencies.

If you are looking for a different kind of webmail, one that is lightweight, secure, and has a complicated install process -check out Cypht. Or don’t. It’s cool.

Plugins all the way down

When I set out to build a new Open Source webmail program last year, one of the main things I wanted to accomplish was a robust plugin system. In the past, I designed functionality not unlike the way WordPress plugins work (which is not unlike the way Squirrelmail plugins work, my first exposure to the concept). I think the strength of a plugin system really helps grow an Open Source project. Would WordPress be such a dominant CMS if the plugin ecosystem was not so vast? During its heyday, Squirrelmail had a vibrant add-on community, and a lot of new users were drawn to the project because of it. Many core developers of that project started out writing plugins, including me.

I decided to take a different approach this time around. What if instead of having a core flow of execution that plugins can add to or alter, everything was a plugin? If all the work needed to process a request was modular, it would provide a way for sites to customize almost anything about the application without having to hack a core file or implement a crazy workaround. It would also make it possible to customize the app simply by including the plugins you want. I decided to build the application this way, and while it turned out a teeny-tiny bit complicated, the results are pretty cool if I do say so myself. And I do.

The system is composed of three distinct components. The “framework” which processes a page request and provides high level constructs for things like session management and page routing. “modules sets” that provide specific types of functionality, such as an IMAP module set that let’s you access IMAP E-mail accounts, and individual “modules” inside of sets, that do a small piece of work. The framework provides an execution environment for module sets. Module sets assign modules to request identifiers and define allowed input and output types. Finally, individual modules do the work of processing page input and building page output. Let’s pretend that was not complicated, because there’s more!

Module sets contain two types of modules, “handler” modules, and “output” modules. Handler modules have access to the session, site configuration, user configuration, and white-listed/sanitized input values. Output modules only have access to the data produced by handler modules (and other output modules). A module set combines handler modules that process and take action on user input, with output modules that format the response to be sent to the browser. The result is a one way data flow through module execution, with default (though configurable) immutability. It’s kind of like the principles of React, without the icky JavaScript part.

Module sets can override each other and replace module assignments done by other sets, or insert a module before or after a module assigned by another set. There is only one required set, the “core” modules. Sites can override core functionality in this set, but it must be included for the others to work. Of course, some or all of it could be replaced with something completely site specific, which means you could change the program into virtually any type of web application you want, and still get the benefit of the lightweight framework.

One of the best things about this system is it requires module sets to white-list and type-cast user input. Modules cannot access standard PHP super globals – they must use the provided data framework, because those super globals are mercilessly unset before modules execute. Modules also have easy output escaping and string translating functionality built-in, so there are no complicated procedures to safely output untrusted data. Modules are designed to have a single purpose, so they end up being concise, which makes auditing and testing easier. Modules use inheritance to access data they need, there is no need to resort to global scope (the application framework and all the included module sets don’t use any PHP globals).

Developing modules can be tricky, so there is a built-in “debug mode” that makes it a lot easier to catch common mistakes and see immediate results. In debug mode, module assignments are dynamically determined – you don’t have to rebuild the configuration file for a newly created module to fire. Assets like CSS and JavaScript includes are un-minified and separately requested – so troubleshooting problems from a browser developer console is a lot easier. Debug mode also enables a constant stream of information to the PHP error log about each page request. Of course there is also a “hello world” module set included in the source, with loads of comments and examples.

To date, I have built more than 15 module sets using this system. So far, I think it’s pretty neat. It’s complicated, but it provides a structured way to modify the program with a safer-than-most API while maintaining a concise separation of concerns. It’s definitely more complex than a lot of plugin systems out there, but also more powerful with an eye on overall performance. At the very least, I think we can all agree that “module set” sounds way cooler than “plugin”, so that’s a plus.

Cypht: New Open Source Webmail

I suck at product announcements, probably because I have no perceptible marketing skills. Usually I just whip out a list of technical details that even I can’t edit because I fall asleep by the third bullet point – A list that significantly limits the number of people who care one iota about what I’m blathering on about. What was I blathering on about? That’s right, sucky product announcements. Let’s pile another one on the heap, shall we?

This is the official wiz-bang super exciting announcement for a NEW shiny NEW Open Source webmail program called “Cypht” (pronounced “sift”). Did I mention it was NEW? New does not necessarily mean better you say? Well, I had not thought about that before deciding to emphasize it. Seriously however, in the world of Open Source webmail, new is actually exciting (and subjective I guess since I have been working on this for like a year already).

Will this software disrupt the E-mail paradigm while streamlining workflow to maximize interpersonal communication channels using a ground breaking application stack and development process? Nope. From a user perspective it’s like a lot of web-based E-mail programs you are already familiar with. From a developer perspective the code is experimental, with a focus on a smart overall design that offsets some of the downfalls of building a complex program in PHP.

Wait, don’t go just because I said PHP. It’s not like the other PHP based webmail programs I have worked on, but it’s also not just another vanilla web-based E-mail client. One of the core ideas and most interesting features is the concept of viewing aggregated E-mail from multiple sources, without actually forwarding the messages to a central account. Let me esplain:

I have a few Gmail accounts. An old Yahoo E-mail. Throw in a few other domains I own and I have about 5 or so accounts I want to keep up with (to varying degrees). I don’t want to auto-forward them all to a single account, I like keeping my work E-mail separate from my personal messages. So I said to myself,

“Self, why not build a webmail application that does that? One that could give self fast access to a list of all the unread messages in all self’s inboxes from the last 2 days. Or something like that. Oh and search and stuff.”

And so it came to pass, that said webmail was created and continues to be tinkered with. I have droned on endlessly about this software on this blog in the past, so if you are one of the people who came here for the “Linux on a Macbook Pro” post (which is why most people come here) and clicked on the wrong link, this probably sounds familiar.

Since this is not really new, not even new to this blog, why the awful product announcement? Is it just an excuse to rehash old post material until someone pays attention to your stupid program? Maybe, but I think we are getting a little off-track, and for the record its my blog so I will be the one asking the questions around here partner.

Did I mention I like lists of technical details? Here comes one!

  • Really small page sizes, like the entire page + JS + CSS + Ajax requests is less than the Google home page. With normal browser caching and HTML5 local session storage the data transferred is reduced to a stupifyingly small amount.
  • An extra emphasis on security and privacy throughout the application. This is an Open Source program meant to be safe and useful, not an ad generating machine mining your E-mail.
  • It’s not your dad’s webmail! Unless you are one of my daughters in which case it is your dad’s webmail.
  • Valid HTML5 pages with accessibility friendly markup and mobile views.
  • A module system that is like a plugin system on crank. It’s all friggen modules!
  • At this point in development, Cypht is a decent E-mail and news reader with limited outbound message support (in this context limited means don’t use it).
  • A huge following on Github of 1, combined with a large developer base of 1. This thing is taking off I tell you!
  • A website with tons of impressive sounding acronyms like IMAP, POP3, SMTP, HHVM, RSS, SSL, PHP, 2FA, PBKDF2, AES, and more!
  • Other stuff!

Still here? I’m as surprised as you are. This concludes today’s craptacular product announcement. This blog will now return to its irregular schedule of posts about video games or goats or whatever.

Fun With PHPUnit

What is the first word that pops into your head when you think about unit-testing? I’m guessing “Fun”, amiright? Ok maybe not fun. Unless repeatedly slamming your head against a brick wall is your idea of fun. Trying to build comprehensive tests for existing code is an exercise in patience. Like the Olympics of patience. Sometimes when I come up for air and wipe the blood from my head (and wall), I realize that while I was toiling away in the unit-test dungeon, I stumbled on something useful. Maybe it’s a feature I had not explored before, or a solution to a tricky situation. Maybe it’s something that will help inject a little fun into your PHPUnit experience.

Before I start throwing out random ideas, let me say that I really like PHPUnit: lots of knobs, good documentation, active development. PHP is the Yoga instructor of programming languages, but that flexibility means it’s easy to write poor quality code. When code and unit-tests don’t get along, the right answer is to send the code to bed without dinner, and if it’s really bad, ground it for the rest of the week. We can blame the code all we want, but sometimes there is no choice but to tweak the system to find a way to make it all work.

One of the biggest obstacles in testing is state. Things like global variables, static values or class instances can easily create a situation in which a test passes when run alone, but fails when run as a part of a suite. There are other types of states to consider aside from just the disastrous PHP global namespace. Sessions for example. Or the state of data in a test db or on disk. Tests that fail intermittently are hard to debug, and usually it’s due to an overlooked state issue.

Tests should be as insulated and independent as possible, and PHPUnit has a feature called “runInSeperateProcess” that forces each test to run in its own PHP process. Using this feature is not as simple as it sounds. If you have a bootstrap file defined in your phpunit.xml file, and it includes any code that defines a constant, you will get an error about redefining said constant in your tests. What gives? I thought each test runs in its own process? It does, but from what I can determine (using the throw-it-against-the-wall method), only the test code itself is run per-process. Assuming that poorly substantiated statement to be true, here is a pattern that does work with process separation.

Process Separation

First the phpunit.xml file, WITHOUT a bootstrap

<phpunit strict="true" colors="true">
    <testsuite name="my_awesome_tests">
<file>./my_awesome_tests.php</file>

Then the my_awesome_tests.php file, with the bootstrap included in the setUp() method and the @runInSeperateProcess and @preserveGlobalState annotations.

<?php

class My_Awesome_Tests extends PHPUnit_Framework_TestCase {

    public function setUp() {
        require 'bootstrap.php';
    }

    /**
     * @preserveGlobalState disabled
     * @runInSeparateProcess
     */
    public function my_test_one() {
    }
    /**
     * @preserveGlobalState disabled
     * @runInSeparateProcess
     */
    public function my_test_two() {
    }
}
?>

Finally, the bootstrap looks something like this

<?php

/* all the things */
error_reporting(E_ALL | E_STRICT);

/* determine current absolute path used for require statements */
define('APP_PATH', dirname(dirname(__FILE__)).'/');

/* get mock objects */
require APP_PATH.'tests/mocks.php';

/* get the code we want to test */
require APP_PATH.'lib/framework.php';

/* get the stubs */
require APP_PATH.'tests/stubs.php';

?>

mocks.php contains stand-in objects used as arguments to methods we want to test. The stubs.php file contains wrappers around abstract classes and traits so we can test them more easily. One advantage of this pattern is it makes it possible to pre-define constants in the setUp() method before the code being tested is loaded, so a test can exercise a code path that triggers on a non-default constant value (Assuming the code being tested checks for an already defined constant). Since mocks are loaded before the code being tested, we can also leverage this to override things unfriendly to testing.

Overriding Stuff

It’s good practice to limit mocking and overriding to a minimum. The more code that is mocked out, the less actual code that is being tested. There are however some built-in PHP functions that simply don’t play nice, like setcookie or header or session_start or error_log or die – you get the idea. Using the pattern as described above in “Process Separation”, we can easily add some override behavior to deal with these problems (sadly this does require changes to the code being tested).

In our mocks.php file we create a class of all static methods for built-in functions that don’t play well with others.

class BuiltIns {
    public static function php_header($header_str) { return true; }
    public static function php_die($msg) { return true; }
    public static function php_error_log($str) { return true; }
}

In the code to be tested, we setup a mirror image of this class that runs the actual built in functions, but only loads if the class is not yet defined.

if (!class_exists('BuiltIns')) {
    class BuiltIns {
        public static function php_header($header_str) {
            return header($header_str);
        }
        public static function php_die($msg=false) {
            return die($msg);
        }
        public static function php_error_log($str) {
            return error_log($str);
        }
    }
}

Then we replace occurrences of these built-in functions in the code to be tested. So this:

if ($error_str) {
   error_log($error_str);
}

Becomes this:

if ($error_str) {
   Builtins::php_error_log($error_str);
}

WOOT! Now we don’t have to worry about an errand error_log spoiling our unit-test party. We can even do something useful in the mocked out versions, maybe a fake session_start() call can populate $_SESSION or another constant in setUp can toggle success or failure from the mocked out function. The sky is the limit people!

Coverage

I have only recently started to look at PHPUnit’s coverage options. When I first tried it out, it bailed with a cryptic message and I was sad. Some head scratching and a few apt-get installs later, I was blown away. The HTML coverage report is incredibly useful. By default it sucks up other included files, so if you are dealing with a big code-base it can be handy to limit coverage to just code you are actively testing. I like to define these limits and enable the coverage report in my phpunit.xml file with something like this:

    <filter>
        <whitelist addUncoveredFilesFromWhitelist="false" processUncoveredFilesFromWhitelist="false">
            <directory suffix=".php">../lib</directory>
            <exclude>
            <file>../lib/not_tested.php</file>
            </exclude>
        </whitelist>
    </filter>
    <logging>
        <log type="coverage-html" target="./coverage.html" charset="UTF-8" highlight="false" lowUpperBound="35" highLowerBound="70"/>
    </logging>

The report is comprehensive. It has summary charts for coverage, complexity, risk analysis, and even line by line detail that makes it brain-dead easy to see what your tests are hitting, and more importantly, what they are missing. Coverage alone does not make a good unit test, but it’s a great tool to help improve your tests.

Extending PHPUnit_Framework_TestCase

This post is really dragging on so I will leave you with one additional trick I came across building unit-tests for a billing system. We wanted a test suite that we could run across a variety of products, but the tests code would be nearly identical. Duplicating the tests for each product was a maintenance nightmare. We needed a way to run virtually the same test code across multiple products. Here is what I came up with:

Start by extending the PHPUnit_Framework_TestCase class with an abstract class. This will be where the actual test code lives.

<?php

abstract class Base_Tests extends PHPUnit_Framework_TestCase {
    public function test_something() {
    /* your normal test code goes here */
    }
}
?>

Next extend that class for each product you want to test, and define the product as a member variable in the setUp() method so it can be accessed from the test method scope:

<?php
class Product_A_Tests extends Base_Tests {
    public function setUp() {
        $this->product = 'product a';
    }
}
?>

PHPUnit will not try to run the tests in the abstract class, but it will find and run the tests in the classes that extend it. For each product you want to test, just extend the base class and define the product details in setUp().

I’m hardly a PHPUnit expert, and there are surely improvements to these ideas or even completely better ways to accomplish the same thing. All of these examples minus the last one were taken from a unit-test suite for some Open Source software I’m working on. You can see the still-in-progress test code at Github, and an example of the coverage report from PHPUnit at jasonmunro.net.