Testing PHP Network Code

I work on this Open Source webmail client. I don’t think I have ever written about it here before. It’s called Cypht. It connects to services, like an IMAP, SMTP, or POP3 server. It uses the PHP function stream_socket_client to create a connection to these services, then it sends commands and reads responses with standard read/write functions like fgets and fwrite.

Recently I decided I hate myself, so I tried to build a way to unit-test this. Turns out it’s possible, and not nearly as hard as I deserve. I did bang my head around the desk area for a few days figuring it out, so not a total loss. Here is how I did it.

Step 1: Abstract low-down-no-good functions

No matter how amazingly awesome your PHP code base is, if your code actually does anything and you want comprehensive unit test coverage, you have no choice but to abstract a few built-in PHP functions that simply don’t play nice (sessions, cookies, header, curl, streams, you get the picture). I use the following pattern for this:

  • Create a class of all static methods that “wrap” the naughty functions
  • Only define that class at run time if it does not already exist
  • Change your code to call the naughty_class::function version
  • Create the same class in your unit test bootstrap, that has friendly versions of these functions (like doing nothing, or returning true or whatever)
  • Include your unit test version before the run time version when running tests.
  • Realize your wildest dreams of success and good fortune.

An example:

class NaughtyFunctions {
    /**
     * @param string $server host to connect to
     * @param integer $port port to connect to
     * @param integer $errno error number
     * @param string $errstr error string
     * @param integer $mode connection mode
     * @param object $ctx context
    */
    public static function stream_socket_client($server, $port,
        &$errno, &$errstr, $timeout, $mode, $ctx) {
        return stream_socket_client($server.':'.$port, $errno,
            $errstr, $timeout, $mode, $ctx);
    }
}

Instead of calling stream_socket_client in code, we call NaughtyFunctions::stream_socket_client with the same (similar) arguments. This pattern (or something like it) is required to make this work, so no skipping step 1. It’s also a great way to deal with PHP functions that disagree with PHPUnit, and as a way to fool tests into taking a different code path they would not normally take, like by overriding function_exists for example. Here is what Cypht uses at runtime:

https://github.com/jasonmunro/cypht/blob/master/lib/framework.php#L59-L202

Step 2: Build a stream wrapper to fake out your code

In PHP you can fake a “stream” AKA a file handle or network connection, by creating and registering a “stream wrapper“. For file operations and stateless protocols like HTTP, this is pretty simple – read until the “file” ends. But for persistent network protocols, this takes a bit of cleverness.

You need the ability to read from the stream until you reach “End Of File” (EOF). But then you need to reset the EOF status the next time you issue a command, so you can read from the stream again. There is no way (I know of) to do this from within the stream wrapper prototype, and we don’t want to alter the network code we are testing.

Thus the cleverness. Using the abstract in step 1, we can save a reference to the stream resource, and rewind it every time we send a new command, effectively resetting the EOF. Seems less clever now that I write this, but it was the most difficult part.

Here is an example of of both the NaughtyFunctions class and a stream wrapper in action:


/**
 * Generic stream wrapper. This will be extended for protocol
 * specific commands and responses.
 */
class Fake_Server {

    /* position within the response string */
    protected $position;

    /* current response string */
    protected $response = '';

    /* list of commands to responses, varies per protocol */
    public $command_responses = array();

    /* open */
    function stream_open($path, $mode, $options, &$opened) {
        $this->position = 0;
        return true;
    }

    /* read */
    function stream_read($count) {
        $this->position += strlen($this->response);
        return $this->response;
    }

    /* write */
    function stream_write($data) {
        $data = trim($data);

        /* look for and set the correct response */
        if (array_key_exists($data, $this->command_responses)) {
            $this->response =  $this->command_responses[$data];
        }

        /* request not found, so set an error value */
        else {
            $this->response = $this->error_resp($data);
        }
        /* CLEVERNESS: here we rewind the stream so we
           can read from it again */
        rewind(NaughtyFunctions::$resource);
        return (strlen($data)+2);
    }

    /* tell */
    function stream_tell() {
        return $this->position;
    }

    /* seek */
    function stream_seek($pos, $whence) {
        $this->position = 0;
        return true;
    }

    /* end of file */
    function stream_eof() {
        return $this->position >= strlen($this->response);
    }

    /* generic error */
    function error_resp($data) {
        return "ERROR\r\n";
    }
}

/**
 * IMAP specific fake server that extends the generic one
 */
class Fake_IMAP_Server extends Fake_Server {

    /* array of commands and their corresponding responses */
    public $command_responses = array(
        'A1 CAPABILITY' => "* CAPABILITY IMAP4rev1 LITERAL+ ".
            "LOGIN-REFERRALS ID ENABLE AUTH=PLAIN AUTH=CRAM-MD5\r\n",
        /* other commands and responses go here */
    );

    /* IMAP friendly error */
    function error_resp($data) {
        $bits = explode(' ', $data);
        $pre = $bits[0];
        return $pre." BAD Error in IMAP command\r\n";
    }
}

/**
 * Naughty functions wrapper to be used in unit tests. Unlike the
 * run time version, this one returns a "connection" to our fake
 * server.
 */
class NaughtyFunctions {

    /* this will hold a reference to our fake network connection */
    public static $resource = false;

    /* we can toggle this to simulate a bad connection */
    public static $no_stream = false;

    /* fake out stream_socket_client and start the wrapper */
    public static function stream_socket_client($server, $port,
        &$errno, &$errstr, $timeout, $mode, $ctx) {

        /* bad connection */
        if (self::$no_stream) {
            return false;
        }
        /* don't call twice from the same test */
        if (!in_array('foo', stream_get_wrappers(), true)) {
            stream_wrapper_register('foo', 'Fake_IMAP_Server');
        }

        /* open, save a reference to, and return the connection
           to our fake server */
        $res = fopen('foo://', 'w+');
        self::$resource = $res;
        return $res;
    }
}

Step 3: Correlate requests and responses for your protocol

Now all you have to do is map requests to the server with appropriate (or inappropriate) responses to exercise your network code from a unit test. In this case that would be adding to the $command_responses array in Fake_IMAP_Server. This is where we cross over from “cool problem solving” to “incredibly tedious unit test production”. looks like I will be receiving extra punishment after all.

Step 4. See a doctor about your wrist pain from writing all the unit tests

Cypht has about 14,000 lines of code I need to test this way. I’m about 1% through the process. I love that it can be done without standing up an IMAP/POP3/SMTP server, but my fingers hurt just thinking about it.

Cypht Development Update

There have been 350+ commits since Cypht 1.0.0 was released, and in this post I’m going to talk about every single one. Kidding of course, but I do want to share some of the super-cool things that have happened since. As always, I want to thank everyone who has written me an E-mail, filed a bug report, submitted a pull request, joined our IRC channel, looked us up on Google, accidentally stumbled across our website, turned me down for grant money, or even thought about Open Source webmail. You guys and gals are the best!

Libsodium
Just before cutting the release, I merged libsodium support. It was a bigger-ish change than I wanted, but I felt adding the ability to leverage well written crypto was worth it. And of course users without libsodium still use our OpenSSL based encryption. Since that time, the PHP maintainers smartly decided to add libsodium as a core extension instead of a PECL package. They have a different calling convention, but I’m happy to say Cypht already supports both.

Travis CI
Travis CI is a freaking awesome service to run unit tests across different system configurations. And like all services this awesome, it’s free for Open Source projects. I have written about it in the past, and improved how we use it since. Now we have 18 different build combinations, from PHP 5.4 to PHP nightly, with 3 DBs, running 5 up, finishing in under 15 minutes. If that didn’t make any sense to you, it’s OK. Know it’s cool, because it is.

Unit tests
Did somebody mention unit test? Oh yeah, I did! Since early on we have had 100% unit test coverage of the Cypht framework. This is good, because it is the environment Cypht modules run in. But it’s not great, because modules are where the action is. Over the last week I have expanded our unit tests to be able to include modules, and have covered 100% of the only required modules, the “core” set. Since then I have come up with what I think is a novel way to use PHP stream wrappers to fake an external network service (like IMAP) to help expand unit tests to other module sets. I’m looking forward to many hours and sore fingers writing tests for all the module sets. Really I am!

Forward compatible
Thanks to Travis CI, Cypht is working flawlessly with PHP 7, 7.1, and nightly builds (eventually PHP 7.2). Man I love that service!

Integration options
Cypht does things differently than most apps. By design. This can make using it to “add webmail to my dynamic site” a bit tricky. Thanks to some great feedback and testing from supporters, we have really advanced this aspect of the program. With our API login module set, you can integrate SSO (Single-Sign-On) for Cypht with any programming language that can make an HTTP API request and build a dynamic form. We also have some PHP integration options, as well as the ability to code your own session and authentication classes without hacking any Cypht internals.

New profiles
In Cypht 1.0.0, profiles are tied to IMAP accounts, and only 1 per account is supported. Since then they have been rewritten, and now support as many profiles as you want. Profiles allow you to correlate an IMAP/POP3 account with an SMTP account, a signature, reply-to, display name and from address. The code is backwards compatible so existing profiles will be converted into the new format the first time you edit them.

Scrutinizer
Last but not least, I want to give a shout-out to Scrutinizer CI, a very cool static analyzer with a free for Open Source service. Static analysis is imperfect, but a great addition to our development process. Aside from just code quality inspection, Scrutinizer runs 16 security related checks. Cypht only fails 15! Another joke, it passes all of them.

It’s not all unicorn farting rainbows since the release. I really wanted to knock out the PGP module set by this time. The proof of concept is there, I just need to CRUD it up. While we have not added a lot of new features over the last 4 months, we have squashed a TON of wiggly little bugs. I smell another official version coming, and for the most part, it does smell like unicorn rainbow farts.

Cypht 1.0.0 Released

After more than 3 years of work, over 3,300 commits, 8 release candidates, 126 resolved issues, and 35,000+ lines of code, I’m pleased to announce the first official stable release of the Cypht webmail program is now available! As anyone who has worked in creating releases for software knows, it’s hard to draw a line in the sand. There is nothing worse than creating a release only to find out the next day you forgot something critical or missed an important bug fix. At the same time, creating releases is a crucial part of getting your software into the hands of users.

I created the release branch 2 months ago with the hope that it would only take a week or two to work out the kinks. After eight release candidates, we finally hit the “it’s good enough, let’s do this thing” point. The way I’m structuring releases in git is to create a release branch from the master branch, then porting applicable bug fixes from the master branch to the release branch while putting out pre-release candidates. Point releases will come from the same branch, but primary development continues on the master, until the next major release, which starts the process over again. I first learned this style of releasing from the Squirrelmail project lo these many years ago. In those days we used diff and patch to port fixes from trunk. With “git cherry-pick” this process is a LOT easier.

The downside to this approach is that over time the master branch diverges from the release branch, and it can get harder and harder to port fixes. The solution is to release often, effectively “dead-ending” the prior release branches as new ones are created. This is a good thing since it encourages frequent releasing. Enough has changed in the master branch in the last 2 months, I’m already eyeballing a 1.1 feature release.

I want to thank everyone who contributed code, filled out a bug report, sent me an E-mail inquiry, requested a feature, donated a translation, or told me they love/hate it. The primary force behind Cypht development is what I want a webmail client to do, but feedback is super important to broaden our user base. I greatly appreciate everyone’s feedback and support for the project.

If you are looking for a secure, lightweight self-hosted webmail that provides access to all your E-mail accounts from one place, give Cypht a try and let me know what you think!

https://github.com/jasonmunro/cypht/releases/tag/v1.0.0

5 Cool Cypht Webmail Features

Cypht is the Open Source webmail program I have been toiling away on for the last few years. It stands out from the competition because of a few unique options, not that it doesn’t have its own warts. But let’s focus on the positive, and not talk about things like the painful installation process, outstanding bugs, or unfinished features. I wouldn’t be doing a very good job converting this blog into a propaganda platform for the project with a title like “5 Shitty Cypht Webmail Features“. Also, I’m a dork. What I think qualifies as cool is known to be subjective.

1. Stand Alone Authentication
Pretty boring opener, but bear with me. Typically webmail programs are designed to point to an E-mail source, like an IMAP or POP3 server, for authentication. They pass the username and password you give them on to the E-mail server, which then tells the webmail program if you are legit or bogus. Cypht boldly (not really) breaks this paradigm by splitting authentication from your E-mail sources.

We support using LDAP or a database to authenticate users, as well as the old-school method of using a pre-configured IMAP or POP3 server. Adding new authentication mechanisms is designed to be easy (relatively), so any source that can verify your username and password can be coded up. We even support a dynamic login process that lets you pick from common E-mail service providers, and can auto-discover E-mail services for a domain (sometimes).

2. Combined Views
Cypht provides combined lists of E-mail messages from all your accounts. One of the reasons I started working on yet another webmail program, was because I wanted this feature. I spend 95% of my time using these views for my E-mail needs, and very little time browsing folders and pages like it’s 1999 (though Cypht supports this as well)

Show me the latest 20 unread messages from each of my accounts over the last 2 weeks in one list? Done. Search for the boarding pass I misplaced even though I forgot which E-mail account I used to make the reservation, and the plane takes off in 5 minutes? Done (this actually happened to me). If you have more than one E-mail account, combined views quickly become the bee’s knees. If you only have one E-mail address, you probably should have skipped this section.

3. Module Sets
Plugins are cool. Module sets are cooler. First of all, they sound cooler. Secondly, module sets are not just a way to bolt on features. Cypht is entirely built of module sets, and a framework to run them. Only one is required, the “core” set. It does things like basic page layout and login/logout. Everything else (IMAP, SMTP, POP3, RSS, contacts, profiles, the list goes on) is its own module set, and can be enabled or disabled independently.

As if that wasn’t the ultimate in coolness, there’s more! Module sets can override each other. Don’t like the default behavior of a core module? You can change it without hacking a single line of Cypht code by creating your own module set that overrides it. I need a sweater it’s getting so cool in here! There is even some poorly written documentation about module sets for aspiring developers.

4. Focus on Security
Security is serious business for a webmail program. So many attack vectors! From filtering out nasties, to TLS everywhere, to encrypting data at rest – Cypht goes the extra mile to try to cover all the bases. Cypht was built with security and privacy as core design principles.

Securing a complex web application is a process, and we welcome feedback and suggestions to continue to improve. For the gory details, check out our security page with a list of impressive sounding technical stuff.

5. Production and Debug Mode
Cypht has two modes of operation. “Debug” mode is what you use when troubleshooting issues or doing development. “Production” mode is what you use when … in production. Debug mode fire-hoses your PHP log with information about each request, enables all errors and warnings, and activates new modules as you create them for a quick write-then-test cycle. Production mode uses combined and minfied assets, silences warnings, and pre-calculates module dependencies.

If you are looking for a different kind of webmail, one that is lightweight, secure, and has a complicated install process -check out Cypht. Or don’t. It’s cool.

Cypht: New Open Source Webmail

I suck at product announcements, probably because I have no perceptible marketing skills. Usually I just whip out a list of technical details that even I can’t edit because I fall asleep by the third bullet point – A list that significantly limits the number of people who care one iota about what I’m blathering on about. What was I blathering on about? That’s right, sucky product announcements. Let’s pile another one on the heap, shall we?

This is the official wiz-bang super exciting announcement for a NEW shiny NEW Open Source webmail program called “Cypht” (pronounced “sift”). Did I mention it was NEW? New does not necessarily mean better you say? Well, I had not thought about that before deciding to emphasize it. Seriously however, in the world of Open Source webmail, new is actually exciting (and subjective I guess since I have been working on this for like a year already).

Will this software disrupt the E-mail paradigm while streamlining workflow to maximize interpersonal communication channels using a ground breaking application stack and development process? Nope. From a user perspective it’s like a lot of web-based E-mail programs you are already familiar with. From a developer perspective the code is experimental, with a focus on a smart overall design that offsets some of the downfalls of building a complex program in PHP.

Wait, don’t go just because I said PHP. It’s not like the other PHP based webmail programs I have worked on, but it’s also not just another vanilla web-based E-mail client. One of the core ideas and most interesting features is the concept of viewing aggregated E-mail from multiple sources, without actually forwarding the messages to a central account. Let me esplain:

I have a few Gmail accounts. An old Yahoo E-mail. Throw in a few other domains I own and I have about 5 or so accounts I want to keep up with (to varying degrees). I don’t want to auto-forward them all to a single account, I like keeping my work E-mail separate from my personal messages. So I said to myself,

“Self, why not build a webmail application that does that? One that could give self fast access to a list of all the unread messages in all self’s inboxes from the last 2 days. Or something like that. Oh and search and stuff.”

And so it came to pass, that said webmail was created and continues to be tinkered with. I have droned on endlessly about this software on this blog in the past, so if you are one of the people who came here for the “Linux on a Macbook Pro” post (which is why most people come here) and clicked on the wrong link, this probably sounds familiar.

Since this is not really new, not even new to this blog, why the awful product announcement? Is it just an excuse to rehash old post material until someone pays attention to your stupid program? Maybe, but I think we are getting a little off-track, and for the record its my blog so I will be the one asking the questions around here partner.

Did I mention I like lists of technical details? Here comes one!

  • Really small page sizes, like the entire page + JS + CSS + Ajax requests is less than the Google home page. With normal browser caching and HTML5 local session storage the data transferred is reduced to a stupifyingly small amount.
  • An extra emphasis on security and privacy throughout the application. This is an Open Source program meant to be safe and useful, not an ad generating machine mining your E-mail.
  • It’s not your dad’s webmail! Unless you are one of my daughters in which case it is your dad’s webmail.
  • Valid HTML5 pages with accessibility friendly markup and mobile views.
  • A module system that is like a plugin system on crank. It’s all friggen modules!
  • At this point in development, Cypht is a decent E-mail and news reader with limited outbound message support (in this context limited means don’t use it).
  • A huge following on Github of 1, combined with a large developer base of 1. This thing is taking off I tell you!
  • A website with tons of impressive sounding acronyms like IMAP, POP3, SMTP, HHVM, RSS, SSL, PHP, 2FA, PBKDF2, AES, and more!
  • Other stuff!

Still here? I’m as surprised as you are. This concludes today’s craptacular product announcement. This blog will now return to its irregular schedule of posts about video games or goats or whatever.