Cypht – Rebooted

[if you need a refresher about my obsession with writing PHP based Open Source IMAP clients there is no shortage of posts on this blog going into excruciatingly boring detail]

It was about the year 2003 that I first decided to author my own Open Source software. It was both a positive decision that helped me reach future goals, and a constant source of disappointment. I dreamed of building a development community around what I was creating, something organic that would self sustain. That never happened. Over the years I built a community of incredible users and contributors, just not of the I-can-code variety. I also built the webmail client I wanted so that was cool. I repeated this cycle about every 8 years:

  1. Build new webmail project
  2. Interact with a supportive community
  3. Do 99% of the coding and project management
  4. Burn out and or start over

In the two decades since starting this ridiculous journey I have been lucky to have project supporters. They were (and are) critical to the development cycle. But the dream of attracting developers to what I was doing never materialized. Until last year. When, ironically, my time spent with the project was winding down. In this case “winding down” means basically doing nothing.

Suddenly we have a group of new devs submitting PR’s and working on stuff. At first I was like “hell yeah!” then a few months later I was like “shit I don’t have time for this”. Hence, after much delay on my part, Cypht (my latest webmail project) is rising from the ashes of my complacency, to be reborn yet again, but this time in a different way.

Our resurgence is thanks to the contributions of Marc Laporte and his development team at Tiki Wiki CMS Groupware (aka TikiWiki or Tiki). Not only have they infused new life into a stagnant project, but they also provide development opportunities to aspiring coders across the world – giving them tangible work experience that is unavailable locally. Pretty easy to get behind that 🙂

So what changes with the project? Most importantly the incredible efforts of the Tiki development team will be unleashed. Things will actually move forward after many months of delay. Releases will be frequent. Aside from that, nothing. We plan to move the project to an organization account on Github for better collaboration. I will still be involved to the extent I can be, but I will no longer be a blocker to continuous improvement.

I’m so thankful to Marc and his team. At a time when I had all but abandoned the project, they rescued Cypht from the very large no-longer-maintained trash heap of neglected Open Source software projects, and I can’t wait to see what the future brings!

JMAP: It’s like IMAP But Not Really

With the new year upon us, I decided it was time to do something ambitious for my Open Source project. This is where I normally quip about never mentioning my project before, even though I literally always do. A new year means new jokes so I’m moving on, and really you should too. Cypht is not just Open Source webmail, it’s like the [COOL THING OR PERSON] of Open Source webmail. And it’s NEW. Well I have been working on it for 5 years but NEW in a relative sense, since time is relative, and really what is time? Anyway, ambitiousness.

There has been a “Support JMAP to replace IMAP/SMTP” request lingering on our Github for almost a year. It’s in the “I’m still interested but effin busy bucket“. Recently the poster of said issue mentioned the JMAP specification will soon be finalized and it might be a good time to give it another look. What exactly is JMAP? Let me verbally circle around it a bit more and I promise I will get there.

I reread the high level bits about JMAP. Then I started digging in, created a new module set, shit-canned that approach because it was stupid, took a different angle, got obsessed, stayed up way too late on weeknights, permanently pinned the docs in browser tabs, filed a couple of Github issues with the Cyrus IMAP project (both of which were mis-configurations on my end), and as of this week – hit the milestone of initial IMAP/JMAP compatibility.

Again, what the hell is JMAP? As we all know there are two widely adopted protocols for getting your E-mail to your eyeballs: POP3 and IMAP. POP3 was designed to work with systems that are not always online because that really used to be a thing. IMAP is awesome unless you write code for it in which case it will ruin you. Regardless IMAP is the common standard these days. It’s complicated and old and hard. Why can’t there be a modern REST like API that can do everything IMAP does but better? Astute readers may have added this up by now, but just to clarify: JMAP is a modern REST like API that can do everything IMAP does but better.

The driving force behind JMAP from what I can tell is Bron Gondwana, the CEO of FastMail. As a subscriber to the IMAP protocol mailing list for the last 15+ years (also an actual thing), I recognize Bron from many informative replies to befuddled posters about the IMAP protocol, or the Open Source Cyrus IMAP server he contributes to.

I would like to take my remaining time and open up the floor to questions, so fire away.

Internet: So what is the JMAP API like?

Me: I’m not going to sugarcoat it, JMAP is complicated. But it is really (really) well designed. JMAP is a REST API so it uses HTTP requests and responses to issue commands and get the results. Almost all requests in JMAP are to the same URL using an HTTP POST to submit a JSON body of “methods”. A method is an action or query you want to perform like “give me the 20 newest messages in this mailbox” or “flag this important message from a Nigerian prince”. Also they have excellent docs. Sometimes you have to jump around to piece together what you are looking for but it’s pretty comprehensive.

Internet: How is JMAP better than IMAP?

Me: My top 5:

  1. JMAP is sane
  2. JMAP is not designed around a persistent network socket, so it’s perfect for webmail clients that connect, do stuff, then disconnect (which is exactly NOT how IMAP is supposed to be used)
  3. JMAP finally brings pagination support into the picture. This alone is a huge performance boost
  4. JMAP allows you to chain methods together with back-references to earlier methods. This allows you to combine queries and actions into a single API request. I have not really used this yet because my initial implementation is mimicking the more inefficient patterns of IMAP, but I think this is the single coolest part of the JMAP API design.
  5. Uids in JMAP are globally unique. I don’t need to select a mailbox then fetch the content for a uid in that folder – I can just fetch the content for a uid as it’s unique across folders.

Internet: Do you think JMAP will really take off?

Me: JMAP is an open, smart, modern, and powerful E-mail protocol, so probably not. Short of the ground breaking (not really) Cypht webmail program, I would say JMAP is the best thing to happen to E-mail in a LONG time. JMAP also supports calendars and todos and contacts and sending outbound E-mail and push notification and state management and deltas and other things I can’t remember. Right now development versions of the Cyrus IMAP server support JMAP, and FastMail is using it in production for some of their users. I hope it takes off because as an E-mail client writer it’s been an absolute pleasure to work with.

So sorry to say, we are out of time! To conclude I want to thank the Cyrus IMAP developers for prompt and helpful replies to my uninformed questions. JMAP may not be the future of E-mail, but it should be. It’s like IMAP but not really. It’s better.

It’s that time again – Cypht release forthcoming

Oh Yeah! It’s that time again! Time for another Cypht update! Calm down everybody, I know you are excited. Shit I am too! Let’s dance! OK. Seriously, I’m happy to report we have updates worth reporting.

The big news is preliminary EXPERIMENTAL probably buggy PGP support. This module set provides the ability to sign/encrypt outbound mail (plain text only), and decrypt (some) PGP message parts. We are using openpgpjs on the client, and the Gnugpg PECL extension on the server. Private keys never leave your machine and passphrases must be re-entered anytime you want to use a private key. Nobody ever said PGP was user-friendly. This module set does not change that fact. However it does make it possible to use PGP from a webmail client – which is a dicey proposition no matter how you slice it.

Another neat new feature is the ability to compose messages in Markdown format. We are not using a WYSIWYG editor for this, but instead a very cool Markdown editor that has the ability to do a quick preview and some handy buttons for inserting common formatting options. Kids love Markdown don’t ya know. Since Markdown is not a commonly supported MIME format, we are converting the outbound message to HTML (sadface!) to preserve the formatting before sending it on its way.

Probably the best integration test for a webmail client is to have it send a message, await its arrival, then open it. It would be really great to have this happen every time changes are pushed to the repo, and even better with multiple versions of PHP. Well the future is now because we are doing this thanks to Travis CI, Browserstack, and a crap-ton of work to prep a VM with a functional (albeit local) web-server and mail system.

One of the great things about this release is the much improved testing infrastructure. We now have Travis/Browserstack/Scrutinizer/Coveralls running and analyzing PHPUnit test, Selenium tests, and static code on pushes to the repo. Unit testing used to be limited to the framework, a crucial but small amount of the code base. We now have 100% coverage for the core module set as well, and the second most important set, IMAP, is at about 70% coverage (though some tests are lacking proper assertions). There is still a LOT of code un-unit tested, but we are making progress.

Currently Cypht is on a 1.6 year release cycle – this is not good. I just started working on candidates for a long overdue 1.1.0 release. Git is telling me there have been about 600 commits since 1.0.0 was let loose. Unencumbered by facts that is 60% testing updates, 30% bug fixes, 10% new features, and 100% awesome. I’m really excited to get this release out the door because doing releases is a pain and I’m lazy. With some luck I will increase release frequency and thus be forced to automate away my laziness (UPDATE: I already started).

Lets conclude with some fun stats:

First commit:

jason [ ~/cypht ]$ git log | tail -5

commit 04952b3121e162e072712149126253f0adcf08b8
Author: jasonmunro 
Date:   Fri Feb 14 08:45:00 2014 -0800

 Initial commit

Total commits to master:

jason [ ~/cypht ]$ git rev-list --count master

4007

Total lines of PHP (not including third party stuff):

jason [ ~/cypht ]$ find . -name "*.php" -print | \
    grep -v asset | \
    grep -v third_party | \
    grep -v site | \
    xargs wc -l | \
    tail -1

46915 total

Total lines of JavaScript (not including third party stuff):

jason [ ~/cypht ]$ find . -name "site.js" -print | \
    xargs wc -l | \
    tail -1

4892 total

Total lines of CSS (same):

jason [ ~/cypht ]$ find . -name "site.css" -print | \
    xargs wc -l | \
    tail -1

616 total

Total unit tests:

jason [ ~/cypht ]$ find tests/phpunit/ -name "*.php" | \
    xargs grep 'function test_' | \
    wc -l

407

Total UI tests

jason [ ~/cypht ]$ find tests/selenium/ -name "*.py" | \
    egrep -v 'base|creds|runner' | \
    xargs grep 'def ' | \
    grep -v __init__ | \
    wc -l

84

Total lines of Travis CI configuration:

jason [ ~/cypht ]$ find .travis/ .travis.yml -type f -print | \
    xargs wc -l | \
    tail -1

1606 total

Average memory use in KB of a Cypht request using DEBUG mode for 250K logged requests:

jason [ ~/cypht ]$ grep 'Peak Memory' php_errors.log | \
    cut -f 2 -d ':' | \
    awk '{ SUM += $1; n++ } END { print SUM/n }'

2882.96

Switching It Up – Personal Gaming History

Editor’s note: I know the author of this blog (he is not me), and I can assure
you that he has confided in me (I am not him), that this is indeed real
(we both know what is real).

Seriously though this is a truthful account as best as I can remember. Also it
helps if you do the Law & Order “DUN DUN” thing in your head before each section.

YEAR: 1982
AGE: 10
LOCATION: Best friends living room
RESOURCES: Atari 2600; Yars’ Revenge cartridge; milk refills
ATTIRE: Horizontal striped turtleneck, matching light blue Toughskins cords
MISSION: Beat Yars’ Revenge
RESULT: Mission failed; friendship maintained; blisters
NOTES: Yar can suck it
YEAR: 1983
AGE: 11
LOCATION: Strip mall arcade
RESOURCES: 5 dollar bill; half pack Big League Chew
ATTIRE: Horizontal striped turtleneck; matching light blue Toughskins cords
MISSION: Spend wad; win games; look cool; avoid inevitable dork move
RESULT: Mission failed; wad spent; games lost; coolness unachievable; dork move successful
NOTES: Got on the leader board of Food Fight once
YEAR: 1985
AGE: 13
LOCATION: Spare room
RESOURCES: Atari 800XL; BASIC cartridge and manual; limited patience
ATTIRE: Over-sized black button down; bright red cuffed pants; knock-off Air Jordans
MISSION: Create my own game
RESULT: Mission failed; rendered single line in 4 hours; high level cursing; surrender
NOTES: If only I had stuck with it
YEAR: 1991
AGE: 19
LOCATION: University of Massachusetts Amherst
RESOURCES: NES; Legend of Zelda cartridge; bong
ATTIRE: LL Bean rugby shirt; kakis
MISSION: Complete Zelda
RESULT: Mission accomplished; classes avoided; bad tone set for future
NOTES: On completion the game reset with new locations, then crashed and wiped
YEAR: 2001
AGE: 29
LOCATION: Trailer home living room
RESOURCES: Playstation; Final Fantasy 7 CD; cats
ATTIRE: Least dirty option
MISSION: Do all the things in FF7
RESULT: Mission accomplished; laundry undone; deep satisfaction
NOTES: Love me some Chocobo racing more than clean underwear
YEAR: 2005
AGE: 33
LOCATION: Trailer home kitchen
RESOURCES: Compaq Presario; Windows 98; wheat beer
ATTIRE: T-Shirt; cargo shorts
MISSION: Collect MAME ROMs of my favorite games
RESULT: Mission accomplished; laws broken
NOTES: Not proud and not sorry
YEAR: 2009
AGE: 37
LOCATION: Basement apartment
RESOURCES: Dell laptop; external monitor; ramen
ATTIRE: Same t-Shirt; same cargo shorts
MISSION: Run Quake3 at acceptable frame rate on Linux
RESULT: Mission accomplished; rail gun expertise gained
NOTES: Still have the Q3 tin box
YEAR: 2014
AGE: 42
LOCATION: Shop behind garage
RESOURCES: Caanoo portable gaming device; PC; misc cables; processed meat sticks
ATTIRE: T-shirt; shorts optional
MISSION: Load and run MAME ROMs
RESULT: Mission accomplished; accidentally smashed unrepairable screen 1 month later
NOTES: Loved this thing; I’m a moron
YEAR: 2015
AGE: 43
LOCATION: Living room
RESOURCES: PS4; Diablo 3; wife; couple kid-free hours
ATTIRE: Jammies
MISSION: Hook my wife on Diablo 3
RESULT: Mission accomplished; excess paragon levels achieved
NOTES: Good couch co-op games are awesome fun
YEAR: 2017
AGE: 45
LOCATION: Living room
RESOURCES: PS4; Rocket League; wife; ability to ignore kid
ATTIRE: Jammies
MISSION: Hook my wife on Rocket League
RESULT: Mission accomplished; still playing
NOTES: Good couch co-op games are awesome fun
YEAR: 2018
AGE: 46
LOCATION: Anywhere
RESOURCES: Nintendo Switch; life savings; wife’s blessing
ATTIRE: T-shirt; not even pretending to wear pants anymore
MISSION: Enjoy cool portable gaming system
RESULT: Mission accomplished; can’t stop playing Zelda
NOTES: The switch is awesome. I have a PS4 so not really looking for
a console but it rocks as a portable and the ability to use as a console is cool.

* DUN DUN *

I’ll Be ABAC

Remember the first time you used an admin flag to expose a special feature in your software? Pretty cool. Remember the first time you had to pee so bad during a game of hide-and-seek that you stood up behind the wood pile where you were hiding and went for it just as you were being found and the finder ran inside and told all the adults and they laughed at you which caused you to have pee anxiety for the rest of your life? Pretty not cool. Back to the point.

Admins and users, the halcyon days. Admin feature? No problem, just check the admin flag. Regular user? NO SOUP FOR YOU! Simpler times. Over the last few years I have been working on complex permissioning systems for web applications (among other things). Thankfully my Open Source software Cypht, being a sort of old school web app (but also new!), is just users and admins. And admins have very little special sauce. Easy-peasy.

However my bill-paying gig requires fine-grained access rights. In between tangents I’m going to talk about it, because like all shitty software tasks, it eventually ends up on my desk. And like all shitty software tasks, you will have to wade through some super boring domain knowledge to figure things out. We started as a team re-rolling the same system we half-ass implemented in the past, only to realize it’s not good. This led us to think there just may be a third-party solution that would ease the pain and make our system better.

In my ever so humble opinion, third-party solutions NEVER ease the cumulative pain, they only move the pain-ball around to different pain-regions. The system we landed on implements what is known as ABAC, or Attribute Based Access Control. It uses XACML, which is a … weird specification. XACML is both a protocol for enforcing access to a resource, and a system architecture. Like all good specifications it uses XML for data transfer *snort*. More importantly, it renames common operations and parts of the system with unique acronyms so nobody really knows what you are doing or even talking about.

Take this statement for example:

Not sure why the policy request from the PEP for that particular resource and action pair returned indeterminate, so I will have to inspect the PDP to see if it has a problem accessing the PIP to enrich the relevant attributes, as well as verify the request XML from the API. End of line.

This actually makes sense based on the ABAC system I’m working with (except I know that indeterminate responses mean there is no corresponding policy to evaluate the supplied resource and action, so I would never actually say this, but somebody else might, so it’s legit). Also I only say “End of line” in my head after I finish speaking.

We had high hopes the ABAC system would save us silver-bullet style from slogging through building our own. It didn’t. But it is pretty cool. In no particular order, here are some things to consider if you are thinking about implementing an ABAC system, third-party or otherwise.

  • XACML systems only issue permit/deny/WTF responses. This is important so don’t skip this bit. You DON’T ask the system “What does Jason have access to?“, you ask it “Does Jason have access to X?“. It’s a critical difference. X can be a list of things, but keep in mind the longer that list, the more time it will take the system to respond.
  • When deciding if a user can access something, XACML systems can be tied into your standard run of the mill RDMS. In the XACML world, this is called “attribute enrichment from the PIP” – where the PIP (Policy Information Point) is the RDMS.
  • I would very much like to throat punch the person who coined the phrase “attribute enrichment”
  • It’s XML all the way down peeps. The request, the response, even the policy used to say yay or nay in the ABAC system. It’s also super verbose with ample namespace usage. Good times indeed!
  • Using an XACML ABAC system does provide a nice separation of concerns. User roles and rights can still be defined in a RDMS, but they are enforced by the ABAC system. This aspect I very much like.
  • The “resources” used in policies, AKA, what you want access to, have a hierarchy. This allows for a cool fallback to a more generic policy if the specific one requested does not exist. And in the other direction, provides a mechanism to create a one-off policy for a special use case. This I also very much like.
  • Just to re-iterate – If you are looking for a system that provides a list of things a user has access to – ABAC over XACML is not that system. It can kind of sort of be shoe-horned to do that, but it’s hacky at best, and does not scale for shit.

Well there you have it. ABAC/XACML is cool, has some neat features, uses crappy formats, invents its own dialect, and overall is boring enough that I hope you never have to work with it. Next in my new series of “Esoteric software related posts that nobody could possibly find interesting“, I’m going to talk about switching out the terminal emulator I have been using for the last decade. Oh snap, I already did that one.

Either way, you have to love the “I’ll be ABAC” title pun. Pure comedic gold that one. Hasta la vista, baby!