I randomly happened upon Travis CI a few weeks ago. Travis is a “continuous integration” platform that can be tied to a Github account. Every time a change is pushed to the Github repository, Travis can run all your unit tests, and it can connect to a Selenium grid provider like BrowserStack or Sauce Labs to run Selenium tests. All 3 (Travis, BrowserStack and Sauce Labs) provide free versions of their services for Open Source projects. “This sounds really cool!” I thought. And it is. But it took a wee bit of work to get it all running. By wee bit I mean a veritable shit-ton. Hopefully this post will save someone out there the hours of Travis config tweaking it took me to get everything ship-shape.

Travis has a lot of good online documentation, definitely a useful resource to get you started. Basically what Tavis does is spin up virtual machines, that you can control using its setup script. Then it will run whatever commands you want to execute your tests (in my case PHPUnit tests and Selenium tests written in python). By using it’s ability to create a “build matrix”, you can generate different server configurations to run your tests on.

As I write this I am running a build with 75 combinations (!). 5 versions of PHP x 3 different databases x 5 different web browsers. This build is not very practical since it will take about 6 hours to complete, but I had to try it once because of course I did. My standard build is 15 different server configurations (PHP versions x database types) with 5 different browsers (3 server-side combinations per browser).

In no particular order here are some tips and tricks for various parts of the configuration that took some time to figure out.

PHP versions
Setting up multiple PHP versions is a snap. You just list the ones you want in your .travis.yml file (the main Travis configuration file), and BOOM – it creates new instances for each version. The only problem I ran into with this is that PHP 7 versions do not have the LDAP extension enabled, while PHP 5 versions do. You can work around this by adding the extensions during your setup process to the travis.ini file that will eventually become the php.ini file.

PHP version setup

Fix for missing LDAP extentions in PHP 7

PHPUnit versions
PHPUnit has 2 major versions that cover PHP 5.4 through 7.1, so you will need to make sure you have the right version installed in each instance. The easiest way to do this is to wget, chmod, and mv the correct phar file based on the PHP version. Travis makes the version available as an environment variable during the setup process, so by looking at that you can grab the correct PHPUnit file.

Set up different versions of PHPUnit

Services
If you want a PHP enabled web server for your UI tests, and I assume you do since you read this far into the post, you need to install and configure that yourself. I cobbled together a couple of examples from the Travis docs and some various other blog posts to make this work. The example from the Travis docs works fine for PHP 5, however you have to work around an issue with a missing default pool configuration for FPM using PHP 7. I also wanted an IMAP account Cypht could connect to for a more real world experience, so my setup creates a system user to test with, installs Dovecot IMAP, tweaks the configuration slightly, and starts the IMAP service.

Set up Apache with PHP FPM (and the Python Selenium driver)

Default config file for Apache used in the setup

Default FPM pool file used in the setup that fixes PHP 7

Set up a system user with a known password

Set up Dovecot IMAP

Databases
Databases are as easy as PHP version, just list the ones you want in the main Travis configuration file. However they are not configured in a way applications normally use them. The current database for an instance is in an environment variable, so you can use that to determine which database to bootstrap with whatever tables or users you need. Cypht runs tests across Mysql, Postgresql, and Sqlite3.

Database setup (note this is for the massive 75 instance build. You only need one row without the BROWSER part for each database you want to test)

Bootstrap databases for the Cypht unit tests

Browsers
To run selenium tests you need to connect your Travis instance to a selenium grid, like Sauce Labs or BrowserStack. I prefer BrowserStack, but both are great. The online docs for this are pretty comprehensive, and it took a lot less time than I thought to get this working. Then I tried to use different browsers and ran into a serious problem. Chrome, Firefox, and Safari all worked fine, but Edge and Internet Explorer always failed.

By using the replay feature in BrowserStack, I could see that logins to Cypht failed with those 2 browsers. After much head scratching and keyboard bashing, I realized the issue was that these two browsers will not set cookies when accessing a site with the host name of “localhost”. Thankfully there is a work-around for this. You can force the tests to run locally, but also give them a real host name instead of localhost.

Config entry to use a different host name (the important bits are “hosts” and “forcelocal”)

Limitations for Open Source accounts
Travis will allow Open Source accounts to run 5 parallel instances, however both BrowserStack and Sauce Labs only allow 2 parallel connections to their service. In the Travis dashboard you will want to limit your parallel instances to 2 to match the Selenium provider maximum, otherwise those builds will break.

Return value
After the setup completes, Travis runs your “script command”. The return value of this command or commands will tell Travis if your tests were successful or not. You must be sure to return a non-zero value on failure, otherwise Travis won’t know something went wrong with your tests. You can string commands together with “&&” to build a set of commands to run, which will exit immediately if a non-zero value is returned by any command in the list.

Script command Cypht uses

In conclusion, Travis CI rocks for Open Source integration testing and I highly recommend it. Now I have no excuse to not write more tests!