Running PHPUnit Tests Outside VVV

Authored by

We use Varying Vagrant Vagrants (VVV) as the development environment for as many projects as we can. It provides a great foundation not only for developing client projects but also for core development itself.

One great thing about VVV is that it installs all of the tools needed to run the core unit tests right out of the box. Our wp-dev-lib project has a PHPUnit plugin bootstrap that makes use of the core unit tests as installed by VVV, and we have a pre-commit hook which can run the plugin PHPUnit tests inside the VM even when making a commit from outside (on the host machine). This ability to run PHPUnit tests inside the VM makes it easy for getting developers quickly set up to run PHPUnit tests, and it’s also something that is suggested for core’s PHPUnit Grunt task (#36190).

Nevertheless…

Even though VVV lets you run core’s PHPUnit tests right out of the box, the tests can run more slowly in the box. If you have a pre-commit hook that runs unit tests with each commit, your workflow can be really slowed down (especially since the tests should eventually by run by Travis CI after pushing anyway). To reduce the temptation to git commit --no-verify to bypass PHPUnit from holding up your commit, you can also run PHPUnit tests out the box, that is to say, outside of the box. (Aside: wp-dev-lib also allows you to selectively skip PHPUnit when committing.)

Running PHPUnit on host connected to DB in guest

Assuming you already have PHP installed, first install PHPUnit on your host machine (e.g. via brew install phpunit). Secondly you need to set the WP_TESTS_DIR environment variable to point to where the PHPUnit tests are located on your host machine (inside the Vagrant synced folder). For example, on my machine I’ve modified my .bash_profile to include:

export WP_TESTS_DIR=~/vvv/www/wordpress-develop/tests/phpunit/

With a PR I made to VVV, the wp-tests-config.php knows whether or not PHPUnit is running inside Vagrant or not. If it detects it is running outside the Vagrant box, it then sets the DB_HOST to be the IP address of the guest VM on the private network (192.168.50.4) in the default Vagrantfile. This allows PHP on your host machine to connect to MySQL in the guest machine.

And with that, you should be able to just run phpunit on your host machine’s terminal and the unit tests should run immediately without having to SSH into Vagrant first to run the tests. When the wp-dev-lib pre-commit hook sees that phpunit is installed and the WP_TESTS_DIR environment variable is defined, it will run the PHPUnit tests from the host machine without stopping to SSH into the VM. This results in a much faster perceived performance improvement to running the tests. But what about actual overall performance?

$ time vagrant ssh -c 'cd /srv/www/wordpress-develop; phpunit'
2m55.007s

$ time phpunit
3m1.222s

So it turns out that running PHPUnit inside the VM is actually faster, although not by a whole lot (~10 seconds). Nevertheless, this went contrary to my expectations 🙁 It seems that latency for PHP connecting to MySQL over the SSH connection is slightly greater than the extra cycles needed to run PHP in the VM.

Let’s try again, but this time run a subset of the test suite, for instance the customize group:

$ time phpunit --group customize
0m11.172s

$ time vagrant ssh -c 'cd /srv/www/wordpress-develop; phpunit --group customize'
0m15.727s

Notice here that running PHPUnit from the host is now significantly faster. 🙂 Since fewer tests are run, the SSH connection to the VM takes a more significant chunk of the time relatively, making it more worthwhile to run them from the host. And in reality, it is unlikely that you’ll be needing to run the full WP core test suite with each release. It is much more likely that you’ll only be wanting to run comparatively very few tests specifically for a plugin you are developing. For instance, running the unit tests for Customize Snapshots takes 0m4.871s from host, but 0m10.724s when connecting over SSH to run completely in the VM: running from the host is over twice as fast.

Running PHPUnit with DB entirely on host

What about abandoning the use of Vagrant altogether for running PHPUnit tests?

Let’s try running PHPUnit with the database completely local in the host to compare with running the tests entirely or partly in the VM.

  1. Assuming you can install MySQL via Homebrew:
    brew install mysql
    ln -sfv /usr/local/opt/mysql/*.plist ~/Library/LaunchAgents
    launchctl load ~/Library/LaunchAgents/homebrew.mxcl.mysql.plist
    

    This will install MySQL which has a root user with a blank password.

  2. Next create the wordpress_unit_tests database:
    CREATE DATABASE IF NOT EXISTS wordpress_unit_tests CHARACTER SET utf8;
    
  3. Then amend your .bash_profile with the following:
    export WP_TESTS_DB_HOST=localhost
    export WP_TESTS_DB_NAME=wordpress_unit_tests
    export WP_TESTS_DB_USER=root
    export WP_TESTS_DB_PASSWORD=''
    

    (Don’t forget to re-source your .bash_profile, or just re-open your terminal app.)

  4. Lastly, update your wp-tests-config.php as seen in PR #846 for VVV.

With these steps in place, you should now be able to run PHPUnit tests entirely on your host machine without any dependencies on the VM. Running the entire PHPUnit test suite with this configuration now results in:

$ time phpunit
2m28.080s

So basically the tests run 20% faster, as compared with 2m55.007s for running in Vagrant over SSH. And as for running the customize group:

$ phpunit --group customize
0m10.895s

This is about 1 second (10%) faster, as compared with running PHPUnit from host with DB connection to VM. Running unit tests for the Customize Snapshots plugin is also about 10% faster (half a second).

So for normal sanely-sized PHPUnit test suites for plugins, the difference in speed is negligible between running them entirely on the host machine versus running PHP on the host machine with a connection to the database on the VM. In both cases the unit tests will run faster than if you have to connect over SSH to run them entirely inside the Vagrant box.

I hope this improves you workflow and encourages you to write more PHPUnit tests for your projects!

One thought on “Running PHPUnit Tests Outside VVV”

  1. If you’re using homebrew to install PHPUnit as referenced above and are wondering why brew install phpunit returns no results, run this command first:

    brew tap josegonzalez/php

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.