The WooCommerce core development team introduced an end-to-end testing suite back in March, which aims to dramatically reduce the time it takes to run through tests before big releases.

In a previous post, Getting Started with End-to-End Testing for WooCommerce, I introduced the WooCommerce E2E testing suite and ended the article by identifying some of the suite’s limitations.

In this post, I want to look further afield and showcase some of the alternative products and/or services that have emerged to help you integrate E2E testing into your workflow.

I’ll look at:

• Codeception
• Ghost Inspector
• Usetrace

And then I’ll finish with a brief look at where Robot Ninja, our own automated E2E testing solution, fits into this landscape.

CodeCeption

CodeCeption is a testing framework that runs on PHPUnit, which adds browser testing capabilities.

One of the great things about CodeCeption is that it utilizes PHP, which every WooCommerce and WordPress developer should know. For this reason, it is more suited to developers and those that want to really get their hands dirty with code.

If you are already familiar with writing unit tests (using PHPUnit), it should be very easy for you to get CodeCeption up and running.

Using Codeception with WordPress

Another great thing about CodeCeption is that it is extensible. There is a WP Browser package available that adds a set of WordPress specific extensions/modules to Codeception.

You can learn more about it here.

Writing Code for Codeception

One of the downsides of Codeception (for the non developers) is that you need to write and maintain your own tests.

The E2E test suite that is part of WooCommerce core uses a different language so it can’t be re-used in Codeception.

That said, the coding style of Codeception is very readable and easy to write with. To show you how readable Codeception code is, take look at the following example (taken from the Codeception Getting Started Guide).

Even if you aren’t a developer it is easy to understand what’s going on here. Starting on the fourth line you’ve got:

• I am a user (so we’re establishing the context and preconditions).
• On the next couple of lines, we help build/define the user story and make it more readable i.e. I want to attempt to log in to the website to access features for logged-in users
• Next we navigate to the login page and check that “i am on that page”
• I fill in the username and password fields
• I click the button or link called “Login”
• I check if i can see the text ‘Hello, davert ’

That’s a pretty basic example of an end-to-end test, but that’s just a taste of what writing E2E tests with Codeception is like.

Codeception in Practice

We’ve used Codeception to help test WooCommerce Subscriptions (one of our products at Prospress). Specifically, we created tests for the shipping method logic to ensure methods and prices were displayed as expected on the frontend.

Other great development teams, like the gang at Delicious Brains, have also posted in much more detail about their experience using Codeception to automate their plugin testing.

Ghost Inspector

If writing code/tests is not your thing there are also services like Ghost Inspector.

Ghost inspector offers automated browser testing with the ability to record the tests you want to perform using a Chrome browser extension.

The extension works by recording the actions you perform in the browser, such as clicking and filling in forms, so that those actions can be formed into a test and the same sequence can be re-run from the Ghost Inspector service. Once you click “Start Recording”, the extension will begin recording your actions.

Assertions are the confirmations that something is true. For example, that an element exists on the page or contains specific text. This is how you define what you want to verify and that your test is working as it should.

Ghost Inspector is a great service, especially if you want the flexibility of custom tests without the need to write code. The downside is that you still need to spend time planning out the tests, and recording and maintaining them.

UseTrace

UseTrace is another automated testing service focused on individuals that don’t want to write or maintain test code.

UseTrace’s automated tests are called “traces”. Traces are visual descriptions of the user paths through the application under test.

Unlike Ghost Inspector, traces are made by interacting with your site as your users would through the UseTrace online editor.

A trace includes steps (i.e. buttons and clicks) and checks (think assertions as per Ghost Inspector) for elements that are visible on the page, e.g. text. UseTrace records these interactions into routines/traces that can then be reused.

The end result, like Ghost Inspector, is “a highly maintainable automation test suite for your application’s user interface”.

If using a Chrome extension to create your test suite isn’t your thing, then we would definitely recommend checking out UseTrace.

Robot Ninja

And then we have Robot Ninja.

Just under 12 months ago, my colleague Jason and I began researching/exploring new opportunities for Prospress, which you can read about here in our post introducing Robot Ninja. Basically, we carried out a lot of market research and discovered that that store managers and developers frequently test their sites manually.

Many people we talked to were very concerned that their carts and checkouts would break after updates. We wanted to help people automate this process and make it as simple for them as possible.

With Robot Ninja, there’s no need to write any code, use tools like Git or GitHub or install any packages via the terminal.

We maintain a suite of tests that developers and store managers can run at any time (or on a schedule) to make sure their store is working as expected.

Currently, we are focusing on the cart and checkout process – this was the most important thing our research identified that people wanted tested. Robot Ninja checks things like core pages are loading (shop, cart pages etc.) existing customers can log in, guests can successfully checkout and existing customers can return to make more purchases. All with real products and using real payments

With Robot Ninja you can:

• Immediately see the results from the last test that was run;
• Add and manage a list of the stores you want to test;
• Tell Robot Ninja to queue up tests now and watch as they’re processed; or
• Schedule recurring automated tests for later, like if you want to run tests every third day or weekly on a schedule;
• You also get an email notification when your store has failed a test with more information about what happened. A screenshot is also taken of your site when a test fails to help you diagnose the cause;
• On the main dashboard you can see your own test history along with a the store configuration at the time of running the test. This is to help see if any recent plugin or theme updates, or perhaps other changes have caused a failure;
• When testing that store’s checkout works, Robot Ninja will use real payments, real transactions and live products to test the checkout (not the WooCommerce dummy ones).

Like UseTrace and Ghost Inspector, we want to make the service as simple to use as possible so that those who don’t want to write tests/code don’t need to.

Summing Up

Codeception is a great choice for PHP developers, those already familiar with PHPUnit or individuals that perhaps want more control/ownership over their tests. It’s also free to get started with.

If you are not familiar with PHP or just want to avoid code, using the visual test builders offered by Ghost Inspector or UseTrace is definitely the way to go. The feature set of these services are fairly similar so I think it will come down to pricing and your preference over use of a chrome extension vs online tool to record your tests.

With the above options there is still some investment required to actually come up with the tests and maintain them. That’s were we hope to make Robot Ninja shine by lowering the barrier to testing WooCommerce stores by offering a great set of tests out of the box that all WooCommerce stores can benefit from.

Most people will agree that testing is something that often gets set to the side, or is the last thing people want to look at. This trend towards tools and technologies that make testing more easy and flexible is something we think is great to see and we look forward to seeing just how far it goes – it can only benefit the WooCommerce community as whole.

Note: This post was adapted from a presentation Robot Ninja developer Matt Allan gave at the WooCommerce Vancouver meetup on 7 June, 2017 about E2E Testing with WooCommerce. If you’re interested in learning more about testing or would like someone on our team to give a presentation at your Meetup or WordCamp, get in touch.