Selenium Case Study – Part Two: Selenium Remote Control

In the first part of this case study, I covered the usage of Selenium IDE for automating UI testing. The second part will introudce Selenium Remote Control (Selenium RC). Selenium RC allows for tests to be produced in a server side language and then run on browsers and operating systems of your choice. In this case study, the PHP implementation will be used (the PHP version uses the PHPUnit library).

We will start by exporting the tests created in Part One. These will be used to demonstrate the features of Selenium RC, such as the production of test suites and allowing you to use multiple systems/browsers for your testing. Following this I will show what else is possible using Selenium RC and limitations of the system.

Introducing Selenium RC

The graphic on the The Selenium RC Project page illustrates the Selenium RC system perfectly:

Selenium RC setup from http://seleniumhq.org

As the diagram shows, there are three components (from the bottom up):

Server based code – Written in the original language of Selenium RC (Java) or one of countless others, such as .NET, PHP, Ruby or Python. The command set is sent to the remote control system for it to be run. The code can be remote from the other parts of the system. This allows you to test on remote systems with specific configurations (e.g. you may want to run tests in Safari in OSX or Internet Explorer 6 in XP).
The Remote Control System – The powerhouse behind the system. This runs the test by instantiating the browser being used and sending each command to the browser. Selenium Grid extends the system further by allowing multiple instances of the Remote Control System to run at once to speed up test times.
Browsers – for carrying out the tests. These can be one of those shown (Firefox, Internet Explorer or Safari) or a custom browser of your choice (as long as Javascript can be used in it). The browser used is the one on the system the tests are running on so if the version of IE on your system is IE8, that is also the one that will be used in the tests.

It has many benefits over the use of Selenium IDE:

All server side language functionality can be utilised
Run in any javascript supported browser, rather than just Firefox
Ability to log results automatically (in PHP this is done through PHPUnit)
Ability to add lots of custom Javascript into tests, if required
Easy integration into a Continuous Integration solution

Once the Remote Control System has been installed, you just need to write the tests in your chosen language and then run the code. In PHP, this involves running test case(s) or test suite(s) through PHPUnit. Please note: You can run the HTML tests Selenium IDE creates directly on the server but you then miss any server side language based functionality such as selection/iteration.

Installation of Selenium RC and PHPUnit

First, to install and run Selenium RC and PHPUnit.

Selenium RC

The documentation provides a straightforward guide to installing Selenium RC. Once installed Selenium RC needs to be started so that the tests can take place (it is not started automatically when tests are being run). Instructions for running it are found straight after the installation notes. To ensure Selenium RC can be launched easily (I am using Windows), I created a bat file on the desktop with the java command needed to start the system (prefixing the path before the call to the file).

Since we’re using PHP, the notes on specific implementations of Selenium RC do not cater to us. Instead, we need to carry out a separate install of PHPUnit.

PHP Unit

The PHPUnit website provides installation instructions for PHP Unit. Selenum extends the base PHPUnit test class and the extension for it is bundled into the PHPUnit installation. The class structures and running the Selenium based tests are carried ut in the same way as standard PHPUnit tests. The documentation for PHPUnit is excellent and well worth a read to make sure you know the basics of using the system.

To allow PHPUnit to be run easily from the command line in Windows, be sure to update your path settings in the system variables. This allows you to enter “phpunit” in the command line without prefixing the path (see here for more details).

Basic RC Usage: BBC Homepage Tests

The first part of this case study produced a test suite for the BBC Homepage (these can be downloaded here). Selenium IDE allows you to easily export these tests out into the correct format for RC (in all supported languages).

Exporting from Selenium IDE to Selenium RC

Unfortunately, all test cases must be exported individually. Although you can export a test suite in some languages (e.g. Java), this does nothing more than create the test suite class and the actual tests themselves are not exported at the same time.

To export a test, open the test to export and ensure it is selected. Then, enter the File menu within Selenium IDE (not the browser File menu) and then select PHP from the sub-menu of Export Test Case As. Be sure to add the file extension to your test name as these are omitted by default. The export of the first test will look similar to the following:

<?php
 
require_once 'PHPUnit/Extensions/SeleniumTestCase.php';
 
class Example extends PHPUnit_Extensions_SeleniumTestCase
{
  protected function setUp()
  {
    $this->setBrowser("*chrome");
    $this->setBrowserUrl("http://www.bbc.co.uk/");
  }
 
  public function testMyTestCase()
  {
    $this->open("/");
    $this->assertEquals("BBC - Homepage", $this->getTitle());
    $this->click("customiseThisPage_tablink");
    $this->click("resetButton");
    $this->waitForPageToLoad("30000");
    try {
        $this->assertEquals("News", $this->getText("//li[@id='blq-nav-n']/a"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("Sport", $this->getText("//li[@id='blq-nav-s']/a"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("Weather", $this->getText("//li[@id='blq-nav-w']/a"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("iPlayer", $this->getText("//li[@id='blq-nav-i']/a"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("TV", $this->getText("//li[@id='blq-nav-t']/a"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("Radio", $this->getText("//li[@id='blq-nav-r']/a"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertTrue($this->isElementPresent("identifier=blq-blocks"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertTrue($this->isElementPresent("css=#promo_area ol.content li.selected h2 a"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertTrue($this->isElementPresent("css=div#promo_area  .sidebar .firstLink a span"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertTrue($this->isElementPresent("css=div#promo_area  .sidebar .secondLink a span"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertTrue($this->isElementPresent("css=div#promo_area  .sidebar .thirdLink a span"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertTrue($this->isElementPresent("document.getElementById('blq-search')"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("Multifaith Calendar", $this->getText("link=Multifaith Calendar"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("Climate guides", $this->getText("link=Climate guides"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("Humans", $this->getText("link=Humans"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
  }
}
?>

A couple of notes about the implementation:

No Verify commands exist, only assertions. Verification errors are caught but the test will return failure on completion (all verification errors will be shown in the test results).
The default class name is Example so this needs to be changed to the same name as your file.
As with all PHPUnit tests, all test methods must begin with test. You can add multiple tests to the same test case by prefixing each of the methods in this way. The default method name is always testMyTestCase but amend this as necessary.
In the other tests created, echo was used to output variable names. In PHPUnit these are converted to print statements which are not needed and so you will end up removing these.

Running a test

I’ve renamed the test to BBCHomepageElementsTest and made sure the file is given the same name (BBCHomepageElementsTest.php). To run the test in PHP Unit, navigate to the directory your file sits in and then execute the following command:

phpunit BBCHomepageElementsTest

The results are presented in the same way as other PHPUnit tests, with a dot showing for each successful test, an F for a failed test and an E for an error running the script. I would recommend updating the script created to see how failures and errors are returned.

The importance of the test class name

In PHPUnit, the name of your class should be reflected in the name of the file containing it. When running an individual test as we have just done, as long as the name passed to PHPUnit is the filename (minus the extension), the test will still run. When you run multiple tests however, this is not aways the case and so it is best to ensure the names of the class and file are the same.

Running multiple test cases

In PHPUnit, there are several ways of running multiple tests at once and you can do either :

Adding multiple test methods to the same test class – This is a case of just creating an additional method prefixed by test. For instance, you may have a couple of similar tests to run where it makes sense to keep them grouped together in the same file. As with all PHPUnit tests, the setup and tear down methods with9in the class are run before and after each test method in the file. Alternatively you may create a test case for each page of your site, but with the complexity of most modern sites, this may not be feasible.
Running a set of tests as a test suite – You can implicitly create a test suite based on your directory structure (by running all tests in a specified directory), or explicitly by using either an XML based config file or through additional PHP code.

I’m going to take a look at how the rest of the tests created can be used to create a test suite, in addition to the class already exported but the first thing to do is to take a look at the code structure that exists when we export the rest of the test cases.

Refactoring the test cases

One of the issues with the current test cases is the duplication of code. Consider the start of the first two test cases:

They both have identical setup methods and the top set of commands within the test methods are repeated in each case. If we were running all of the tests within a single test case it would be easy to extract the common functions into a single function and then call this when necessary. Since we are not doing this, I will create an abstract class to sit between the actual test cases and the PHPUnit_Extensions_SeleniumTestCase class. Unfortunately the top commands within the test method cannot be moved to setup so an additional method is required. The implementation for this is as follows:

<?php
 
require_once 'PHPUnit/Extensions/SeleniumTestCase.php';
 
abstract class BBCHomepageBaseTest extends PHPUnit_Extensions_SeleniumTestCase
{
 
  protected function setUp()
  {
	$this->setBrowserUrl("http://www.bbc.co.uk/");
	$this->setBrowser("*firefox");
  }
 
  /*common commands required at the start of each test. These cannot be run in the setup as they require the browser to have been opened*/
  protected function init()
  {
	$this->open("/");
	$this->assertEquals("BBC - Homepage", $this->getTitle());
	$this->click("customiseThisPage_tablink");
	$this->click("resetButton");
	$this->waitForPageToLoad("30000");
  }
 
}
?>

The code for the first test case then becomes:

<?php
 
require_once 'BBCHomepageBaseTest.php';
 
class BBCHomepageElementsTest extends BBCHomepageBaseTest
{
 
 
  //Test 1: Checking for each element
  public function testHomepageElements()
  {
    $this->init();//function to revert page back to correct settings
 
	try {
        $this->assertEquals("News", $this->getText("//li[@id='blq-nav-n']/a"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("Sport", $this->getText("//li[@id='blq-nav-s']/a"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("Weather", $this->getText("//li[@id='blq-nav-w']/a"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("iPlayer", $this->getText("//li[@id='blq-nav-i']/a"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("TV", $this->getText("//li[@id='blq-nav-t']/a"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("Radio", $this->getText("//li[@id='blq-nav-r']/a"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertTrue($this->isElementPresent("identifier=blq-blocks"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertTrue($this->isElementPresent("css=#promo_area ol.content li.selected h2 a"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertTrue($this->isElementPresent("css=div#promo_area  .sidebar .firstLink a span"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertTrue($this->isElementPresent("css=div#promo_area  .sidebar .secondLink a span"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertTrue($this->isElementPresent("css=div#promo_area  .sidebar .thirdLink a span"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertTrue($this->isElementPresent("document.getElementById('blq-search')"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("Multifaith Calendar", $this->getText("link=Multifaith Calendar"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("Climate guides", $this->getText("link=Climate guides"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
    try {
        $this->assertEquals("Humans", $this->getText("link=Humans"));
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
        array_push($this->verificationErrors, $e->toString());
    }
  }
 
}
?>

I’ve exported the other tests and refactored them in the same way. These can be downloaded here. First, to make sure they are all working correctly, I will run all of the tests created by running all tests within a specified directory. This is done with the following command (replace . with the path to the directory if you are not in the same directory as your tests):

phpunit .

You should see Selenium RC running each test case in turn. The result for each will appear in the command window as and when each is completed. The failure and error messages will all be displayed once the final test has been run (if there are any).

Please note: Make sure to run each test case you export. I came across two errors while exporting the five tests. One error was that a variable created missed the $ sign in front of the variable name, causing a command to fail unexpectedly. The other issue was the use of a semicolon where is was not needed. Due to the location of it, an error was returned before the test case was able to run.

Creating a PHPUnit Test Suite

By creating a test suite, you can easily adapt how and which tests you run in different situations. You may for instance want all tests within a project to run or just those of an affected section of the site. By composing Test Suites you can make sets of tests as extendible as necessary.

There are a couple of key features which make coded Test Suites most flexible approach to running multiple tests:

Setup and Tear down functions similar to the test case functions can be created for the start/end of each test suite
Test Suites can contain both Test Cases and other Test Suites, allowing the creation of Test Suite Hierarchies.
Fixtures created in the Test Suite are available to all Test Suites/Test Cases contained in the suite.

In this case, the Test Suite will be straightforward. I’ve moved the tests created so far to within a directory called “tests” and have created the Test Suite in the same location as the tests folder:

<?php
/*Need to provide locations for each test case:*/
require_once 'tests/BBCHomepageElementsTest.php';
require_once 'tests/BBCHomepagePromosTest.php';
require_once 'tests/BBCHomepageLinksTest.php';
require_once 'tests/BBCHomepageCustomElementsTest.php';
require_once 'tests/BBCHomepageSearchTest.php';
 
class BBCHomepageTestSuite extends PHPUnit_Framework_TestSuite
{
    public static function suite()
    {
		$testSuite = new BBCHomepageTestSuite();
 
		$testSuite->addTest(new BBCHomepageElementsTest("testHomepageElements"));//must pass in the name of the test method in the class to run
		$testSuite->addTest(new BBCHomepagePromosTest("testPromos"));
		$testSuite->addTest(new BBCHomepageLinksTest("testLinks"));
		$testSuite->addTest(new BBCHomepageCustomElementsTest("testCustomElements"));
		$testSuite->addTest(new BBCHomepageSearchTest("testSearch"));
        return $testSuite;
    }
 
    protected function setUp()
    {
		//run before the first test
    }
 
    protected function tearDown()
    {
		//run after the final test
    }
}
?>

Note that for each test you add, you must pass in both the name of the test case and the method within the test case to run. Since we have no custom code in this suite it would have been easier to create the Test Suite as an XML based config file.

For more details about creating Test Suites I’d recommend reading the Organizing Test Suites chapter of the PHPUnit documentation.

Running Tests in multiple environments

One of the key uses of Selenium generally is to provide a way to check complex AJAX functionality on complex websites. The limitation within Selenium IDE is that it can only be used to run tests in Firefox. Since AJAX implementations tend to vary greatly between some browsers this is a big limitation. As mentioned previously, Selenium RC allows you to run tests in browsers other than Firefox very easily. To demonstrate this, I will add code to the abstract class created previously to ensure the tests created will run both in Firefox, and in the version of IE and chrome loaded on my current system.

There are two ways of specifying which browser/system to use in Selenium. Up until now the setUp method has been used:

<?php 
...
  protected function setUp()
  {
	$this->setBrowserUrl("http://www.bbc.co.uk/");
	$this->setBrowser("*firefox");
  }
...
?>

This allows you to specify a single browser (you can also specify a different host using a similar function). To allow multiple browsers, a public attribute is used instead. Start by removing the setBrowser line within setup. Then add the following with details of your other browsers. In my case all browsers are on my localhost:

<?php
 
require_once 'PHPUnit/Extensions/SeleniumTestCase.php';
 
abstract class BBCHomepageBaseTest extends PHPUnit_Extensions_SeleniumTestCase
{
   	public static $browsers = array(
      array(
        'name'    => 'Firefox on Windows',
        'browser' => '*firefox',
        'host'    => 'localhost',
        'port'    => 4444,
        'timeout' => 30000
      ),
      array(
        'name'    => 'Internet Explorer on Windows',
        'browser' => '*iexploreproxy',
        'host'    => 'localhost',
        'port'    => 4444,
        'timeout' => 30000
      ),     
	 array(
        'name'    => 'Chrome on Windows',
        'browser' => '*googlechrome',
        'host'    => 'localhost',
        'port'    => 4444,
        'timeout' => 30000
      )
 
    ); 
 
  protected function setUp()
  {
	$this->setBrowserUrl("http://www.bbc.co.uk/");
  }
...
?>

Please Note: There are permissions issues using the default *iexplore and so be sure to use *iexploreproxy instead.

Other features

Screenshots of Failed Tests

To get screenshots of failed tests requires a couple more lines of code to be added to the top of the abstract class created earlier:

<?php
 
require_once 'PHPUnit/Extensions/SeleniumTestCase.php';
 
abstract class BBCHomepageBaseTest extends PHPUnit_Extensions_SeleniumTestCase
{
    protected $captureScreenshotOnFailure = TRUE;
    protected $screenshotPath = '/var/www/html/screenshots';
    protected $screenshotUrl = 'http://localhost/screenshots';
...
?>

The screenshots will be a complete screen capture of the page at the time of failure. Induce an error state to see how it works in action.

PHPUnit Features

Features common to all PHPUnit tests are also available to Selenium RC tests in PHPUnit, for instance logging, code coverage checks and easy integration with CI based tools. For more details, see the PHPUnit documentation

Specify Browsers in an XML config

Allowing you to set browsers in one common place, the browsers can be added to an XML config along with any other PHPUnit specific settings. For more details click here.

Taking things further…

In this case study I have concentrated on the features of Selenium RC with PHPUnit and not deviated from the original test cases created in Selenium IDE. The refactoring of the code does show how the system is much more malleable than the IDE version but the key power is what else is possible in Selenium RC. For a complete list of the commands in Selenium RC in PHP, see the PHPUnit documentation. Be sure to note PHPUnit can also run any HTML based tests you have.

I will now consider what else can be done using the system and how other products can make use of Selenium RC during the development process.

PHP/PHPUnit

Rather than being limited to HTML based commands, the use of a PHP provides much more power and flexibility. For instance, test data can be loaded from XML files to store in the system or the latest news on the site can be checked with the latest created in the database. Any PHP functionality required can be added as and when it is needed. As previously stated, the use of PHPUnit provides additional functionality, such as code coverage checks and logging. It’s usage also makes it easy to implement the testing structure within a continuous integration solution using CruiseControl or Atlassian Bamboo.

Use in Frameworks and Content Management Systems

Although a number of Frameworks and Content Management Systems make use of PHPUnit for unit tests (The Zend Framework for instance), each creates a testing framework by extending the PHPUnit_Framework_TestCase which the PHPUnit_Extensions_SeleniumTestCase also extends. Because of this there is no built in framework in any case for Selenium. This being the case, to use Selenium with a framework/CMS, you need to extend the same class we have used in this case study (PHPUnit_Extensions_SeleniumTestCase) and then include the relevant classes you need in the Framework/CMS. It should be noted that although others (e.g. Drupal, CodeIgniter) make use of their own testing framework, there are user contributed implementations of PHPUnit for them (Drupal / CodeIgniter).

Flash

There is a Google Code Project for Flash (and also Flex). This is not currently available for PHP but only in Java, Ruby and .NET. It works by providing an interface from the Selenium RC tests to the functions found within the Flash movie (making use of ExternalInterface within Flash). Since Selenium RC runs commands through Javascript it’s quite a simple extension of that. The Google Project homepage for the project shows a basic example of it’s use. It is essential for the testing of the swf that the functions called within the flash return a value the Selenium test can make use of to verify each command is successful. It should be noted these can be integrated into existing tests alongside other Selenium commands, they do not have to be standalone tests.

Selenium Grid

To make Selenium RC run a lot more quickly, Selenium Grid provides a way to run tests concurrently across a number of machines (Selenium RC can only run one at a time). It works by running multiple instances of Selenium RC in parallel. For more details see the Selenium Grid website.