Selenium provides automated User Interface testing for web applications. There are two main ways of using it:

• Selenium IDE – A firefox plugin that records your actions and allows you to play them back later. The actions are run through the javascript based Selenium Core which is the original version of the testing system. There as quite a few limitations in the JS based language such as a lack of any possible iterative or selection statements (although some things like this can be added through additional extensions, such as this one).
• Selenium Remote Control – A client/server setup which allows you to programme the tests within server based languages to provide a greater level of functionality within your tests. It also allows you to test within particular browser/operating system combinations. For PHP, PHPUnit provides the functionality to allow you to effectively run Selenium tests.

In part one of this case study, I will demonstrate the functionality available within the IDE and the limitations of it. In Part two I will cover the basics of Selenium RC before exporting the tests created by Selenium IDE into PHP files for use with RC. I will then demonstrate how RC can be used to test Flash applications and also it’s usage within a couple of PHP Frameworks and Content Management Systems.

Selenium IDE

The Selenium website provides a good walkthrough of what you can do with Selenium IDE. I will therefore give an overview of what features the IDE has, creating a test suite based on the BBC homepage to show off this functionality.

To use the system, you firstly need to download the Selenium IDE. There are a number of other extensions available for Selenium on the Firefox add ons site. At the very least I would recommend downloading the Selenium Buttons extension. This just gives you a couple of buttons to use on the Firefox main menu rather than having to access the menu option every time you want to use Selenium IDE.

Once you have restated Firefox Selenium is available through the tools menu.

Creating tests using the IDE tends to be a combination of two things:

• Using the firefox based functionality – recording actions and using the contextual menu options added by Selenium IDE.
• Manually entering commands using the Command/Target/Value options in the IDE. Using this is essential for tweaking and editing commands you already have and removing any which are not needed. Since the length of most commands is very small it is quicker to use this than the functionality within the IDE, once you get used to the commands.

Creating tests

The best way of learning how to use the system is to create some tests using it. I’m going to use the current BBC site, concentrating on the current homepage in particular:

This provides a lot of content which can be tested effectively using Selenium, such as the custom content blocks in the centre, the hero carousel at the top and the database driven links throughout the page. It also shows some of the limitations and the difficulty in setting up some tests with Selenium.

I’ve setup a test suite to test the following:

1. Validates key items on the page exist (the logo, top navigation, search bar, heroes and footer links)
2. Verifies the carousel functions correctly
3. Checks links on the page direct you to the correct locations (both static and dynamic links)
4. Customise the page elements – updates the background colour and the blocks displaying on the page
5. Verifies the search functionality works correctly

Each of these items will be created into individual test files. This allows tests to be built, understood and maintained more easily. Selenium IDE allows you to group the tests into a test suite and then run them all at once or to run individual items.

The test suite created can be downloaded here: BBC Homepage test. It is by no means a thorough test of the page however it will demonstrate a lot of the commands present in the IDE. Once I’ve covered ow the test was created, I’ll briefly cover the limitations of the IDE.

Here is a video of the test suite in action:

I’ve skipped the start as there is very little visual effect in those tests (skip to about 52 seconds in if it does not skip for you).

Starting/setting up the test

There are a couple of things needed at the start of each test. Selenium RC in PHPUnit provides the ability to have setup and teardown methods which are run every time you carry out a test but this can not be done using the IDE. For this reason each test in the test suite will include the setup code at the start of it. Once you open the IDE, it will record automatically so make sure it is empty before you start carrying out any actions for the test. the Open the Selenium IDE. If it is already open, make sure that no actions already exist in it (if there are, select the items in it and then just press delete to remove them).
The location for this test is the BBC Homepage. Once you visit this page and start a test, the Base URL at the top of the IDE should state the correct URL (http://www.bbc.co.uk).

First, to check the correct page has been loaded, we can check the page title to confirm we’re on the right page. Selenium adds a lot of contextual options for adding new instructions to the IDE command list. Right click anywhere on the page to bring up the Firefox contextual menu and you should see a list of commands similar to the below which have been added to the bottom of the menu (see the screenshot below). There are two lists: Firstly there are a list of suggests based on what you clicked on. There is then a further list of all possible commands you can use (titled “Show all Available Commands”). Depending on where you click, the initial suggestions for the commands may or may not have the command we need which is titled: “assertTitle BBC – Homepage”. If you cannot see it in the menu on the left, hover over the item called “Show All Available Commands” and it will be listed in the menu which appears. You should see this item be added to the bottom of your command list in the IDE (you will also notice an item called open / which makes use of the base URL we have set).

To test the commands are working we can run the test. Click on either of the play buttons just underneath the Base URL. You will see the page load and then the “assertTitle” line will be tested, hopefully with a successful result. Please note: There are two play buttons as they both work in different ways if you have multiple tests loaded in a test suite. One can be used to run the whole suite of tests whereas the other will run just a singular test (hover over each one to see which is which). In our case we currently only have one test anyway so we can click on either. You will notice the record button will change to it’s “off” state once you start running the test.

Assert or Verify? If you viewed the list of all commands you may well have noticed as well as a set of “assert” commands there was also a list of the same items which used “verify” instead.These commands carry out the same functionality but handle the result in a slightly different way. If you want a test to fail on an error then use an assert statement. This will stop the rest of the test from running. If you use verify, an error will be flagged but selenium will continue with the test. In our case, we need to make sure we are on the BBC Homepage before we run any subsequent tests so it makes sense to use assert. When we check for the presence of items on the page we will revert to using verify instead.

There is one final step we need to take before we run our tests. We will be updating some of the customisation features on the page in one of our tests. Because of this we need to make sure the page is always in a consistent state. Luckily there is a link on the page that will do this for us. Make sure Selenium is recording again and scroll down to the bottom of the page. You will notice tabbed content which looks similar to the below. Click on the second tab and you will see a link in the top right called “Reset Homepage”. Click on this and wait for the page to reload.

You will now notice the actions we carried out have been added to Selenium. Although the ability to do this in this way is very intuitive, this is not always the case and it is often easier to write some commands (especially ones where you are verifying or asserting content) directly into Selenium instead.

Verifying Page elements are present

Now we are ready to go, we’ll start the first test. This will check some key items exist on the homepage:

• Top navigation items
• The BBC logo
• The Search form
• The three page heroes
• Footer Links

We’ll load each of these in slightly different ways to show the flexibility you have in terms of finding and targeting specific items on the page.

Top Navigation Items

The top navigation items appear as follows:

We will do these first as we will use the default functionality to link to them. Right click on news and choose the option verifyText //li[@id='blq-nav-n']/a News. Do this in turn for each of the other items, up to Radio.

So how does this command break down?

What to do?	Where to look?	What to look for?
verifyText	//li[@id='blq-nav-n']/a	News

By default, Selenium looks for the content using either the name of an item (if the element has an ID or name) or failing that, by XPath. XPath can be done manually, or if you have Firebug, right clicking the element you are inspecting an element in the HTML allows you to copy the XPath location of the item (see below).

The resulting XPath from Firebug is not quite so compact, this is the XPath for the News item in the top navigation bar:

/html/body/div/div[2]/div[3]/ul/li/a

As well as XPath, there are therefore other alternatives that can be used:

• By Element ID
• By CSS selectors
• Hyperlinks by Text
• By DOM

We’ll look at how you can use each of these using the other elements on the page (the other menu items can be added to the script using the same technique as before).

The BBC logo

The BBC logo in the top left of the page (see the previous graphic) contains an ID so it is the perfect choice for searching by Element ID.
It is possible to search by ID, name or by either (see the Selenium documentation for details). Since the logo has an ID though, I will search for it solely by ID.

This time I will write the command directly into the IDE. Since we do not have text to search for in this case, we cannot do the same VerifyText statement we used before. Looking through the other commands, the one that stands out is verifyElementPresent which will just tell us if the ID we supply exists on the page. Searching by identifier requires the prefix identifier= at the start of the element you are searching for. We therefore need to look for identifier=blq-blocks since blq-blocks is the name of the identifier. Once entered the script looks as follows:

.

We are only validating the element exists, not the contents of an element so no text is needed under value.

To subsequent commands such as this more quickly, you can do one of two things:

• Run any individual command in your tests by clicking on them
• To run a sequence of commands, right click on the start point and select the menu option Set / Clear Start Point.

Once you have run a line or sequence of lines, it will be highlighted in green or red depending on whether or not it ran successfully.

The Search Box

We’ll locate these using the DOM. DOM elements can be tested using the console within Firebug. Inputting the element and then running it should return the element you are expecting. Although I could find the search box using the DOM and Firebug (document.forms[4].elements["q"]), Selenium could not pick this up. I therefore resorted to the one of other DOM based methods:

document.getElementById(‘blq-search’)

Since this is equivalent to the ID search I have already covered there is not really much point of using the getElementById method, I am only using it due to the issues with the other DOM method in this case.

The full command is:

Command: verifyElementPresent
Target: document.getElementById(‘blq-search’)

The Page Heroes

The hero comprises of two sections. The main area contains elements of the currently selected item. This is taken from the three promos on the right hand side. Rather than being time based, items are selected when you roll over one of the other heroes.

All of the content within the hero is different from that of the promo for it on the right hand side so we cannot compare content between the two Another change for this content is that we do not know what text we will be looking at for the heroes as the content is database driven rather than being static content. Because of this we will test the following:

• Ensure a main hero area exists with content.
• Ensure there are three separate promos on the right hand side.

We’ll later test that the javascript powering the hero section of the page works.

We’ll target the hero section using CSS. This is one of the most powerful ways of targeting content and has the advantage of being the most familiar to most web developers. To use CSS selectors, Firebug is essential. Below shows the structure within the HTML along with the styles each section uses. Based on the structure, we can come up with what we need to target based on this:

So in this case, the following will target the title:

#promo_area ol.content li.selected div.headline h2 a

It should be noted that the BBC Heroes do not have a consistent structure (you can see this by comparing the first hero block with the two below it. Because of this I will simplify the above selector to the following, as it should then work across all heroes:

#promo_area ol.content li.selected h2 a

Now let’s move this into Selenium. To target using CSS, you need to prefix the CSS with css=. Since we do not know the text that will appear in this block, again we will just verify this item exists using verifyElementPresent. Enter the following details into selenium to add it to the bottom of the list of commands:

Command: verifyElementPresent
Target: css=#promo_area ol.content li.selected h2 a

The hero promos are consistent and so are easier to target, using Firebug again we can easily pick up the structure of the three as follows:

• First Item: div#promo_area .sidebar .firstLink a span
• Second Item: div#promo_area .sidebar .secondLink a span
• Third Item: div#promo_area .sidebar .thirdLink a span

Adding them to Selenium is then the same as above, here are the details for the first one:

Command: verifyElementPresent
Target: css=div#promo_area .sidebar .firstLink a span

Once the others are done the Selenium window will look as follows:

Footer Links

I will only complete a couple of these but do it using a text based search. These are possible using the contextual menu. If there are multiple links with the same text on the page then the first link is always used and so care is needed when doing text based links (Selenium automatically picks up on this if you are picking commands from the contextual menu).

For these, right click on some of the links in the footer and choose the item titled verifyText link=? where the question mark is the text for the link. I’ve selected some of the unusual items which only exist in the footer. The final command list for this test is found below (the bottom three items relate to the footer links):

Although a lot of items are not checked the same approaches can be used for the other elements. Personally, I tend to use the CSS selectors as they are extremely powerful and much easier to create and to understand. The rest of this case study will use these where possible as a result.

Verifying the carousel works correctly

The hero carousel works by user interaction rather than being time based. When the user hovers over a promo to the right, the main hero displays the relevant information.

The text on the main hero and within the promo is also different and so we cannot match based on this, once we hover over each item. Instead of this we will need to do the following:

1. Hover over the first hero
2. Store the title of the selected hero
3. Hover over one of the other heros
4. Verify the title is different to the title stored

I’ve setup a second test for this to separate the functionality to be tested. To do this right click in the box on the left hand side which lists the name of the test which already exists and select the New Test Case option. Since it’s a new test, I’ve copied the commands to set up the test ready to begin. In this test, there is the need to store and retrieve variables and also to deal with mouse events.

Dealing with Variables

Variables are essential in the IDE, especially when you want to carry out testing across multiple pages. We are looking at a simple example of it here. Variables can be set using the targeting we have already been using.

Setting Variables can be done through the contextual menu. The issue with doing this in a lot of cases is that the resulting command will use the text as a basis for the variable. Although this is fine for static links, it obviously becomes an issue when looking at dynamic, changeable content. I will therefore store the variable manually. Enter store in the command box and you should be given details about it under the reference box. You provide the expression (usually the target to store) and then under value store the name of the variable. Below is the example I am using:

Command: store
Target: css=#promo_area ol.content li.selected h2 a
Value: selectedPromo

Using variables is just a case of using it in place of text. You do, however, need to display the variable as follows:

${selectedPromo}

Variables can be tested using the echo command to write the variable to the console:

Command: echo
Target: css=${selectedPromo}

Simulating mouse events

Starting to type “Mouse” in the Command box will display a list of all of the mouse events available. We are going to use the mouseOver event. These events are not available through the contextual menu and so they need to be entered directly into the IDE. As an example let’s get Selenium to Roll over the three promos in turn. Enter the bottom three commands as they appear below (these were copied from the verifyElementPresent commands at the top of the list and the commands replaced):

Putting it together

So to test the mouse over functionality we have what we need. We need to use the following commands:

1. mouseOver the first promo
2. store the title text
3. mouseOver the second promo
4. verifyNotText, rather than verifyText, to ensure the text of the title is different from the text we already have stored

Here’s how the completed commands should look, I’ve tried to make sure the titles are not too specific in keeping with the changing structure of the hero on the page. The final test should look as follows:

Checking page links

To test links on the page, I will look at the custom block of News content (found in the left hand column below):

This will allow testing of both static and database driven links. Again, I have setup a new test and included the commands needed to make sure the page is reset.

Static Links

To click on a link on the page, you can either use the record functionality within Selenium IDE (it will automatically pull out the name of the link clicked) or you can enter the clickAndWait command manually, providing the target you want to click on.

In my case, I’ve just made sure Selenium is recorded and then clicked on the News link at the top of the news box. We are now on the News homepage. To check where we are, we can use the same code we used right at the start of this case study, to check the page title. Right click to bring up your contextual menu and choose the option verifyTitle BBC News – Home (note this may be in the first contextual menu but you will probably have to view all commands to access it from the second one).

Before we can check any more links though, we need to go back to the homepage. To do this, right click again to bring up the contextual menu and go to the list of all commands. The very first item will be open /. Click on this command and then double click on it in Selenium to test it works. This will return you back to the Base URL which in our case is the BBC homepage.

The command list for the static links should look as the below:

Dynamic Links

To look at one of the dynamic links, on most sites I would expect to do the following:

• store the link text in a variable
• clickAndWait on the link to visit the news article page
• verifyText on the main h1 title to ensure it matches the link text
• open the base URL

The issue on the BBC homepage though, is that the homepage links quite often use shorter titles than the main news story pages. Looking at the page source, the news article does contain the shortened title as well but it is in the meta data of the page and is not visible to the user.

For information you need to check such as this, when the information is in the head, you can use XPath to navigate to the required element. Selenium provides a verifyAttribute command so you can then test the value of the content attribute to make sure it ties in with the previous page.

• store the link text in a variable
• clickAndWait on the link to visit the news article page
• verifyAttribute using the Meta Tag Headline
• open the base URL

The verifyAttribute method needs to know what attribute is to be returned, along with the usual targeting. This is all set within the Target field in the IDE. You specify what attribute to test using an @attribute at the end of the target. In our case, it will be as follows:

xpath=/html/head/meta[@name='Headline']@content

This is stating: Get the Meta tag named Headline and return the attribute Content

The other commands are similar to those we have used before so the final command set is as follows:

Custom Content Blocks

As well as the news block, there are other custom blocks present on the page. Selenium allows us to test add and remove these (although moving them proves extremely difficult). I’ll start with Removing as that is the easiest, followed by adding new content blocks.

Again, I’ve setup a new test and added the setup sommands used in the other tests.

Removing existing content blocks

Let’s delete the News, Weather and clock blocks on the page. We need to do three things:

• verifyElementPresent to make sure each element exists
• Click to press the close button Please note: we are not using ClickAndWait since we are not reloading the page
• WaitForElementNotPresent to make sure the item gets removed (we may need to wait for a slight delay until an item disappears from view.

The commands are all verify similar to those we have already used, the commands below will remove all of them, each in turn. It is worth setting the start point to the first command and letting them run through with Selenium set up to run at a slower speed (the commands start by the play marker in the command list):

Adding New Content Blocks

These are added through a form at the bottom of the page. Although it can be done manually, this kind of task is easiest when carried out through the Record functionality in the IDE. Press record and navigate to the bottom of the page.You will see tabbed content with the titles Explore the BBC and Customise this page. Click on the Customise this page tab and then click any of the sections not currently on the page. Now click the save button and stop the recording in the IDE. We now just need to check the items were added correctly. Scroll back up the page and for each block you added, you need to add a verifyElementPresent, using the ID of the block as the target. For example, for the history block the command would be:

Command: verifyElementPresent
Target: css=#history
Value:

The command listing should look similar to the below (again, starting from the play button):

Changing the background colour

The background colour s updated in the same way as the content blocks were added to the page. To change the background involves using the Customize this page, click on the theme you want to change it to and then press the save button. Again, I’ve used the record functionality in Selenium IDE for this functionality.

So how’s the theme applied? The theme is specified through a class on the body of the page. To test the theme has been applied, we can use the verifyAttribute function we used previously to access the page meta data, we can check to see if this class exists in the body tag. In this case, the body tag has more than one class so we also need to ensure we account for this by using the Selenium IDE string wildcard (*). The following code carries out the background change and then tests the result:

Search functionality

There are a couple of aspects of the search I will test:

1. Ensure the auto complete displays relevant results
2. Ensure the user is taken to the appropriate results page

This will be the final test. I have again created a new test case and entered the setup commands for the test case.

Testing the autocomplete

The commands for this will do the following:

1. Type into the search box the start of the query (my full query will be The Apprentice)
2. Verify a suggested list of queries comes up including the known suggestion

There are two ways of simulating typing in Selenium:

• Type – In this instance, the text box will not remain in focus afterwards
• TypeKeys – Focus is maintained in the text box at the location you stop typing

We will use TypeKeys as we do not want to simulate a complete query. The type commands do not show up in the contextual menu but it is easy to enter them directly into the IDE. Once we’ve created the TypeKeys command, the suggestions list should appear so we can check it’s working. The following set of commands carries this out:

Testing the Results page

To finish the test, we need to finish writing the query to search for and then redirect the user to the search results page. This should then contain the search term in the main title. The commands are straightforward for this:

Extending Selenium IDE

One of the issues with Selenium is some functionality programmers are used to just does not exist. Selenium provides the ability to create custom javascript extensions to add functionality required for their site. One of the issues I had whilst creating the tests for the BBC Homepage was due to the complexity of the javascript used to move the blocks around the page. Due to this, it was impossible to replicate the user’s actions using Selenium. By writing a custom extension for this page it would be possible to improve the ability of Selenium to carry out this task.

Probably the most well known of the extensions is to provide goto and if statements to allow the branching of the code (it should be noted this can lead to very complex command lists. The original extension for this can be downloaded here. A complete list of extensions can be found here.

Selenium IDE Limitations

There ar equite a few limitations to using Selenium IDE which are important to consider. Some relate to the IDE itself whilst others are an issue within both Selenium IDE and Selenium RC.

Verify Elements have content

When we check for an element being present (verifyElementPresent), we do not actually check whether the element itself has valid content. For instance, the image required may not be found or the title or one of the fields populated from the database may not be displayed properly, leaving us with an empty set of tags. Selenium IDE does not account for this but since RC uses server based languages you do not have the same issue.

Flash content

Selenium IDE provides no Flash support. An extension does exist for Selenium RC (details here which I will be cover briefly in the next post.

DOM issue

Occasional DOM issues such as the inability to be able to target a specific element earlier in the case study. Although this can be worked around using the other techniques for matching elements, it is one to be aware of.

Testing styling

Styling can not be tested to any great degree due to the precision required. On a basic level, how many of the tests carried out in this example would of failed if the main content block being tested had no styling applied? Adding this functionality into your tests in IDE is extremely difficult, especially to a finely tuned level.

Verifying complex Javascript works

Complicated Javascript, for instance the issues with moving custom item blocks around the page did not work in this case (despite Selenium having Drag-Drop functionality). With the complexity of the Javascript in modern web applications it is hard for the IDE to simulate certain situations due to how the web page has been coded.

No access to backend data sources

For instance, if you have caching on your site, you may want to verify the stories on your homepage match the latest items posted within the CMS. Although in this case you could manually enter items into the CMS using Selenium to test, it is adding complexity and it may not always be possible.

Handling non-consistent paths

The BBC promo panel used a different panel structure for some heroes. Although it can be argued that the structure should have been coded consistently for each panel, in certain situations this may not be possible. There is not a native way within Selenium IDE of handling a situation where you have two possible structures an item or some text may be within (some of the Selenium IDE extensions allow you to handle this situation.

Browser Support

Selenium IDE only works in Firefox.

Where to go from here?

Selenium Remote Control extends the functionality provided by Selenium IDE a great deal. The next part of this case study will show how you can export tests created in the Selenium IDE and extend them with a much greater degree of power than you can do in the IDE.

Useful Links

• Selenium Homepage
• Selenium IDE project page
• Selenium IDE Tutorial
• Selenium extensions for Firefox
• Flow Control in Selenium IDE
• Must know commands for Selenium IDE
• Google Testing Blog – My Selenium Tests Aren’t Stable