Intive-FDV Blog

The Eternal Quest for Stability in Selenium

I have tried to achieve tests stability in Selenium for years. We often take it as a fact that the changing nature of web development has a direct impact on our testing, but I can tell by experience that in general, instability isn’t an intrinsic property of websites, Ajax, ReactJS, Ember or any other framework. Most likely, the fault is in our code.

We will look at a few common errors and find a solution that leverages the tools provided by Selenium.

The Most Common Errors with Selenium

Although it’s hard to admit, we weren’t born knowing. When I started out, in the world of automation it was common to see a code written like this:

There’s a serious problem with this code I wasn’t initially aware of: we have to wait for a certain amount of time to pass.

In this case, it doesn’t matter whether the amount of time is hardcoded, referenced from a file or exported into a variable for others to easily use. Pausing the code is something to avoid.

In a context where we’re running several tests (permutations), that can cause serious delays in the testing. There are more effective and suitable strategies for cases like these, like waiting for events and changes in the webpage to happen instead of waiting a certain amount of time.

We will now analyze such strategies and recommend a few coding practices, as well as advise against others.

Why It Isn’t Advisable to Insert Pauses

Let’s start by analyzing the example code. There are three reasons why inserting a pause isn’t recommended:

  • Considering that a webpage can take less loading time to show an element, pausing the code has negative effects because it increases the execution time of the set of tests.
  • Considering that sometimes the webpage can take longer to load, adding a pause reduces stability.
  • If we insist on pausing and start allowing exceptions, then we must try to avoid the first two issues. That means that the code will have pauses of different duration, significantly reducing maintainability and eventually stability, in spite of our effort to avoid that.

To put things in perspective, I looked for all the references made to methods allocated to waiting for elements or events in a framework webpage that I wrote for a client, with only 10 automated test cases taking only two and a half minutes to run this small set of tests (including the time spent setting up and downloading dependencies).

As a result, for those 10 test cases, in different scenarios we waited 40 times more in total for the following events:

  • For the webpage to change
  • For the webpage to finish changing
  • For the webpage to finish loading elements through Ajax
  • For an HTML element to disappear
  • For an HTML element to appear

If we had inserted pauses and added just 5 more seconds to the normal waiting period in each element in order to keep tests stable in case of speed variations, the result would have been an execution time of 6 minutes! If that was the result of inserting short pauses, imagine if we inserted unnecessary pauses of 10 seconds or more. And just imagine if we inserted pauses in a set of 100 tests or more!

We have presented a strong case against using code pauses and we even argued against allowing exceptions in some cases. What should we do, then? Basically, the solution is a summary of the list I just described: it’s better to wait for events (changes in the webpage elements) instead of waiting for a certain amount of time to pass.

Usually, this results in another, though less serious issue: not using the tools provided by Selenium.

What Happens When We Don’t Use Selenium’s Tools?

Let’s look at a few cases:

  1. Changing the setting of the implicit waiting time for the entire project

In some cases, we often choose a solution that affects the whole development process, when maybe we just wanted to insert one long and one short pause to wait for elements. Setting a long implicit waiting time has the side effect of increasing the execution time for the entire framework, in cases where there can be real errors.

Here’s an example:

In the case of an element that just changes text in the webpage, which normally responds immediately, adding a global implicit waiting time of 30 seconds will cause the execution time to increase in several more minutes if the code is used in many test cases, when usually with a short waiting time we could have spotted the error earlier and speeded up development times.

        2. Redeploying existing functionality

In this case, we can appreciate the utility of using one of Selenium’s tools: WebDriverWait. But the programmer won’t be inventing anything new because this has already been solved, and it’s always better to use a code that’s been tried and tested by thousands of people:

Note that this not only is a guarantee that we are using a code that has been tried and tested by many developers, but it’s also shorter and easier to maintain.

          3. Redeploying existing functionality, part 2

This is quite similar to the previous case though a bit more complex. It isn’t commonly known that there’s a simple way to handle it.

They tried to manually solve the issue of an HTML element that changes while we are using it and becomes stale, which prevents us from clicking on it and produces an exception instead.

The term ‘stale’ is used to represent the web element instances we obtained which, due to changes in the webpage, have become old and obsolete.

This happens occasionally with developments built on Selenium with lazy loading, like Selene. But it can become a frequent issue when using Selenium without any layer of abstraction.

The solution is similar to that for the previous case, adding those exceptions we want to ignore:

Again, the code is easier to read and maintain, and in addition, it has the flexibility of expected conditions, thanks to which we could even customize one based on our needs. Here’s an example of how to implement our own expected condition:

How to Achieve Stability in Selenium

Selenium’s library includes all the tools we need in its code and documentation. Usually, it’s in us to successfully run our set of tests in a stable and predictable way.

I took some of these examples from the answers to this question in StarkOverflow which, in my experience, were wrong. I hope this proves useful! Feel free to contact me for suggestions or amendments!

Emilio Moretti

Emilio Moretti is a software developer at intive-FDV since 2017. Computer engineer graduated from the National University of the Northwest of the Province of Buenos Aires (UNNOBA), Emilio is currently studying the Specialization in Embedded Systems at the University of Buenos Aires (UBA). Free code enthusiast, he contributes and collaborates in the open projects that he uses for his hobbies, that goes from the development of games to 3D printers.

Add comment