Recently, as a
DevOps consultant, I helped improve the performance of running large scale browser automation tests.
The client’s current infrastructure and setup was running the tests locally on their QA machines in a single windows server 2012 installed with selenium web driver. This setup is simplistic but unable to scale to large number of tests.
One good solution is using selenium grid to form a cluster of
webdrivers to run test in parallel. Selenium grid allow multiple selenium drivers to work together.
However, the pricing is steep, for example, 2 concurrent sessions is priced at $199 monthly. In my opinion, the commerical options is great for cross platform tests while running selenium grid on cloud providers is great for large scale functionality testing.
Hub and Node
In selenium grid, there are 2 roles for the web drivers,
hub is responsible for accepting all webdriver requests from the client. The
node is a selenium-web-driver capable of running tests on various browsers. e.g. Chrome, Internet explorer, Firefox.
In the example grid above, it is setup with 1 hub and 7 nodes. Each node is capable of having 5 browsers sessions, and supports IE, Chrome and Firefox. With this setup, we can run 35 concurrent browser sessions.
Connecting to the grid
From the selenium webdriver library, we simply instantiate the
RemoteWebDriver with the
Both the hub and node can be easily configured through the cli or via json file.
There are several factors in chosing what infrastructure to run the grid on. We explored various options.
Linux/ Docker or Windows?
There are considerations to make on which OS/platform to runs on, and or containers.
Firstly, if there is a need to run internet explorer browser, the nodes need to be provision on windows because IE only runs there.
The docker route totally simplify the installation of the various drivers into a single image!
Comparing various Web Drivers
Chrome and Firefox drivers are most common here because they strike a balance between performance and has modern webkits.
PhantomJs is a good options if we want the tests to run fast because its a headless browser; it is faster because it doesn’t really render the dom.
Dynamically provisioning the grid
Since we need to have internet explorer support, we are running on windows. This means we can’t go with the
Having a 50 windows servers sitting around idle in amazon is expensive and non-cost optimized.
So as a major component of the project, we have build tools to automatically provisioned the nodes when needed and then de-provisioned the nodes when we are done running the tests. Our setup resolve around windows machines because our client need internet explorer.
The tool does the followings:
use terraform to provision windows servers.
Run the tests againsts the grid.
terraform destroy when tests is completed to de-provisioned the nodes on our cloud provider.
We then ran this tool as part of our client continous integration pipeline in jenkins. With this setup, we achieved a cost efficient solution for running large number of browser automation tests.