Through my work as a test automation consultant, I hear from a lot of people, teams, and organizations about how test automation is not working. Across the industry, this is true, as stats point to about only 20% of tests being automated. This is pretty interesting (and bleak), seeing as test automation has been around for the last 30 years. So, yeah, I think it's safe to say that test automation doesn't work...the way you are doing it.
One of the biggest problems I see with test automation is how people are using it; i.e. what is being automated and tested. Often times people are using tools for things they aren't designed for, or even worse, are they are trying to automate the wrong things.
The most egregious example of this, is also the most common, using Selenium to verify the application layout and design.
Selenium is a Functional Testing Tool
Don't get me wrong, Selenium is a great tool, and I absolutely love it, but there are simple things it doesn't do well. Selenium is great at checking out functionality, and while it can check layouts, design, etc, it is painful to code it up, and it is very costly to do so.
The below examples are one-line changes in the source code. These are easy mistakes for developers to make, but hundreds to thousands of lines of code to write tests for (not to mention very complex in the logic required).
Correct Site
Alignment off - thousands of lines of code to check for in an automated fashion
Missing image - a few hundred lines of code to check for every image
Color off - a few dozen lines to check for all colors
Image too large - a few hundred lines of code to check for every image
Wrong image - a few hundred lines of code to check for every image
Wrong icons - a few hundred lines of code to check for every image
Now, I'm not saying you can't or shouldn't automate these things, just that it shouldn't be done via Selenium. These are painful things to code for, and the checks are endless.
As manual testing goes, these things wouldn't even be listed in your test case, but a manual tester would immediately spot them just by virtue of running an automated test. Remember, automation only looks at what you tell it, and telling it to look at everything can be costly.
Luckily, there is a better way!
MoSCoW Method
Instead of trying to write your automation just like your manual tests, think about what makes sense to automate. The MoSCoW method is a process to do just that. Think about what Must you automaton, what Should you automate, what Could you automate, and what Won't you automate.
It is a prioritization technique used in management, business analysis, project management, and software development to reach a common understanding with stakeholders on the importance they place on the delivery of each requirement. To be effective, go through your use cases, user stories, and risks to determine the ROI on different components/features. Then, take the largest ROI and automate from there.
What you'll find is that design and layout checks have very low ROIs, while functional checks have high ones. If you follow this process, you should end up changing your automation stance to one that is more sustainable, and effective. What this, in turn, pushes you towards, is automating functionality over other things.
Don't Stop There
Now, I'm not saying don't perform layout testing, I'm simply pushing to ensure it is separated from your functional testing. Maybe have your automation capture screenshots, and have a manual analysis of them to verify the layouts. Maybe part of your exploratory testing is this layout testing. Or maybe you do want to automate it. In that case, choose the proper tools, possibly even a different framework, to do this testing. Ensure you are looking at visual inspection tools, such as SikuliX, or use a layout analyzer such as Galen.
Just be sure to explicitly separate out your tests. Security and performance testing are called out from functional testing, why shouldn't layout? It's a different type of testing, different techniques are utilized and needed, and even associated risks are different as well.
As always, leave some comments below, and happy testing!
