Survey Results – Automated Testing trends from LogiGear’s 2010 Global Testing Survey

LogiGear Senior Vice President Michael Hackett analyzes the Automated Testing portion of the 2010 Global Testing Survey.

Data was compiled and analyzed by Michael Hackett, LogiGear Senior Vice President. This is the first analysis of the 2010 Global Testing Survey. More survey results will be included in subsequent magazine issues.

The target audience of the survey were black box testers. Please note that to these respondents, test automation is mainly about UI level automation, not unit, performance or load testing.

These automation survey results contain two mutually exclusive sections. There were sets of questions for teams that currently automate tests and another set for teams that currently do not automate any tests.

I. Overview

Before delving into the respondent’s frame of mind with answers to questions from Test Automation, I will highlight some results from The Politics of Testing, Training, Strategy and Overview sections that will set the stage for a better understanding of the issues faced by these respondents in test automation.

PT1 (Politics of Testing)- What phase or aspect of testing do you feel the management at your company does not understand? (You can select multiple answers.)

Projects are most often behind schedule because of shifting requirements not test delays. 43%
How to measure testing 41%
How to adequately schedule testing 41%
Test automation is not easy 40%
Testing requires skill 35%
The inability of complete testing 32%
Choosing good coverage metrics 23%
None, my management team fully understands testing. 17%

Result analysis: The 3rd highest response, virtually tied as the area of testing that management does not understand test automation is not easy!

PT2- What challenges does your product team have regarding quality? (You can select multiple answers.)

Insufficient schedule time 63%
Lack of upstream quality assurance (requirements analysis and review, code inspection and review, unit testing, code-level coverage analysis) 47%
Feature creep 39%
Lack of effective or successful automation 36%
Poor project planning and management 33%
Project politics 31%
Poor project communication 27%
Inadequate test support tools 25%
No effective development process (SDLC) with enforced milestones (entrance/exit criteria) 25%
Missing team skills 23%
Low morale 16%
Poor development platform 8%

Result analysis: By far, the #1 answer here is insufficient schedule time. The #4 answer is a lack of successful automation. It is too easy to say more investment in test automation will solve all your team’s problems—but it will definitely help! More and more effective test automation always helps projects.
It is not the answer to all problems, as clearly emphasized in the second highest choice, that a lack of upstream quality practices cannot be solved by downstream test automation! But better test automation will go far in helping the manual test effort and by doing so, at least relieve some tester stress.

T1 (Training)- What skills are missing in the group? (You may select multiple answers.)

Automation skill 52%
Coding/tool building 42%
Technical understanding of the code, system, platform, environment 35%
Subject matter/domain knowledge 33%
QA/testing skill 32%
Test tool (ALM, test case manager, bug tracking, source control, continuous integration tool) 22%

Results analysis: Interesting but never surprising, the highest chosen answer by teams in regards to what they lack─more than half the respondents─is test automation skills! It is obvious and clear that acquiring more test automation skills is the single most important job security point.

S1 (Strategy)- How do you do regression testing?

Both 47%
Manual 34%
Automated 15%
We do not do regression testing 4%

Results analysis: A very big surprise to me─the lack of automated regression! Wow. That is one of the biggest and most surprising results of the entire survey! Why do 1/3 of teams still do all manual regression? Bad idea, bad business objective.

O1 (Overview)- Do you currently automate testing?

Yes 63%
No, not currently 37%
If no, has your team ever automated testing? Yes 90%
No 10%

Results analysis: With over 1/3 of respondents currently not automating tests, these results, however, are contrary to popular belief and any sort of best practice. What I see out in the business world are many teams that think everyone automates and they themselves automate enough. I also see many teams where all testing is manual and see automation as not common, too difficult, and not something testers do. This number is alarmingly high. Any team not automating has to seriously look at the service they are providing their organization as well as the management support they are receiving from that organization!

Read more in LogiGear Magazine

Michael Hackett
"Michael is also a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics.

Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), available in English, Chinese and Japanese, and Global Software Test Automation (HappyAbout Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries."