LogiGear Senior Vice President Michael Hackett analyzes the Automated Testing portion of the 2010 Global Testing Survey.
Data was compiled and analyzed by Michael Hackett, LogiGear Senior Vice President. This is the first analysis of the 2010 Global Testing Survey. More survey results will be included in subsequent magazine issues.
The target audience of the survey were black box testers. Please note that to these respondents, test automation is mainly about UI level automation, not unit, performance or load testing.
These automation survey results contain two mutually exclusive sections. There were sets of questions for teams that currently automate tests and another set for teams that currently do not automate any tests.
I. Overview
Before delving into the respondent’s frame of mind with answers to questions from Test Automation, I will highlight some results from The Politics of Testing, Training, Strategy and Overview sections that will set the stage for a better understanding of the issues faced by these respondents in test automation.
PT1 (Politics of Testing)- What phase or aspect of testing do you feel the management at your company does not understand? (You can select multiple answers.)
| Projects are most often behind schedule because of shifting requirements not test delays. | 43% |
| How to measure testing | 41% |
| How to adequately schedule testing | 41% |
| Test automation is not easy | 40% |
| Testing requires skill | 35% |
| The inability of complete testing | 32% |
| Choosing good coverage metrics | 23% |
| None, my management team fully understands testing. | 17% |
Result analysis: The 3rd highest response, virtually tied as the area of testing that management does not understand test automation is not easy!
PT2- What challenges does your product team have regarding quality? (You can select multiple answers.)
| Insufficient schedule time | 63% |
| Lack of upstream quality assurance (requirements analysis and review, code inspection and review, unit testing, code-level coverage analysis) | 47% |
| Feature creep | 39% |
| Lack of effective or successful automation | 36% |
| Poor project planning and management | 33% |
| Project politics | 31% |
| Poor project communication | 27% |
| Inadequate test support tools | 25% |
| No effective development process (SDLC) with enforced milestones (entrance/exit criteria) | 25% |
| Missing team skills | 23% |
| Low morale | 16% |
| Poor development platform | 8% |
Result analysis: By far, the #1 answer here is insufficient schedule time. The #4 answer is a lack of successful automation. It is too easy to say more investment in test automation will solve all your team’s problems—but it will definitely help! More and more effective test automation always helps projects.
It is not the answer to all problems, as clearly emphasized in the second highest choice, that a lack of upstream quality practices cannot be solved by downstream test automation! But better test automation will go far in helping the manual test effort and by doing so, at least relieve some tester stress.
T1 (Training)- What skills are missing in the group? (You may select multiple answers.)
| Automation skill | 52% |
| Coding/tool building | 42% |
| Technical understanding of the code, system, platform, environment | 35% |
| Subject matter/domain knowledge | 33% |
| QA/testing skill | 32% |
| Test tool (ALM, test case manager, bug tracking, source control, continuous integration tool) | 22% |
Results analysis: Interesting but never surprising, the highest chosen answer by teams in regards to what they lack─more than half the respondents─is test automation skills! It is obvious and clear that acquiring more test automation skills is the single most important job security point.
S1 (Strategy)- How do you do regression testing?
| Both | 47% |
| Manual | 34% |
| Automated | 15% |
| We do not do regression testing | 4% |
Results analysis: A very big surprise to me─the lack of automated regression! Wow. That is one of the biggest and most surprising results of the entire survey! Why do 1/3 of teams still do all manual regression? Bad idea, bad business objective.
O1 (Overview)- Do you currently automate testing?
| Yes | 63% | |
| No, not currently | 37% | |
| If no, has your team ever automated testing? | Yes | 90% |
| No | 10% |
Results analysis: With over 1/3 of respondents currently not automating tests, these results, however, are contrary to popular belief and any sort of best practice. What I see out in the business world are many teams that think everyone automates and they themselves automate enough. I also see many teams where all testing is manual and see automation as not common, too difficult, and not something testers do. This number is alarmingly high. Any team not automating has to seriously look at the service they are providing their organization as well as the management support they are receiving from that organization!
Read more in LogiGear Magazine
Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), available in English, Chinese and Japanese, and Global Software Test Automation (HappyAbout Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries."