I got some comments on my post “Test Everything all the Time” — most notably people commenting that it’s impossible to test “everything”. I can’t agree more. The intention of the post was to make the point that we need to be able to test “everything we can” all the time. That is, you should be constantly iterating your automation, and not “waiting to run” the automation. Also, the point was to talk about how test design can solve all of your problems, not the automation tool. The tool is just a means to an end.
Automation allows us the ability to test wide swaths of functionality that we would not be able to do manually. It’s virtually impossible for a manual test team to adhere to the adage “Test Everything all the time”, they can’t do it.
Separately, a trend is occurring in our industry where testing is moving toward more and more code inspection and code execution strategies. I notice many more testers trying to determine quality by working with the code, rather than the finished product. I think this is a sound approach, in fact, the sooner you can test the better, however for many in our industry code inspection is a skill that many of us don’t have. However, many of us do have test design creativity, and it’s here that when married to a good automation tool, can produce very reliable and consistent results.
As I noted in my blog post, you need to strive to have all of your tests done automatically, and that manual intervention is used only when necessary.
i agree 300% with Bill on automation developed should be exersised constantly on everyday basis, so that it is ready for real battle with new build/release just in time. I would like to stress the special importance of this notion with UI test automation where even ‘under the hood’ change in the application under test (for example web grid control properties change) would cause UI automation choke. In order UI automation would produce predictable an reliable reults it’s got to be run on every single minor release before verifying major release of the product. Working in Agile environment automatically puts automation under test every day or so. Dmitry (#1 automation expert:)