Facts About Automated Testing

Software Testing performed using a tool or program and executed with minimal human interaction is Automated Testing. Automated Testing is often mistaken as something that is automatic and just magically works out-of-the-box with the purchase of a testing tool. In reality, a QA Engineer teaches the tool to test by specifying (recording) the steps and defining the Test Pass/Fail criteria.

This article covers only Automated Functional Testing not automated unit testing.

Why do Automated Testing?

In the last few years software applications have grown in complexity. Applications are designed to run on multiple machines, work with feature rich graphical components and boast many business features. All these make testing an application manually a nightmare. To ensure quality software, testers have to test more code in less time and do this more often. Repeat the same task over and over again? Humans can't do that. Yes and that is why we have Automated Testing. Automated Testing shortens the test cycle time and reduces cost (in the long term by eliminating the manpower to execute the tests). Also, automated tests are the only way you can discover and consistently reproduce problems due to heavy load, concurrency, dead locks, etc and measure application performance.

How do we run Automated Tests?

Automated Tests have to be well planned, not all tests can be automated and ideally you run these tests in an isolated environment to avoid influencing the results of the tests. So we start with detail test cases and develop a set of regression tests. These tests are run every time there is a change to the application. We could have different categories of tests and run these for varying degrees of application change. A good start would be to automate a set of Smoke Tests and keep adding tests to create an exhaustive suite of tests.

The real big challenge is step one, picking the right tool for test automation. My two cents on this is that a single tool will not be able to test everything your application does. Any application today writes to the database -- so you need a tool that tests this capability by verifying the database content, it has a piece to send and receive messages (e-mails, instant messages, file transfers) -- so you need a way to check messages and it has a web component with a rich user interface -- so you will have to test the application behavior by simulating user interaction. So a single tool is not going to work for all these areas. Pick a tool that does one task well, for example: to test a web application's browser client, pick a tool that easily records user interaction and plays them back to test the application. However, it would be good if all your test tools can be driven and managed using a common interface.

Often features (more true about a Graphical User Interface) change or evolve a lot before major releases. This also results in broken test cases. Hence it is a good idea to start implementing these automated tests once the features stabilize and the tests can be used for regression testing. A test that is run only once, for example -- testing the correctness of application configuration parameters, does not need to be automated. The cost of automating such tests is much higher than the gains of running this automated test. Also, it is very unlikely that such tests will find defects. So automate tests for features that go through constant change, are complicated and require a lot of manual effort/time to test. Write your tests to a framework that lets you easily add test case variations by modifying a test script file or by editing some configuration values. This way writing the first test might be time consuming, but you can add many tests quickly.

The initial cost of creating Automated Tests may be high, but you definitely win in the long run.