March 2009 Monthly Meeting Summary
Topic: Automation Estimation - Roundtable Discussion
Initial proposed subtopics were:
* Issues in estimation of test automation projects
* Automation estimation experiences
* Automation requirements planning
* Agile approaches to test automation
* Test automation metrics
* Defining and verifying test automation progress/success
Took place on: Wed. March 4 2009 6:30 PM
Meeting Notes from roundtable discussion:
Considerations in test automation project estimation -
- Maturity and stability of AUT
- Tool chosen
- Experience and knowledge of automation engineer(s)
- Maintenance of automation suite over time; eg with each AUT release or change
- Automation stakeholder expectations
- Needed refactoring of automation code over time
- Use of common modules/modular codebase
- To baseline an estimate, get some experience with a tool, doing some initial tests first; subsequent test dev typically goes much faster
- Some QA engineers more interested in or better at test automation than others; automation test engineers need to be able to take a developer perspective
- Automation test environment consideration can sometimes be critical
- Consider automation result reporting requirements
- Start with understanding what can be automated and what can't.
- Expect to have missteps/failures in initial automation efforts
- SW estimation is hard, so automation estimation is hard too especially if not as experienced at estimates as the dev teams
- Estimates based on historical data better than guesstimating
- Useful to have some automation requirements or criteria before doing estimates, and to prioritize them
- Useful to consider automation 'effectiveness', what that means, and how it can be measured or monitored
- As with most software project estimates, an accurate full automation estimate may result in decision not to pursue the automation project. Sometimes an initial pilot automation project that does not include all the long-term considerations may make initial automation spending easier to accept and allow for time to build support and understanding of the benefits of automation and support of a larger budget.
- Consider types of automated tests needed - just functional tests? Performance tests? Security tests?
- Be aware of possibility of overemphasizing value or importance of a particular tool; ignoring its limitations
- Sometimes a more effective automation approach is to start out small and grow
- Usually manual testing is needed first as a precursor to planning and developing automation
- Consider requirements for information reported by the automation tool for its usefulness in analyzing problems found.
- Consider automation of real user scenarios and real paths
- Consider automation of the more tedious manual test cases as higher priority automation candidates
- Targeting a 'coverage' level can be problematical given the wide possible varieties of the meaning of 'coverage'
- In reporting automation results, ROI, etc it may be useful to consider the variations in the meaning of the term 'test case'
- ROI determinations may give an incomplete picture of pros/cons of automation - calculation based solely on cost of manual vs automated may or may not be appropriate
- 'Speed' of testing may be improved by automation once tests are developed, but the time needed to develop automated tests can slow things down; however once developed they could make it easier to meet schedules.
- May need to manage expectations of automation stakeholders.
- Automation does not eliminate need for manual testing but may be able to replace some
- Good automation may require close interaction/cooperation between dev and QA
- Can find many bugs during automation development and also learn a lot about AUT
- Pitching the 'speed' of testing via automation may confuse management about appropriate goals of automation
- A disadvantage of automation is that while people running manual tests can be tolerant/adaptable to variations, automated tests usually are not. Also people can notice anomalies during manual testing that an automated test might not detect
- There may be significant productivity value from automation due to the more immediate feedback to developers
- Consider how much the automated tests will be reused, and how much a common codebase will be reused.
- In agile projects, may not see automation value until 2 iteration cycles past the dev cycle - need to do manual testing initially
NoVaTAIG Home Page
Copyright 2008-2009 Northern Virginia Test Automation Interest Group
Test Automation Interest Group