Talk Accelerate 2018: Test Impact Analysis: How to Find New Bugs 100x Faster
Coordinates
- Speaker: Elmar Jürgens
- Conference: Accelerate
- Date: Tuesday, October 9th 2018, 15:45 — 16:10
- Venue: Austria Center Vienna, Keynote Room F
Abstract
Large automated test suites are often too slow for CI. Since their execution is deferred, many defects tend to slip through CI — increasing the time, cost and difficulty required to find and fix them. Test Impact Analysis selects only the tests impacted by code changes made since the last test execution. It then sorts these tests according to their bug discovery potential. This way, it can discover new bugs extremely efficiently. In fact, our tests could identify 80% of failing builds in 1% of the test execution time.
This allows for fast CI builds and high bug-detection rates, even for large test suites. This session explains the concepts behind Test Impact Analysis, presents research findings from large case studies and shows how it can be integrated with Tricentis Tosca.
Slides
Recording
This talk was recorded and can be watched in this video. See below for times of specific talk contents.
Content
- 01:32: Introduction of the presenter Dr. Elmar Jürgens and his backgrounds
- 02:43: Long lasting test suites are a problem
Part I: Introduction of Test-Impact-Analysis (TIA)
- 03:52: What can we do cleverer in executing tests?
- 08:19: TIA uses two techniques: Selection and Prioritization
- 09:40: Using code coverage of test runs to influence the set of executed tests and its execution order in subsequent runs
- 11:06: Limitations of TIA
Part II: How good is TIA in practice? An Empirical Evaluation using a Mutation Testing Based Benchmark
- 13:18: How big is the Speed-Up of TIA Test Executions?
- 14:59: Role of tests’ locality (unit vs. system tests)
- 15:37: How long does it take to the first test failure?
»Test Intelligence«
- 17:01: Brief summary of TIA steps
- 17:23: Which analyzes build up »Test Intelligence«?
- 19:14: Roundup TIA
- 20:15: Questions
References
