Pareto Testing

Performing a full regression test often takes too long to receive early feedback.
Pareto testing helps you to find new bugs as quickly as possible by relying on an optimized test selection.

Motivation

Full regression testing is expensive

A full regresstion test can take up to hundreds of hours execution time in complex set ups. Often it's not possible to significantly speed it up because of hardware in the loop or surrounding systems. A single bug which likely causes multiple tests to fail simultanously can thus become quite costly. Detecting those bugs before performing a full regression test would save a lot of time.

Test selection based on coverage

Pareto optimization

A single test case often covers only a small part of the system. Similar test cases overlap and, thus, cover some parts of the system redundantly. Pareto optimization selects those test cases that contribute the most additional coverage with the least execution time. This increases the probability to detect new bugs in the system very early on, before running the full regression test.

Detecting bugs early

Cost reduction

Executing a pareto optimized test suite achieves a significant portion of the complete system coverage with only a fraction of the execution time of the full regression test. In scientific studies we showed based on mutation testing that a pareto optimized test suite finds 95% of the mutants already within 15% of the full regression test execution time. After a successful, fast pareto test you can, thus, more confidently launch your full and expensive regression test.

Technical Integration

Usage during development

To apply pareto optimization on your test suite, test case specific coverage data is required. Our tool Teamscale then calculates which subset of tests  achieves the highest coverage in minimal time and suggests you the Pareto optimzed test suite. You can execute this test suite frequently during development and use it to safe guard your complete and expensive regression test.

Visit www.teamscale.com for a detailed overview of Teamscale. 

 

Dr. Elmar Jürgens
Experience exchange

Would you like to share experiences on Pareto Testing?

Any complex analysis raises questions. Is it applicable to you at all? What experiences have other companies in your industry had with Pareto Testing? Are the technologies you use supported?

I'm happy to chat with you about Pareto Testing.  Please contact me, I am looking forward to our exchange :-)