Test Intelligence for and with the Robot Framework

Dr. Elmar Jürgens



Many test suites test both too much and too little. Too much, since some tests are redundant (but still expensive to execute and maintain). Too little, since important functionality remains untested. In this talk, I present test intelligence analyses that harness data from the development process to help you optimize your Robot Framework tests.

Many teams have to test more and more functionality in less and less time. Historically grown test suites often don't cope well with this, since they test both too much and too little. Too much, since they contain tests that are expensive to execute (and maintain) but add little value over similar tests in the same test suite. Too little, because important functionality remains untested.

In the talk, I will present analysis techniques that reveal such problems in your own test suites:

  • Test gap analysis shows, which code changes are still untested.
  • Test impact analysis and pareto-optimization of test suites identify tests that can find new bugs faster than other tests in the suite.

All analyses work on data from the system under test, such as change information from the version control system and test-wise coverage from the automated tests.

For each analysis, I outline the underlying research and present examples from practice. For some analyses, I use the Robot Framework itself as example, for others I use customer system or data from our own software development process.


Dr. Elmar Jürgens