In a recent software quality audit, our static analysis tool Teamscale found that the comment completeness ratio of the application under study was 96%. But when examining these comments manually, we found that the majority of them was generated automatically and therefore of limited use.

This example illustrates that a combination of software tool and human expertise is necessary to get a holistic picture of and ensure software quality. To argue for this position in more detail, this blog post sketches software quality tasks which should be performed by software tools, software quality tasks which should be performed by human experts and software quality tasks which should be performed jointly.

Tool Results vs Reality

Software quality is an important aspect of a software application, especially for long-living software which is used and evolved over decades. To ensure software quality, much effort is put into quality activities like testing, bug fixing, code reviews, (static) quality analysis, and refactoring. Furthermore, many tools have been developed in the last years to automatically assess software quality and support developers to improve the quality of their application.

But often a software tool is introduced and induces a short-term improvement of software quality but has negligible positive effect on software quality in the long run. We argue that to achieve such a long-term positive effect, a combination of software tool and human expertise is necessary.

The example of a recent software quality audit demonstrates the interplay of software tool and human expert: When analyzing the Java source code of the application under study, our static analysis tool Teamscale found that the comment completeness ratio was 96%, i.e., that 96% of all public types, attributes and methods were commented. This appears to be a very good value and would rank the application very high in our benchmark of industry software.

But when we manually analyzed a subset of these comments, we found that many of them were trivial comments that were generated from identifiers in code in order to adhere to the company policy »More than 95% of types, attributes and methods have to be commented«. The following code listing shows an example of such a trivial comment:

When further investigating the matter, we found that about three-fourths of all existing comments were generated and therefore useless, bringing down the comment completeness ratio from 96% to 24%.

Warning This blog post describes my experiences from 0.5 years of work as a software quality consultant (which was performed together with colleagues working in this field for more than ten years and borrowing from their experience). Hence, the blog post is highly subjective and incomplete.

Tasks of Software Tools in Software Quality Activities

The following non-exhaustive list sketches tasks of software tools in software quality activities:

• Aggregate information Pull together information from different sources automatically and continuously. For example, extract information about code changes from a git repository and information about test coverage from the test system and combine both.
• Analyze software quality Perform quality analysis that can be automated such as calculation of code structure metrics, clone detection, or test gap analysis (identification of changed methods that have not been tested since the change). This is especially true for large code bases as well as code bases with frequent changes which are impossible to analyze manually.
• Present analysis results Present results of (automated) quality analysis to developers for inspection and interpretation by using visualization, statistics, or list views.
• Monitor software quality Monitor the development of quality over time and notify the developers when a pre-defined threshold value is reached or a critical finding is detected.
• Perform tests Execute automated test cases to check whether the application behaves as expected again and again, e.g., after every build.

Tasks of Human Experts in Software Quality Activities

The following non-exhaustive list sketches tasks of human experts in software quality activities:

• Advocate software quality Create awareness for software quality and its importance among managers, developers and testers and educate them how to improve software quality.
• Promote software tool(s) Demonstrate managers, developers and testers the benefits of software tools and educate them how to use the tools in their daily work and how to integrate them in their work flow.
• Define relevant quality aspects Define (in cooperation with the development team) the aspects of software quality which are important for the current application and how these can be measured. For example, define the coding guideline to be used.
• Define scope of analysis Determine (in cooperation with the development team) which code NOT to analyze. For example, identify code that is generated and not manually maintained. This information has to be taken into account when configuring software tools.
• Define quality goals Coach the development team to define quality goals they wants to achieve, e.g., to introduce no new findings or to remove findings from changed code.
• Configure software tool(s) Usually software tools come with many configuration options. Hence, human experts have to configure them, e.g., define the threshold value from which a method is considered too long or the set of systems from which to consider coverage information. This configuration task is especially important if several tools are used together and have to exchange data.
• Customize software tool(s) If the software tool(s) do not cover all relevant quality aspects, human experts have to extend them by implementing automated checks for relevant quality aspects not yet covered.
• Administrate software tool(s) Administrate software tools, e.g., create user accounts for new developers or install new releases of the tool and migrate the data to it.
• Validate tool correctness Manually check the results of automated analysis to make sure that they are correct (not because the analysis algorithm is wrong but because the results might not hold in the current situation - see the introductory example).
• Ensure correct tool operation Monitor the software tool(s) and ensure that they continue to work properly and operate on the correct data. Repair the tool configuration in case of changes, e.g., when the code repository is migrated to another location or test coverage from an additional system has to be considered. This is especially true in the context of a large enterprise where the IT infrastructure is administrated by a central IT operation unit which might change the configuration without notice.
• Prioritize findings Usually (static) analysis tools create a large amount of findings that are impossible to address one by one. Hence, human experts have to prioritize the findings, create specific tasks from findings and assign these tasks to developers.
• Analyze software quality Perform analysis of software quality that cannot be automated such as the quality of code comments or the quality of identifiers used. Usually this is done in code reviews.
• Perform tests Manually test aspects of the application that cannot be tested automatically such as acceptance testing.

Conclusion

There are several tasks that can be performed only by software tools and not by human experts (for example ‘Aggregate information’ or ‘Analyze software quality’ for large code bases or frequent changes). Similarly, there are several tasks that can only be performed by human experts and not by software tools (such as ‘Define relevant quality aspects/ scope of analysis/ quality goals’ or ‘Configure/ customize/ administrate software tools’).

Furthermore, there are several tasks that have to be completed jointly by software tools and human experts because each contributes a subpart of the overall task (for example ‘Analyze software quality’ or ‘Perform tests’). Hence, we conclude that a combination of software tools and human expertise should be used in software quality activities (‘man and tool’ instead of ‘man vs. tool’). Only the combination of both gives a holistic picture of software quality and only human commitment ensures software quality and its improvement.