In many domains software is a highly critical part of the products, especially in automotive and avionics systems, in medical equipment, and in military systems—software failures may cause severe damage to machines or even cost human lives. Thus, it is broadly accepted that software quality is the key to developing safe systems. Most people out there have deep trust in the quality of the software controlling the cars they drive, the planes they fly with, and the equipment that is keeping them alive during medical surgery. To be honest, most of this software works really well. However, when looking behind the curtains, I sometimes wonder why.

Safety critical systems need to be certified and thus undergo rigorous qualification processes. There are different kinds of standards, defining quality and safety regulations for different kinds of applications. Having a closer look reveals that most of these standards state that certain kinds of activities must be performed—usually resulting in documents as main deliverables. Hence, some regulations also enforce certain kinds of quality assurance activities, e.g., reviews must be performed or a certain degree of test-quality (coverage) must be achieved. In principle, this is of course a good idea. When looking into their practical application, however, it sometimes becomes pretty weird:

15 developers did a formal (Fagan-like) review of a design specification, sitting together in a meeting room, collecting defects.

The only outcome of this meeting were issues of the kind »a page number is missing on the fourth page«, »the date on the title page is wrong«, and »there is a typo on page X«. After the meeting all participants were happy that no severe problems had been identified and that they successfully passed the review. What went wrong?

1. The specification had 800 pages—nobody ever read it completely. The reviewers only skimmed it through during a 1.5h meeting.
2. Nobody was really interested in identifying real issues because this would cause re-work leading to re-review leading to a massive delay of the project coming along with trouble with customers and management.

Thus, although the formal process requirement that the specification must pass a formal review had been fulfilled, no effect on the quality of this artifact had been achieved.

A system successfully passes a test-suite that measures 100% statement coverage.

However, the system still contained several severe bugs leading to data inconsistencies. What went wrong?

1. The tests had been tuned solely in order to satisfy the 100% coverage requirement.
2. The tests hardly contained any assertions, so the tests did not test anything.
3. The tests did not execute any real use case and were executed solely on a fine-grained level stimulating even the last fail-over code.

This is another example of a fulfilled formal qualification requirement that hardly had an effect on the quality of the product. Admittedly, these are only anecdotes and cannot be generalized. However, we often encounter people having a religion-like believe in standards, regulations and formalisms. In some projects the fulfillment of each and every bureaucratic paragraph becomes dominating, hindering developers to spend efforts on activities that really affect quality and correctness. So for all developers and auditors out there—please do not look only at the pure existence of process documents. Look at what really counts: the system, the tests, and the people.