CIA-Style Software Intelligence
Dr. Tobias Roehm
Software intelligence has been defined as »offering software practitioners information to support their decision-making«. When researching software intelligence, I came across the CIA approach for gathering foreign intelligence and found that it has many commonalities with my understanding of software intelligence. Hence, I present CIA’s approach and what we can learn from it for gathering software intelligence in this blog post. Furthermore, I present a practice example of software intelligence.
How CIA Gathers Intelligence
The Central Intelligence Agency (CIA) is a (in)famous organization which is featured in news, books, and movies. While specific goals and actions are generally unknown to the public, the CIA describes its abstract mission and approach clearly on its homepage:
»CIA’s primary mission is to collect, analyze, evaluate, and disseminate foreign intelligence to assist the President and senior US government policymakers in making decisions relating to national security.«
CIA Approach (Slightly shortened)
»First, we have to identify a problem of national security concern to the US government. Then, we look for a way to collect information about the problem. There are several ways to collect information: Translating foreign newspaper articles, analyzing satellite pictures, decrypting coded messages, or recruiting foreigners. After the information is collected, intelligence analysts pull together the relevant information from all available sources and assess it. The result of this analytic effort is provided to senior US policymakers in the form of finished intelligence products like written reports and oral briefings. One of these reports is the President’s Daily Brief which the US president receives each day.«
»It is important to know that CIA analysts only report the information and do not make policy recommendations - making policy is left to policymakers which use the information that CIA provides to formulate US policy toward other countries.«
Gathering Software Intelligence
So what can we learn from the way CIA gathers foreign intelligence for gathering software intelligence? I see the following points:
- Overall Goal The overall goal of CIA is »to assist the President and senior US government policymakers in making decisions relating to national security«. Similarly, the overall goal of software intelligence is to assist business decision makers and software practitioners in making decisions relating to software.
- Stakeholders The stakeholders of CIA are »the President and senior US government policymakers«. Analoguously, the stakeholders of software intelligence are business decision makers as well as software practitioners like software managers, software developers, software architects and software testers.
- Problem Focus CIA intelligence gathering is problem focused as the first step of their approach is »to identify a problem of national security concern«. Likewise, software intelligence should be problem focused and its first step the identifcation of a problem which is of concern to its stakeholders.
- Key Activities Key activities of CIA are »to collect, analyze, evaluate, and disseminate intelligence«. This nicely maps to key activities of software intelligence as it also collects information, correlates information from different sources, analyzes and evaluates collected and correlated information, and disseminates its results to its stakeholders.
- Information Collection CIA uses »several ways to collect information: Translating foreign newspaper articles, analyzing satellite pictures, decrypting coded messages, or recruiting foreigners«. Equally, software intelligence uses several methods to gather information: Analyzing code, analyzing code history in version control systems, analyzing issue repositories, searching the Web, interviewing stakeholders, analyzing release notes, or analyzing documentation, among others.
- Role of Analysts An important part in CIA’s approach play human analysts who »pull together the relevant information from all available sources and assess it«. This denotes three main activities: identifying which information is relevant, correlating information from different sources, and assesing the value of information. Alike, human »analysts« play an important role in software intelligence with a similiar responsibility: to identify relevant information, to correlate information from different sources, and to assess gathered information. »Relevant« denotes whether the information helps to address the stakeholder problem under investigation. »Assessing information« means to interpret information and contextualize it.
- Presentation of Results CIA presents its results in different forms (»written reports or oral briefings«) and with different frequency (e.g. daily). Correspondingly, results of software intelligence can and should be presented in different forms (audit reports, findings integrated in IDE, dashboard of software intelligence tool, Microsoft Office, …) and with different frequency (once, continuously, daily, …), depending on stakeholder needs.
- Role of Intelligence Agency vs. Role of Stakeholders The focus of CIA is on intelligence and it leaves making policy to its stakeholders. This setup might or might not be the case for software intelligence: Depending on its mandate, software intelligence might focus solely on gathering intelligence or additionally make recommendations to its stakeholders, e.g. to use a standard API instead of self-written code or to improve software quality by removing code clones.
Software Intelligence Example
To make the abstract description of gathering software intelligence more tangible, I describe a practice example of our work at CQSE in the following.
Assume the following situation: a team of software developers is working on a software application. In parallel, a team of software testers tests the changes implemented by the developers. The time to release a new version of the software application according to the release schedule is approaching. To decide whether the release can be performed as planned, the team has to decide whether the software application is ready to release. Furthermore, assume that all changes since the last release have to be tested before the release and that the team agreed that a change is successully tested when all changed methods were executed by at least one successful test case.
The analysis for this situation is known as »test gap analysis« and I review it from a software intelligence perspective now:
- Stakeholders The stakeholders are the release engineer(s), the team of software testers as well as the software manager who is in charge of the software application.
- Problem Focus The problem focus is the release decision, i.e. whether to release the software application or not.
- Key Activities Key activities are to collect required information, to correlate information from different sources, to interpret collected and correlated information, and finally to make the release decision.
- Information Collection The following information is necessary: which changes have been implemented since the last release (i.e. which methods have been changed) and which tests have been executed since the last release (i.e. which methods were executed by tests and whether test runs were successful). The changes since the last release are extracted automatically from commit logs of the version control system and the test information is automatically extracted from test logs. Now, both types of information are automatically correlated. More specifically, for each changed methods it is checked whether a successful test run exists that executes this method. Furthermore, the time is considered: the test run must be performed after the change.
- Presentation of Results The results are presented both as a list of changed but untested methods as well as test gap treemap. The test gap treemap provides a visual overview of the code basis and color-codes each changed but untested method. It allows an interactive inspection of the results and an easy navigation to corresponding code.
- Role of Analysts To make the release decision, stakeholders have to inspect and assess the results. If there are no changed but untested methods, the release decision can be made easily. If there are changed but untested methods, they have to assess the severity of these methods, i.e. if the software application can be released without testing them. If this is not the case, testers have to test the critical changes before the release can be made. In this assessment, technical considerations like the criticality of untested changes as well as business considerations like promises to customers have to be taken into account.
Please do not hesitate to contact me at CQSE or leave a comment if you have comments, questions, or suggestions about software intelligence.
References and Further Reading
- Hassan, Xie: Software Intelligence: The Future of Mining Software Engineering Data, FoSER 2010
- CIA Homepage - What We Do
- CQSE Homepage - Test Gap Analysis
- Blog post by Fabian Streitel: Take control of your testing process!
- Blog post by Andreas Göb: Testing Changes in SAP BW Applications
- Eder et al.: Did We Test Our Changes? Assessing Alignment between Tests and Development in Practice, AST’13
- Jürgens et al.: Regression Test Selection of Manual System Tests in Practice, CSMR’11