Help -- My System Is Being Audited!
Posted on 12/21/2016 by Dr. Daniela Steidl
It’s not that normal pressure wouldn’t be enough. Your next release is already delayed. There are numerous bugs that still need to be fixed. Your customers are getting impatient. And here comes your manager, telling you that CQSE will perform a code and architecture audit on your system. You might very well think »What the hell…«. Yes, we know. And believe us, you are not alone. No development team on earth has free time to spend, in particular not on an audit that seems more scary than useful. But the audit sounds worse than it actually is. That’s why this blog post is supposed to take away some of your fears and clarify what is, in fact, ahead of you (and what is not). And believe it or not, you and your team can actually benefit from the audit, too.
What Do We Do?
Let’s start with what we actually do during an audit. A code and architecture audit targets one main question: How future-proof is your system? Which means: How well is your system prepared for changes that might occur in the future? To get an understanding of your system and to answer these questions, we perform the following activities:
Documentation Review. We review whatever type of documentation you can provide us – from user manuals, over architecture specifications to coding guidelines.
Code Analysis. With static analyses, we analyze the quality of your code. We use both automated analyses based on our tool »Teamscale« as well as manual analyses in forms of code reviews.
Architecture Analysis. We need a basic understanding of your system’s architecture – how it interacts with other systems, how different components relate to each other, which technologies are used, how it is deployed.
Process Analysis. Briefly, we want to understand how your development process works, from the way you derive requirements for your system over the actual development planning to testing, releasing and deploying the system.
Scenario Analysis. Taking into account the information from the previous three analyses, we identify – together with you – certain change scenarios which might become relevant for your system in the future. We analyze them with you to see how well the system would currently support them.
Interviews. We also interview certain stake holders to better understand different individual views on the system. Usually, we interview, for example, the system’s product owner, its architect, one or two developers and a tester. Of course, information from these interviews is treated anonymously.
Afterwards, we sort all collected information and conduct a SWOT (Strengths-Weaknesses-Opportunities-Threats) analysis. Usually, we do this assessment internally, without your involvement. Finally, the audit results in a list of identified strengths and challenges of the system. These challenges reveal when counter measures are necessary and provide you the opportunity to take them early on. Thus, the challenges help your team and your management to create a clear roadmap for the future.
How Much Time Does It Take?
While this sounds all nice in theory, the crucial question is, of course, how much time it takes. The following list should give you a rough overview of the required meetings and their stakeholders involved:
Kick-Off. Ideally, we have a one-hour kick-off with everyone who we are in touch with during our audit. It’s good to know each other and to clarify how we work. It is important that you know we do not operate in a quality police style manner.
Code Analysis. For this, we need your code – either on our machine, on your machine or with remote access. Yes, we know, your code is your most precious crown jewel. So we have appropriate mechanisms in place to treat it securely. To get started, we need only one person at the beginning. This person, for example, the lead developer, should have a decent overview of the entire code base but does not need to know every detail. For the rest, we will do the analysis on our own. But we might come back to you occasionally for some clarification questions.
To start the analysis, there is one main question: Which code do you maintain manually? While this might sound like a trivial question, it is for many teams, in fact, very hard to answer. We want to know which code is application code and which code is test code as we are analyzing both (however, usually separately). To obtain clean and meaningful results, we exclude generated code, 3rd-party code and old or experimental code. Hence, in a well organized code base, this code discrimination can be done quickly. However, if test, application, generated and 3rd-party code is all mixed together and no developer really knows which is which, then we have to figure it out on our own. This is time consuming. Likely, we will come back to you frequently to ask questions. Also, each time we encounter new code that should have been excluded from the analysis, we need to reconfigure and rerun our setup, which takes quiet some time and might, in the worst case, even delay the audit.
Architecture Analysis. Usually, this works best in form of an architecture workshop involving the product owner, the architect and a lead developer. To prepare for it, we are happy to pre-study any existing documentation and to start from there. Otherwise, we will just start together with a whiteboard and some markers and work our way through it in an open QA-style. Depending on the simplicity and clarity of your architecture, this can either be done in a couple hours or need a bit longer.
Process Analysis. In this session, we would like you to present us your development process. You can do this informally or with a set of slides or maybe in parts with a live demonstration. The session will also be mainly QA-based. To give examples, we would like to see your sprint planning if you are doing SCRUM, your test setup and also your issue tracking system. Depending on the extend of your audit request, this can be done in a couple of hours (if we conduct only a light-weight analysis) or need some more time (if we should dig into detail).
Scenario Analysis. Depending on the scenario, we pick a suitable stake holder for each. During the analysis, we identify together with him the steps which would be required to perform the scenario. We might also have a look in the code base together to locate where specific changes would be necessary. Analyzing one scenario usually takes between 30 and 60 minutes. The number of scenarios depends on the extend of your audit request. For small audits, we might analyze only two or three. For large audits, we might analyze up to five or six.
Interviews. We usually conduct around 5-7 interviews, depending on the size of your team and the number of different stakeholders involved. One interview takes roughly 30 minutes.
This is it! So the amount of time you need to investigate is limited. We might come back to you every now and then to ask few clarification questions, but we really do not prevent your development team from working. Often, we get the feedback that the analysis workshops are not only helpful for us, but for you as a team, too. When we learn how things work in your system, you might learn, too!
How Are the Results Presented?
After we finished our analysis, we usually create a presentation and a written report. We always give the presentation to you as a team first before we talk to management. This is your chance to validate the results and also an apportunity for you to point out if we misunderstood something. Afterwards, we will give the presentation also to management and deliver the final report.
Is There Anything You Should Do Beforehand?
Yes, first of all: Don’t panic. Within the short time prior to the audit start, you will not change your system fundamentally anyway. So stay calm.
But you can prepare, for example for the code analysis. Please consult with your developers which code is manually maintained. Let them clarify where test code, generated code and 3rd-party code is located. Very likely, the first claim will be that no code is generated. But very often, this turns out to not be true. So rather double check and skim through your entire code base once.
You can also prepare for the workshops. Questions we will most certainly ask are:
- Who are the relevant stakeholders?
- What are the main use cases?
- What does the business case look like?
- What is the unique selling point?
- What is the pricing model?
- What are relevant quality criteria (security, availability, scalability etc.)?
Also in terms of architecture documentation, you can provide different information ahead of time :
- Context Diagram: How does the system interact with stakeholders or other systems?
- Component Diagram: How is the system designed on component level and how do these components interact with each other?
- Deployment Diagram: On which hardware is the system deployed?
- Technology Stack: Which languages, libraries and frameworks are used?
If these questions are well prepared, our audit will be a lot easier and also shorter in time. Which gives your team more time to focus on your actual tasks!
Our audit is no monster that will take away all time from your developers. We do need, of course, to interact with you to some degree. But we will not be disturbing you for days. Usually, we do the code analysis from our office. For the workshops, we come visit you in your office for three or four days in a row. But, as described above, we do not need your entire team, but only few representatives at a time. So no reason to panic!
- S. Zörner: Softwarearchitekturen dokumentieren und kommunizieren, Hanser, München, 2012.