For some clients where we perform security testing, the client requests that we report on all findings on a daily basis.

Now, I am 100% behind reporting progress in terms of what has been tested (assuming there are multiple elements) or more importantly reporting problems in progressing as soon as possible. However, there are still some clients where they expect this plus findings to be reported.

I wanted to jot down some thoughts on some pros and cons to this approach.

Advantages

A1: Feeling of progress

The client feels like we are working, progressing and finding stuff. (Although status reporting without findings should also mostly accomplish this).

A2: Immediate feedback and fix

The client receives immediate feedback on findings and can start to look at how to fix them even before we finish testing.

They may even be able to fix the finding and allow us to retest before the end of testing. I am always a little wary of the client making changes to an application in the middle of testing but if they are going to fix something but break something else that is going to happen regardless of if it happens during the test or after the test.

A3: Enforces reporting as you go

There is a tendency for consultants to save all the reporting for the end of the project. Hopefully they took enough screenshots along the way but even still, suddenly you are at the end of the project and you have 20 findings to write up. Having a daily report ensures that findings are written up as they are found, whilst they are still fresh in mind.

Disadvantages

D1: Time consuming

Whilst we would have to write up all findings anyway, it is still more time consuming to have to prepare a report daily. The report has to go through a QA process every day instead of just once and if it is necessary to combine reports from multiple people, it can get even more complicated. Especially if we are using a complex reporting template.

D2: Difficult to update already reported findings

Sometimes we will find something and only afterwards find another angle or another element to the issue which means that the finding needs to be updated. This leads to more duplicated effort with the finding being reviewed multiple times and the client having to read and understand the finding multiple times.

D3: Less time to consider findings in detail

Sometimes it takes some time to consider the real impact of a finding. For example, what is the real risk from this finding, can it only be performed by an administrator? Will it only be relevant in certain circumstances? Having to rush the finding out in a daily report loses that thinking time and can lead to an inaccurate initial risk rating.

D4: Getting the report ready in time

Every day becomes a deadline day with a race to get the report ready in time. It can disrupt the testing rhythm and mean that consultants have to break from testing to prepare the daily report therefore losing focus and momentum.

D5: Expectation of linear progress

Testing doesn’t progress in a linear fashion. A consultant might spend a lot of time trying to progress on particular test on one day or on another day find a bunch of quick, lower risk findings. A daily report creates an expectation of news every day and a feeling that no news means a lack of progress.

D6: Increase likelihood of mistakes

With the increased pressure of daily output, the likelihood of mistakes is also increased as report preparers are under pressure to deliver the daily report by the deadline and reviewers are under pressure to quickly release the report to the client.

D7: It might not even get to the client!

If there are a few people in the review process, if just one of them is delayed in looking at the report and they have a query, the report may not make it to the client in time to be relevant before the next day’s report is released anyway!

D8: One size doesn’t fit all

Once you get into the habit of expecting daily reports or you create that expectation with the client, suddenly it is expected for any project regardless of whether it makes sense. This can mean that ongoing discussion with the client is discouraged because “we’re doing a daily report anyway” or alternatively a project which requires in depth thought and research is being constantly disturbed with unhelpful daily reports.

Conclusions

I agree that is is a bad idea to do a load of testing and then only weeks later the client finally sees some output. Especially where there are particularly serious findings that immediately expose the client to serious risk.

However, the need to provide a continual stream of updates leads to time inefficiency, lower quality findings and disturbs the progression of the test.

As such, whilst the reporting format should be discussed at the start of the project with the client, the aim should be to agree on the following points by communicating the reasons discussed in this post:

1. If this is a large project where there are multiple parts which are being tested one after the other in a short time-frame then it is worth reporting on progress over these parts on a daily basis.
2. Problems with testing should always be reported as soon as possible plus a daily status update on these issues to make sure these are not forgotten.
3. Critical threats which immediately put the client at severe risk should always be reported as soon as possible.
4. If the application is currently under development or there is specific pressure to deliver key findings as fast as possible, then high risk findings or medium risk findings can be delivered during the course of the test but should not be restricted to a strictly daily frequency.

Additionally:

• If this is a short project (up to a week) without lots of different elements or if this a long project (several months) then daily status reporting is not appropriate.
• Reporting of all findings on a strictly daily basis will never be appropriate.

I was recently involved in an application security testing project for a large client covering around 20 applications with multiple consultants working simultaneously in just three weeks of testing. By discussing with the client up front and agreeing on points 1, 2 and 3 above we kept the client fully in the loop whilst not burdening ourselves with reporting every tiny detail everyday.

I will probably update this post as I think of more advantages/disadvantages but feel free to send me feedback in the comments or via Twitter.

• Uncategorized