“Alex, the Board wants us to get an external penetration test of our application. They think it’ll help out with marketing. Get it set up.”
If you’re a CISO or a Product Manager, the above might sound familiar – today, more and more enterprises are requiring the products they purchase to demonstrate they’ve gone through application security testing, often by an independent third party. If you’re new to such a role, or to penetration testing in general, this article will explain NCC Group’s overall general process and what you can do to prepare to ensure a successful application security assessment.
Phase 1: Outline the scope and goals of testing
The very first step for every penetration test, before you even engage with any security vendors, is to make sure you know what you want tested and why. Arguably, the why is the most important; the goals of the test will help to determine both what needs to be assessed and how rigorous the assessment needs to be. For example, the scenario at the top of this article sets the goal as ‘marketing’, also known as ‘reassuring our clients’. Therefore, the testing should include the systems and functionality that those clients will use, and address any genuine, specific concerns clients have reported to you.
Other examples of possible reasons to get security testing are to meet the vendor security requirements of a company you want to sell to. In that case, that company will usually provide an explicit list or directions for what they want to see included in the scope of the penetration test. Finally, there’s also internally driven testing – your organization wants to bring in a third-party security vendor to provide their expertise in assessing some or all of an application’s implementation or deployment.
Knowing the goals of the testing also helps to set the expectations on what output from the testing you need. If it’s internally driven testing, a detailed technical report that lists all the issues found and how to recreate them is a great output for the development team. However, if the goal is to have a document you can share with customers, you’ll probably want something in addition that summarizes the results, without providing deep technical information or step-by-step instructions on how to exploit flaws in the system.
If the goals of the testing are somewhat undefined, bring what you do know to the table. Any good security vendor will work with you to help define the goals and craft an assessment that will address commonly seen scenarios. There are many types of penetration tests, we’ve done lots of them and are more than happy to discuss your concerns and issues to suggest areas that should be tested.
Phase 2: The scoping call
Once you know what you’re looking to have tested, and generally what the goals of the testing are, it’s time to reach out to a security vendor. At this stage, you’ll want to let the vendor know as much as you can about the driver behind the testing and the systems, as they will most likely want to set up a call to discuss technical details of the system(s) being tested and will want to make sure they have their most relevant technical staff available on that scoping call.
The scoping call is to get to the heart of the testing – what’s included, how large or complex the system is, what are the parameters around the testing. There are a couple of things that can make any penetration test more successful, and if these are available, you should be sure to inform the vendor.
1. Test application/environment- Security testing carries some inherent risk, it’s specifically looking for the potential for ‘bad behavior’ in the application. Therefore, having a test environment to use for the security assessment helps to limit any impact on your actual customers. We often request to test in a staging or UAT environment – close enough to production to accurately reflect how the application works, without being the production systems. This allows for the broadest use of testing techniques and fullest coverage of faults to look for. Also, the environment should be stable (i.e. not having new code deployed or configurations changed) for the duration of the testing, and ideally should be dedicated to the testing (i.e. not be a shared environment with developers or other clients also using it at the same time).
2. Internal knowledge- This can be design and architecture documents, existing threat models, email access to developers, and read-only access to the source code. Access to all of this will accelerate the testing (saving on effort and, therefore, costs) and provide better results. At NCC Group, we strongly advocate a code-assisted penetration testing methodology.
It is very valuable to have one or more senior technical person on this call, such as a product architect or a network engineer (depending on the kind of testing to be performed); someone who can answer most of the high-level technical questions that will come up during the scoping call. If this isn’t possible, very often questions can be sent via email after the call that can be forwarded on to your appropriate technical resources to get the answers needed to build an engagement plan for the penetration test. These questions can also be sent ahead of a scoping call, to gather the information first, and then reviewed and any clarifications that are needed can be done on the call.
Phase 3: Prepare for test launch
The output of a scoping call will be an engagement plan for the testing. As there are many kinds of penetration testing, it’s not possible to describe every possible permutation of how an engagement can be set up. However, the general structure of a security assessment is as follows.
Prior to the start of the testing, our Project Management team will reach out to you to help make sure that everything is ready for testing to start. This includes getting information on accessing the test environment and gathering test credentials, providing our IP addresses for any access permissions needed, setting up secure methods of information sharing for internal documents you share with us and our status updates and reports we provide back, and generally making sure that everything on both sides is ready to start on day one.
Phase 4: Testing
On day one, we would have a kick-off meeting between the technical testing team and your team supporting the assessment. The consultants will have reviewed the information provided during scoping and may have some additional technical questions about the systems being testing. Additionally, this call is a good opportunity to demo an application if it’s not intuitively obvious how it operates, and to review any other engagement parameters so that both sides have the same understanding about methods of communications, what testing limitations there may be, or other engagement-specific logistical details.
After the Kick-off call, the consultants will start interacting with the application to develop an understanding of how it works and what functionality it contains to put together an initial test plan. They will then start executing the test plan, using various techniques to attempt to discover vulnerabilities in the application and characterize their ease of exploitability and their impact to the overall security of the system or its users. This testing often leads to new areas to test, so this is an iterative process – develop an understanding of an area, test it, find some issues to report and some new areas to add to the test plan. This process continues for the duration of the testing. At this point, the consultants are mostly self-sufficient – this is what they do, so they don’t need much support from you during the testing. However, you will want to provide a technical point of contact in case something does go wrong – a test takes down a server, an area of functionality doesn’t seem to work as expected in the testing environment – so that the consultants can reach out immediately to get the issue resolved.
Phase 5: Reporting
As issues are found during the testing, they can be reported in multiple different ways. Usually, for any issues that have an overall severity score of High or Critical, we would report them immediately by emailing the point of contact provided and setting up a quick call, or writing up the results and passing them over through the secure file exchange. These issues are severe enough that we want to alert you to them immediately, so that you can start to address them in production right away. For most other issues of lower severities, we would report them informally during a weekly status update or the end-of-project Readout call.
Phase 6: Wrap-up
At the end of the project, we would schedule an initial Read-out call for early afternoon. Prior to the call, the consultants will share a draft of the report that includes all the findings discovered, their impacts, reproduction steps, and any other technical details on the findings. We encourage your technical teams to review the report before the call. On the call itself, we will provide a summary of how the engagement went and the findings, but the main goal is to allow you to ask any questions about any of the findings or the details in the report. We then add any additional details or comments provided on the call into the final version of the report. Additionally, as we know questions may arise later, we are available for further follow-up after the call, either via email or additional calls as needed.
Phase 7: Retesting and final reporting
For engagements like the initial scenario earlier, part of what a business’ customers want to see is not only that the business undergoes security testing, but that it also takes any results seriously and addresses them in a timely manner. Frequently, for this purpose, we will also return to retest the application after the findings have been remediated. In these cases, we need the same setup as for the original testing – a test environment with the deployed changes. We also would want information on how each finding had been fixed. Retesting is only re-evaluating the previously reported issues, not searching for new issues (unless they are found in the deployed fixes), and therefore usually is only a fraction of the effort of the original testing. Once completed, the consultants would update the original report with the status of each issue – fixed, not fixed (with details as to why), or even won’t fix (if you determine that an issue isn’t able to be fixed in the near term).
In addition, we can provide a summary report that the business can share with their existing or prospective customers detailing the engagement: the scope, the amount of time spent on the assessment, the methodology used during the testing, a summary of the results of the initial testing, and most importantly, the status of the issues reported after the retests. This document is the ‘evidence’ customers are usually looking for when evaluating the security posture of their vendors or partners – have they had a security assessment performed by a reputable company and have they addressed all the issues uncovered.
Conclusion: Pen testing phases summary
Now that you have seen the general flow of setting up and executing a security assessment, you should feel more prepared to meet the challenge. To summarize:
1. Know why you are getting a security assessment. What are your goals for the results? What kind of outputs do you need from the testing – just an internal report, or additionally one for your customers?
2. Know what needs to be in scope to meet those goals.
3. Provide a dedicated test environment, to keep issues caused by testing away from your production environment and paying customers.
4. Provide as much internal knowledge as you can to the testers – they have a limited time to perform the testing, so the less time they need to spend to figure out the system, the more time they must actually test it.
5. Provide a technical point of contact for during the engagement and set up secure means of information exchange, so that if issues crop up during testing, they can be communicated and resolved, and testing can resume as quickly as possible.
6. Before the Read-Out, review the draft report so that we can make sure the final report is as clear and understandable as possible.
7. If you are having the system retested, provide the same kind of environment as the initial testing as well as information about what was fixed and how.
Head of Solution Architecture, NCC Group North America
Graham and his team provide technical expertise in developing solutions to the cybersecurity challenges NCC Group's clients face. Prior to this role, he spent eight years working as a Principal Security Consultant for NCC in the Chicago office. He has performed security research on operating systems (primarily Windows, Mac OS and iOS), source code analysis, mobile platforms, and most recently hardware security. As a leader in the field, he’s also been responsible for managing complex projects and delivering quality results to clients.
Before joining NCC Group, Graham worked for the U.S. Department of Defense for over ten years as a Global Network Vulnerability Analyst. His technical knowledge has led him to author several security configuration guides, including the “Guide to Securing Microsoft Windows 2000 Encrypting File System.”
Trust in a partner with decades of penetration testing experience.
NCC Group combines industry-leading expertise with world-class service to build an assessment plan tailored directly to your organization's objectives. Learn more about our application & product pen testing solutions, or contact one of our experts for help today.