The following is a loose taxonomy of tool functions and commercial services. There is not always a clear distinction between a tool and a service. Some tools are just that – a piece of software you use with unchanging functionality until you choose to upgrade. Other tools have a set of security experts behind the scenes that are enhancing and modifying the behavior daily. And of course some service offerings have no “tool” element at all.
Different tools tend to focus more on one sort of analysis than another – both from a marketing and a technical perspective. But it quickly gets a muddled. Many tools can perform more than one of these functions and many of them also provide functionality that is not aimed strictly at application security.
Disclaimer: The market for security tools and associated services is large – around $5B in 2021. Examples of some representative tools and services are provided; this is not compressive and it is not intended as an endorsement of specific tools or service offerings.
Selecting proper tools and services is an important part of an Application Security program. This material is intended to help frame that selection process.
Tools
This section describes different classes of tooling. The Resources page provides examples of specific tools and links to other lists of tools on the Internet.
Classic Static Analysis
Static analysis tools (also referred to as SAS or SAST – Static Application Security Testing) examine a body of source code and then make predictions about possible misbehaviors. For example if there is a path through code that might reference an uninitialized variable or leak a resource. This is accomplished by building a model of the code rather than literally trying to execute it. Static analysis tools may also point out bad code smells such as dangerous C macro definitions, code indentation that does not match the logical structure, and abuse of cut-n-paste.
Legacy Static analysis tools originally focused on code quality issues. These tools have moved to embrace security via technical extension (e.g. identification of code injection opportunities) as well as marketing (that buffer overflow we always cared about is now a security issue).
Static analysis is the first line of defense for the code your team is writing.
Dynamic Analysis
Dynamic Security Testing involves running the application and looking for problems along the way. This class of testing tool needs to be configured for the application being tested – at the very least with a set of starting URL’s; but often much more.
OS Supply Chain Verification
These tools aim to insure the Open Source you are using is in fact the open source you think you are using. These tools may also attempt to root out zero-day vulnerabilities via some combination of automated analysis and security expert services.
The significance of this type of threat has increased with the advent of package managers such as NPM and Maven since these tools are typically configured to go out to the Internet automatically to resolve dependencies. With nested Open Source references, a developer may now be using a package they did not even know they were using.
Composition Analysis
From a security perspective, Composition Analysis is a subset of OS Supply Chain Verification. It differs in two respects:
- The tools usually also provide functionality for managing licensing. In fact, like the legacy static analysis tools, this is likely where they started.
- The tools may provide functionality for managing commercial as well as Open Source.
These tools typically operate on “inputs to the build process” including source files, Java jar files, package manager databases, etc.
The security side of these tools relate what they find to known vulnerability databases.
Security Scanners
Security scanners typically probe a running system. At the most basic level this is a port scan, but most tools go considerably beyond that by leveraging some provisioned knowledge of the system. For example to log into a Linux shell and examine the file system. These scanners relate what they find to known vulnerability databases. They also leverage knowledge of known exploits to “try things” on the open ports to ascertain vulnerability.
A notable aspect of these tools is that they are configured via a set of continually updated rules managed by the tool vendors. The upshot of this is that a system that tests clean this week might show vulnerable next week as new vulnerabilities are discovered.
Malware Scanner
A malware scanner examines (typically) binary files and looks for known byte patterns corresponding to computer viruses and other malware. We usually think of this more in the context of IT hygiene. For example most enterprise e-mail systems subject incoming e-mail to this sort of evaluation. If our work environment is completely clean and if we have complete confidence in all input sources there would be little to gain here. But that is seldom true, so this sort of scan does provide an additional layer of defense (and some types of customers will try to demand it).
Image Scanners
Image scanners start with a production image of some sort and then perform various types of analysis including composition analysis, static analysis, and others. Arguably this is not so much a “type of analysis” as it is a “type of analysis input.” That said, there does tend to be a difference in emphasis. It important to keep in mind that just like security scanners, this sort of tool is often available to your customers.
Compilation Environment Tools
In the GNU and CLang world we have tools such as ASAN (Address Sanitizer) that augment static analysis allowing us to detect additional coding flaws that are not subject to static analysis detection at the current state of the art. Other tools exist for different environments and development languages.
Fuzz Testing
Fuzz testing involves injecting malformed data into a system in an attempt to break it. Like several of the other testing mechanisms discussed here it has long been used to support the more general pursuit of system quality and robustness. The Synopsys Defensics tool is an example of a commercial multi-protocol fuzz tester. There are a number of other commercial and open source tools.
Other Pen Testing Tools
Some of the tools already discussed such as security scanners and dynamic testing will be used by Penetration testers, but there are lots of other tools used by Pen testers – and no doubt by black-hat hackers too. For example in support of scripting known vulnerabilities. The applicability of these tools to a particular development activity depends on how far the team is going in the direction of “validating invulnerability.” It us useful for developers to understand that they exist because the tools provide a sense of how actual exploits work – and especially how easy it is for even novice hackers to leverage known vulnerabilities.
Services
There are many firms with Security Consultants ready to work for you. You will likely find that if you can describe a problem you can find a number of vendors able to propose a consultive solution. That said, is helpful to consider a few common services.
Architecture and threat modeling
These services may be used “up front” to help to “design security in” or after the fact in an attempt to enumerate the threats to a system.
Process and Policy Audits
This service would be used to assess security related process and policies. Possibly in a standalone fashion or possibly in the context of specific industry guidance. A variation on this would look for guidance in the design of security programs.
Security Assessment
A security assessment aims to answer the question “In what ways is this system likely vulnerable?” The input is usually a technical discussion with the development team and often hands-on interaction with the system in question. The output is a report. Security assessments may also be available via partner relationships. For example AWS offers the Security Lens for applications be deployed on AWS.
Penetration Testing
Penetration takes the hands-on Security Assessment to the next level. The penetration test aims to actually demonstrate weaknesses by compromising a system. Like the security assessment, the output of the penetration test is a report of some sort.
The penetration test is the poster child for security consulting. If you are only going to arrange one service, it is likely this.
Penetration testing and the associated hands-on security assessment is highly skilled work that typically leverages both open-source and proprietary tools.