OWASP Top Ten 2021 April Update
We wanted to send everyone updates on our progress related to data analysis, survey, and format of the OWASP Top Ten 2021.
The Data
We are wrapping up the analysis of the data submissions, which is more intensive this cycle than before. Previous data calls for the Top Ten were limited to requesting ~30 CWEs with an option to write in more. While this would make analysis more straightforward, it has the potential not to show what’s going on in the wild. For this Top Ten cycle, we changed it up and asked for all the CWEs that organizations would map to their findings. Boy, did we get what we asked for; instead of 30 CWEs of data to process, we have data for ~380 unique CWEs to review and analyze. In 2017 we had data submitted that represented testing of ~144k applications; for 2021, we are looking at ~500k applications worth of testing data.
Similar to the last Top Ten, we are looking at incidence rate instead of frequency. We do this for two reasons. The frequency will allow a small number of risk categories to dominate the data analysis and hide more impactful issues. For example, if we have a population of 1000 applications and ten of them have an average of 4,000 instances of Cross-site Scripting (XSS) and 200 applications with an average of five instances of SQL Injection the frequency data might look like this:
Cross-site Scripting: 97.6% of vulnerabilities
SQL Injection: 2.4% of vulnerabilities
If we look at incidence rate, or how many applications in the population have a particular vulnerability, this provides a very different perspective (and arguably more accurate for measuring risk).
SQL Injection: 20% incidence rate
Cross-site Scripting: 1% incidence rate
One other item we are adding to this year’s analysis is the concept of “confidence.” When each organization sent us data, we had them send it in the form of:
“X apps were tested for Y CWE, and Z apps were positive.”
We can calculate an incidence rate for that population tested. However, we also have a larger population overall, and not all of these applications may have been tested for that type of vulnerability; this is where “confidence” factors into the equation. If we have a CWE with an incidence rate of 50%, but the total number of the overall population tested for it is just 1%, we determine there is lower confidence for that incidence rate. On the other side, if we have a CWE with an incidence rate of .67%, but the total number of the overall population tested for it is 92%, we have high confidence that it’s accurately representing what’s in the wild.
Side note: Yes, you can argue against this by bringing up factors like quality of test cases, differences in testing between organizations, and so on. However, trying to get to that level of detail is far beyond what a group of volunteers can be reasonably expected to undertake at this time.The Survey
Many thanks to everyone that took a little time from their day and completed the survey for the Top Ten! Without you, there are no results to analyze :-D. We finished with a total of 437 responses, which is close to the 516 we had in 2017. Reviewing the timeline of submissions, nothing looks strange or out of the ordinary, so no real worries about someone trying to game the survey.
As we have mentioned before, the reason for the survey is that solely relying on testing data has some limitations and blind spots. We will only get a volume of data on vulnerabilities found once we figure out how to manually test for them, convert that to automated testing, and scale it beyond a few organizations. As a result, we have a time lag, and looking at just the data will always be looking at some point in the past. The time lag is why we create the survey for people in the front lines to share what they believe are essential categories based on their experiences. We will use the survey results to pull in up to two categories that we don’t have data (yet) to represent.
I would like to look at some of the metadata of who completed the survey, as I think it holds a lot of insight and value. All the metadata questions were optional; the only required question was for ranking the top four categories one thinks are worthy of consideration in the Top Ten; thus, not all of these answers will have 437 responses.
For the first question about experience, the distribution is reasonably balanced; we can see the industry is aging which is no surprise. One takeaway from this distribution that we should pay attention to is the 28.5% of 0-3 years. The longer we are in the industry, the further from our origins we get, and we tend to forget that new people are joining our industry daily. We need to make it a priority to build paths to help group and mature people joining our ranks; otherwise, we will not be able to make the progress we dearly need to help improve security.
Looking at the position of people completing the survey, in-house security is the largest group, with consultants at number two. We support the write-in option for most of these questions because the industry is still a long way from maturing, so we can’t possibly provide every viable option to select. 6.6% of responses wrote in something other than the two options. We have representation from both Professors and Students, people who wear multiple hats, and others.
The primary role question usually receives the most write-in options, and for a good reason, we don’t have a good role standardization in our industry. There is quite a laundry list of submitted roles that people are assigned.
Looking at the position of people completing the survey, in-house security is the largest group, with consultants at number two. We support the write-in option for most of these questions because the industry is still a long way from maturing, so we can’t possibly provide every viable option to select. 6.6% of responses wrote in something other than the two options. We have representation from both Professors and Students, people who wear multiple hats, and others.
The primary role question usually receives the most write-in options, and for a good reason, we don’t have a good role standardization in our industry. There is quite a laundry list of submitted roles that people are assigned.
Advisor / Analyst
Developer / Engineer
Tester
Monitor / Responder
Management
Architect
Researcher
Consultant
Advisor, Tester, Management, Respon…
Audit
Vuln Researcher
Secure Coding Educator
Vendor / Advisor
security architect
Hacker
Multiple
Technical Evangelist - advisory and training fit most closely
Security Architect
Information Security Architect
Security engineer
Red Team
The last metric to look at is how the Top Ten impacts work. The respondents are allowed to select all that apply, as the Top Ten can be used for many things and explains why it doesn’t add up to 100%. The number one answer is “It helps provides structure for standards, requirements, security tests, test results, etc.,” with 72.8% of the respondents selecting that option. Number two is “Mostly for the education of developers,” and number three is “We build processes around it.”
Formatting
For this round of updates for the OWASP Top Ten, we plan to focus
on a more mobile friendly version of the output. We also plan to update
graphics and provide more supporting content and links from the Top Ten. More updates to follow in the near future.
