DevSecOps Days - DC

The scale and complexity of mass production software has made manual testing for security related vulnerabilities effectively impossible. Automated security testing tools are required to rapidly assess the security issues in software.  In our study, we ran ten popular SAST tools against 8 large (7.7 MLOC total) mature open source projects. The tools found 685K total defects. Less than 1% of the defects were common between two or more tools. To determine commonality, the defect results were normalized based upon the consistent and accurate application of MITRE defect definitions. It was found that 61% of defect rules were either mis-characterized or mis-aligned. Using mathematical analysis of code properties that causes false positives by defect rules, it was also found that 61% of  Java and JavaScript defects had high validity confidence, whereas only 21% of C/C++ findings were high confidence.  Detailed analysis of attack patterns showed that only 5.5% of the found defects were easy to exploit. By using a novel probabilistic approach to determine severity of consequence, it was discovered that only 6.5% of the defects were highly severe.