NOVEMBER 9, 2021
The current market of Endpoint Protection (EPP) and Endpoint Detection & Response (EDR) products are supposed to be far more effective than their signature based anti-virus and anti-malware predecessors. While we believe the new generation of EPP tools are better, how effective are they?
Working with the threat intelligence team at SOC-as-a-Service provider Pillr, we’re publishing a monthly report on the efficacy of modern EPP and EDR tools. The report includes information about the data set and the process used to evaluate the data.
About the Data Set
The data is collected by Pillr and is comprised of millions of file hashes collected from customer endpoints. These may include:
- Executables
- Office docs (.docx, .xlsx, .pptx, etc)
- PDFs
- Scripting files with executable code
Full list of file extensions analyzed:
exe, msi, ps1, doc, xls, ppt, docx, xlsx, pptx, cab, dll, pdf, 7z, gz, tar, zip, iso, com, jar, js, vbs, py, php, cs, bat, vb, wsf, pif, xlm, otm, cmd, scr, ear, wsh, nexe, sct, jseThe Process
The following steps are taken to evaluate the data set:
- We start with 30 days of endpoint data which includes every file hash seen.
- Hashes that fall into one of the following categories are then filtered out: benign, unknown, unclassified, whitelisted, and duplicates, as well as anything else our research team doesn’t believe is malicious.
- The remaining hashes are compared against multiple sources of threat intelligence, both commercial and open source feeds. Anything that those don’t agree is malicious is filtered out. That way we are in 100% agreement with those 3rd party threat research teams as a secondary validation.
- The data is then filtered again to remove any hashes which at least 5 EPP/EDR vendors don’t agree are known malicious.
- Submit the data to an independent and impartial aggregator of data on viruses, malware, malicious IPs, malicious domains, malicious URLs, and other threat types. In this case, Virustotal.
After filtering October 2021’s 26 million collected file hashes, 1751 were submitted to Virustotal, who returned data on which EPP and EDR products detected the threats.
The Result
October 2021 data set | Percentage of known threats detected:
Conclusion
Of 22 EPP and EDR products in use today, none performed better than 50%. All but two vendors were below 40%. Our sense is that efficacy is frighteningly lower than what consumers expect when purchasing these products.
