University Researchers Pit Top 18 EDR Products Against Real-World Attacks

As the industry becomes more committed to relying on the latest class of endpoint protection products, one has to ask “just how effective are these solutions?” While modern EPP and EDR offerings are better than the old crop of anti-virus and anti-malware, it is only recently that researchers have begun to evaluate their effectiveness with real-world threats.

A report published by security researchers from the Athena Research Center at the University of Piraeus in Greece titled “An Empirical Assessment of Endpoint Security Systems Against Advanced Persistent Threats Attack Vectors” tested 18 vendors against four scenarios to see how they would perform. Spoiler: the vendors as a whole did not do well.

While only 4 scenarios were used in the evaluation, it does line up with the findings in our research from November 2021 in which known bad hashes from thousands of endpoints in the wild were used to test a large group of EPP/EDR vendors for accuracy in identifying them as such.

University of Piraeus report

Recorded Future published an article last week highlighting the findings of the Greek researchers’ report. The two researchers, George Karantzas and Constantinos Patsakis, erected a Cobalt Strike C2 attack framework intended to test 18 different EDR products with real-world attack vectors.

Their experiment was designed to answer the following questions:

• RQ1: Can state of the art endpoint security systems detect common APT attack methods?
• RQ2: Which are the blind spots of state of the art endpoint security systems?
• RQ3: What information is reported by EDRs and EPPs and which is their significance?
• RQ4: How can one decrease the significance of reported events or even prevent the reporting?

They began with purchasing a mature but expired domain and secured it with a legitimate Let’s Encrypt SSL certificate in order to host 4 different files types necessary for the diverse but realistic attack vectors:

1. .cpl – A DLL file crafted using CPLResourceRunner which can be executed by double-clicking under the context of the rundll32 LOLBINS which can execute code maliciously under its context. A shellcode storage technique using Memory-mapped files was employed then triggered using delegates.
   
2. A legitimate Microsoft (MS) Teams installation that will load a malicious DLL. In this regard, DLL side-loading will lead to a self-injection, thus, allowing the attack to ”live” under a signed binary.
   
3. .exe – An unsigned .exe to execute process injection using the “Early Bird” technique of AQUARMOURY into werfault.exe. They would spoof the parent of explorer.exe using the PROC_THREAD_ATTRIBUTE_MITIGATION_POLICY flag to protect their malware from an unsigned by Microsoft DLL event that is commonly used by EDRs for processes monitoring.
   
4. .hta – Get a user to visit a harmless HTML page containing an IFrame from where they are redirected and prompted to run an HTML file infused with executable VBS code that will load .NET code which performs self-injection under the context of mshta.exe.

Karantzas and Patsakis thoroughly describe the construction and execution for each of these vectors in the paper. It is representative of a fairly standard modern attack on a Windows endpoint, with each of the links in the chain being discrete and theoretically well-documented; something a well-designed EDR should detect.

Result

Their attacks were tested against EDR software from Bitdefender, Carbon Black, Check Point, Cisco, Comodo, CrowdStrike, Elastic, ESET, F-Secure, Fortinet, Kaspersky, McAfee, Microsoft, Panda Security, Sentinel One, Sophos, Symantec, and Trend Micro. 

Only two vendors had full coverage for all attack vectors. Here’s how they fared:

EDR	CPL	HTA	EXE	DLL
BitDefender GravityZone Plus
Carbon Black Response
Check Point Harmony
Cisco AMP
Comodo OpenEDR
CrowdStrike Falcon
Elastic EDR
F-Secure Elements Endpoint Detection and Response
FortiEDR
Microsoft Defender for Endpoints
Panda Adaptive Defense 360
Sentinel One (without test features)
Sentinel One (with test features)
Sophos Intercept X with EDR				–
Trend micro Apex One

Endpoint Protection	CPL	HTA	EXE	DLL
ESET PROTECT Enterprise
F-Secure Elements Endpoint Protection Platform
Kaspersky Endpoint Security
McAfee Endpoint Protection
Symantec Endpoint Protection

• – Successful attack
• – Successful attack, raised medium alert
• – Successful attack, raised minor alert
• – Successful attack, alert was raised
• – Unsuccessful attack, no alert raised
• – Failed attack, alerts were raised
• – In two experiments supplied by the vendor, in the first it was detected after five hours, in the second it was detected after 25 minutes
• – Initial test was blocked due to file signature, second one was successful with another application.

The Innovate Cybersecurity research

Innovate Cybersecurity evaluated the efficacy of 22 popular EPP and EDR products, finding that none scored better than a 50% detection rate.

The aggregated detection results used data from SOC-as-a-Service provider Pillr, which collected 26 million file hashes from endpoints during the month of October 2021. File types included executables, Office docs (.docx, .xlsx, .pptx, etc), PDFs, and scripting files with executable code.

Read the full detail on how the test was performed.

The Result

October 2021 Pillr data set | Percentage of known threats detected:

Of 22 EPP and EDR products in use today, none performed better than 50%. All but two vendors were below 40%.  Our sense is that efficacy is frighteningly lower than what consumers expect when purchasing these products.

What should security practitioners do with this information?

Cybersecurity teams should consider a layered approach. Today this may look like more than one EPP tool and a Managed EDR or SOC-as-a-Servce solution layered on top.

References

George Karantzas and Constantinos Patsakis. “An Empirical Assessment of Endpoint Security Systems Against Advanced Persistent Threats Attack Vectors [2108.10422].” ArXiv.Org, 11 Jan. 2022, https://arxiv.org/abs/2108.10422