Threat hunting is a topic that’s used pretty liberally these days to describe everything from research to investigation to incident response, but as Elise Manna-Browne explained in her October Innovate Cybersecurity Summit presentation Insights of a Threat Hunter, it’s a very specific discipline with a focused scope and methodology.
This brief primer on threat hunting based on her presentation will cover the process of building out a threat hunt: how to plan it, the strategy involved, and how to execute it.
What Threat Hunting Is, What It’s Not, and Approaches
Threat Hunting is a proactive measure – emphasis on the word proactive.
Elise Manna-Browne, Director of Threat Intelligence
Threat hunting assumes that compromise has already happened in some way, shape or form. The process involves proactively searching for cyber threats, vulnerabilities, and malicious actors hiding in your environment that have somehow escaped detection by the rest of the security toolset.
Still, a threat doesn’t always come from a cyber attacker; sometimes risks are just present in your environment, perhaps in a more mundane form. Examples include: exposed stored application credentials waiting to be stolen, or a critical unpatched vulnerability in a critical appliance. Not malicious, but certainly risky and considered a threat.
What It Is
- Proactive searching for evidence of compromise, anomalies, or risks in the environment
- Assumes that compromise has already occurred
- Can discover other risks in the environment
What It’s Not
Threat hunting covers a lot of ground, but here is what it’s not:
- It’s not searching for indicators of compromise (IOCs) such as:
- Domains
- URLs
- Hashes
- It’s not incident response or responding to alerts
Both incident response and alert monitoring are reactive, not proactive activities and really a waste of your threat hunters’ time.
Threat hunters are on a mission to find something, often based on some type of data or intelligence. They are a scout exploring the unknown—a human alerting mechanism as opposed to an analyst investigating alert data or prepared advisories.
Approaches to Threat Hunting
While there are many different approaches to threat hunting, the three main go-tos are:
- Intelligence-driven
- Data-driven
- Experience-driven
The approaches are differentiated by the motivating force. Intelligence drives a hunt by using an inkling of which threats might be actively targeting the environment. Data drives the hunt by providing a raw bulk of collected data that a hunter can analyze to find anomalies or things that contrast against expected results. And experience drives the hunt when a hunter can rely on a past association of patterns or previous similar events to draw conclusions or clues.
Intelligence-Driven Threat Hunting
The intelligence-driven method relies on intelligence data from external sources such as threat feeds or research. It can also be internally collected form inside the environment, such as alarms/alerts.
For example, a particular threat actor is perpetrating a known malicious behavior, or we know a particular malware X does a specific thing. These specific notable characteristics and actions provide a starting point or initial target for the hunt.
Data-Driven Threat Hunting
Data-driven threat hunting starts with raw data. The data may be tabled-based, a chart, or something similar that the threat hunter examines using data analysis techniques in the search for anomalies.
Whether a small or large data set, using a query language to sift through it by clustering, stacking, using frequency analysis or pattern identification can reveal something critical to a hunt. Finding one needle in a haystack of data may result in a cascade of additional discoveries.
Experience-Driven Threat Hunting
Finally, there is experience driven threat hunting which considers the common hallmarks or characteristics shared by similar environments. While not everyone will notice them, threat hunters may pick up on specific user behaviors or known weaknesses of a particular set of tooling in an environment as well as other indicators that can help drive the hunt forward.
An experienced hunter sees patterns of the same things happening again and again.
Building Out a Threat Hunt: The Process
A good place to start is the intelligence space, often with external sources, such as FBI Flash Reports (example) or CISA. These can act as an impetus for the hunt, to explore vectors perhaps not previously considered.
Focusing internally, event-oriented intel or indicators might include:
- A tool not doing what it should
- A log source going offline
The idea is to take external and internal intel, multiple sources combined, and de-duplicate the data. If multiple sources say something exists, it’s both corroborating and allows it to be noted in the hunt plan only once.
Planning Your Hunt
There are two questions that should lead the planning:
“What am I looking for?”
“Where can I look for it?”
An example of good planning is identifying where the resources being examined are located. If looking for events that exist in IIS Logs, don’t waste time mining the EDR tool. Knowing or learning the role each tool plays allows a threat hunter to make effective use of their resources.
In the case study below we’ll further explore the elements of the threat hunting process.
Executing the Hunt
Mining and analyzing data is central to a hunt— it often begins with executing a query from the appropriate tool. Good query language and data analysis chops are absolutely the most valuable skills for hunting, as knowing how to include, exclude, or conditionally key on certain data is what allows a hunter to pinpoint a clue buried in millions of rows of logs or telemetry data.
Getting millions of result rows back? Search times out entirely? Knowing the tricks for handling large data sets is also a crucial skill. Getting it right on the first go is rare—it takes a lot of fine tuning, sometimes through exhaustive iteration to achieve the final, perfected version of the query that returns an expected result.
Once completed, the result can be a fully optimized alert custom to your environment.
It’s helpful to document all steps of the process, including any queries, as it’s common for a similar hunt to be repeated, or cases to be handed off to a new team entirely.
The Pyramid of Pain
The Pyramid of Pain is a graphic meant to depict how difficult it is to detect threat actors based on specific indicators.
An important consideration is the utilization of threat hunters—Where should automation be employed and where should human hunters be used to optimize resource usage?
Example: Tasks near the top of the pyramid, such as Network/Host Artifacts that contain user agents or things left behind by an attacker, Tools such as PS Exec, and TTPs all represent behaviors of threat actors. These can be combined to conduct a hunt on a specific threat group based on gathered intel, and should always be carried out by the threat hunter. Those on the lower end, however, can be achieved through automation as many tools exist to perform IP and hash lookup.
Lockbit Case Study
Lockbit 2.0 is a notorious ransomware known for extremely fast encryption and triple extortion tactics. Let’s apply the threat hunting process to this particular threat:
What is Triple Extortion Ransomware?
ex·tor·tion /ikˈstôrSH(ə)n
nounthe practice of obtaining something, especially money, through force or threats.
In double extortion ransomware, threat actors encrypt the data and then threaten to leak it if the ransom isn’t paid.
Triple extortion adds a denial-of-service component to the mix, adding attacks against the target’s customers, employees, or patients. Taking a trick from the Egregor group they may even take over network printers and print out paper ransom notes.
Step 1: Gathering Intelligence
Initial intelligence gathering involves checking many resources—wherever information on the given threat can be found. Public intelligence sources such as blog posts are a good starting point. Making use of internal intelligence of at least TLP:Amber classification is also extremely useful.
When Manna-Browne’s group actually performed a hunt for Lockbit 2.0, these three public intelligence sources were useful:
- Cyble.com: A Deep Dive Analysis of Lockbit 2.0
- Trend Micro: Lockbit Resurfaces with Version 2.0
- Cybereason vs Lockbit 2.0 Ransomware
Overall, the goal is to diversify the types of intel to improve success in the hunt. One security research group may have a single sample of the threat, but another may have 30 based on different industries and geographic regions. It lets them analyze different components of the same threat actor group or malware family.
Multiple sources of intel helps to account for variability that make attacks look somewhat different even when the actor is the same.
Step 2: Synthesis
The execution of the hunt hinges on the types of intelligence available. Consider the diagram below which shows all Tactics, Techniques, and Procedures (TTPs) used in Lockbit 2.0 ransomware, mapped out using MITRE ATT&CK Navigator.
That’s a lot of behaviors to cover. An in-depth look would require one query per behavior at minimum. This isn’t a hunt that could be covered in one day; more likely several days.
After collecting TTPs, de-duplicate and supplement any gaps with other intel. This is helpful in mapping out each tactic for the entire kill chain.
Add actionable items to a TTP hit list. There are many ways to do this, but the simplest is to make a list in a spreadsheet which can be later used to copy and paste into a query for an EDR product or Splunk.
Step 3: Formulate the Plan
Armed with intelligence and a strategy for which techniques to search for, the next logical step is to ask “where to look?”
It’s essential to know which sources potentially contain the evidence and/or indicators. Different types of indicators are going to be found in different places. Here’s a short breakdown of where to look for what in this case study:
Found in EDR or OS Logs:
- Process name
- Process command-line
- Parent-child process relationships
- Script launch by name or default program
- Registry (create/write/modify/delete)
- Process + network connections
- Service/task creation
Endpoint:
- Script contents
Firewall/Proxy or Visualization:
- Anomalous network traffic
PCAP or Web Server Logs:
- HTTP request/response
- URIs
The above is a sample hit list. It may be on the small side, but it contains many different types of information. For example, there is:
- a URI which includes
uploadfile.php - a POST request to a naked IP address are two components of the same TTP that come from a Lockbit scenario.
Lockbit will utilize an IP address with no associated domain, forward slash it via the URL uploadfile.php, then append all the stuff with the system info at the end.
By separating the behaviors on the hit list, it can allow discovery of other possible threats on user devices. For example, browser traffic to a naked IP address can be something else. It can reveal a different malware family than was originally being searched for.
Step 4: Hunt Execution
In developing a hunting plan, it’s essential to consider how different EDR products handle data and character formatting. In performing searches or queries of the data, be sure to accommodate for anomalies, such as diversity in attacker scripts or their order of operations.
For best chance of success, adjustment of queries may be necessary to cover any variations employed by the threat actor.
In executing queries, tune the query parameters to achieve adequate filtration of the data to allow some meaningful analysis, but avoid timeout on large data sets by not filtering too much. A good general approach for data scope is:
- Accommodate potential changes in attacker scripts or order of operations
- Accommodate for largest time range possible without degrading search performance
- Accommodate for different potential parent processes
- Accommodate for cAmEl CaSe, CAPS, or other weird character casing
Accommodate for different parent processes (example: cmd.exe OR wscript.exe launching Powershell—that variation may change from attack to attack), as well as weird case scripts by using case-insensitive options if they’re available in your chosen tools.
Here’s an example case in querying a EDR data set to refine results by specifying additional fields for specific command line and accommodating for multiple character case scenarios:
(Name = "powershell.exe" OR Cmdline ContainsCIS "powershell") AND Cmdline In Contains Anycase ("-nop -w hidden -C","GetProcess -FileVersionInfo -id","Set-Content-Path","FileName-Force -Confirm","RemoveItem -Path","-Force -Confirm")Intelligence-Driven vs Data-Driven
Example of an intelligence-driven query:
index=netproxy sourcetype="zscalernss-web" | search url="*/uploadfile.php*"This allows the query to run using a specific source type while looking for the URI pattern. It includes wild cards that allow for any hostname and any trailing query string parameters after the file name.
Example of a data-driven query:
index=netproxy sourcetype="zscalernss-web" | search url != "*.co.uk*" AND url != "52.11*"The data driven approach shows the same index and source type but limits the limits the scope of the query by using conditionals to exclude top level domains and IP ranges in known data, and removes specific octets used in this environment.
In this scenario, uploadfile.php wasn’t included to allow for variations that could be other malware or threats in the environment.
Conclusion
Threat hunting is a proactive endeavor. Results are best when it’s executed outside of an incident response scenario and adequate time can be spent going down the necessary rabbit holes to thoroughly explore any clues or new suspicious findings that occur during the hunt.
When starting a new threat hunt there are a few tenets that can improve success:
- Decide the objective of the hunt and have a hypothesis in mind.
- Collect available intel and synthesize it.
- Map intelligence and behaviors using available data and tools.
- Construct accurate (if not precise) hunt queries.
- Tune noise and false positives out.
- Create alerts or adapt security controls.
