Establishing an effective cyber threat hunting program is among the top priorities of enterprise security leaders seeking a proactive approach to detecting and counteracting potential threats. Furthermore, implementing a proactive threat hunting program, security teams that leverage formalized frameworks or threat hunting methodologies are far more likely to detect vulnerabilities or in-process malicious activities in their environments than those that do not. However, data from a 2023 threat hunting survey revealed that while 73% of organizations have adopted a defined threat hunting framework, only 38% actually follow it.

In this introductory guide to threat hunting frameworks and methodologies, we explore the benefits of threat hunting frameworks, how to hunt for security threats using the scientific method, and the capabilities security teams need to support effective threat hunting and enhance the organization's cybersecurity posture.

Let’s get started.

Why Employ a Formal Threat Hunting Framework?

Threat hunting is a purposeful and structured search for evidence of malicious activities that have not yet triggered existing security systems - it’s a human-centric activity that pushes the boundaries of automated detection methods.

While there are many different activities that can help detect threats, ranging from simple file-matching exercises to complex log analytics for security operations and threat hunting, not all of these detection activities are of equal value.

As a starting point, cyber threat hunters will structure their hunts to seek out evidence of particular indicators of compromise (IoC) that, for example, they may have learned about from threat intelligence feeds or other sources of cyber attack data.

More mature organizations will begin to curate their IoC lists by incorporating contextual information (i.e. from SIEM and threat intelligence reports) to improve the quality of the intelligence. Some will add machine learning-aided anomaly detection to help them better understand baseline conditions in the environment. And the most mature of all will perform hypothesis-driven hunting that relies on anomaly detection and IoCs selectively as inputs that inform a more comprehensive understanding of the organization’s specific risk profile.

Just as not all detection activities are of equal value, not all IoCs are similarly worrisome if discovered. Generally speaking, the easier it is to detect an IoC, the less likely its detection is to interfere with attackers’ ability to accomplish their objectives.

Celebrated network security expert David Bianco makes this point in a graph he calls the “pyramid of pain.”

Source: Enterprise Detection and Response

The IoCs at the base of the pyramid are trivial to detect with a simple anti-malware scan. However, this finding means that attackers have successfully installed previously-discovered malware in your environment.

This is often among the earliest stages of an attack, and without evidence that further exploration of the network has taken place, doesn’t mean – in and of itself – that a major incident has occurred.

Formal threat-hunting frameworks enable DevSecOps teams to reach the top of the pyramid far more quickly and easily by providing a comprehensive library of all the steps that attackers may take in order to achieve their objectives. Relying on frameworks enables security teams to be thorough and exhaustive when hunting. Moreover, a standard methodology enables the SecOps team to ensure consistency over time in their threat hunting function, and fosters continual learning and improvement as lessons learned from each hunt are captured and used to inform future hunts.

Knowing What to Look for

Though adversaries are always seeking to enhance their capabilities by exploiting previously undiscovered vulnerabilities or honing new techniques, the majority of attacks follow the same general trajectory — from initial exploration of the victim environment, through detailed reconnaissance and attempts to gain access to privileged credentials, all the way up to data exfiltration or another means of achieving the objective.

Image Source: The general trajectory of a cyber attack, sometimes known as the cyber kill chain.

Threat hunting frameworks help security teams focus the threat hunt process on uncovering traces of the most meaningful activity patterns. Hunters can then draw upon comprehensive and relevant research and intelligence when formulating the hypotheses they’ll investigate within the hunt.

Signs of a Data Breach or Attack

In pursuing a sophisticated, long-term attack, the attackers will often:

• Combine multiple techniques within a single campaign that occurs over a relatively short time frame,
• Repeatedly attempt to gain or elevate access privileges,
• Change the contents of endpoint file systems- often with one or more endpoints as well as other network resources,
• Alter system configuration settings to deviate from what’s normal for the organization,
• Move data or suspiciously alter remote server connection patterns

The biggest challenge in threat hunting is distinguishing the “signal” — that is, true evidence of malicious activity — from “noise” — the wide range of diverse activities that take place among legitimate users across the computing environment every day. Threat hunters know that the true signals are there, hidden in the daily noise. Threat hunting is the art and science of analyzing the data to uncover these hidden clues.

Applying Threat Hunting Methodologies

The most mature threat hunting teams follow a hypothesis-based methodology that’s grounded in the scientific method of inquiry. They acquire their knowledge through logical reasoning and empirical evidence, with the intent of preventing biases and assumptions influencing results.

Each threat hunt progresses through some version of each of the following steps.

1. Define the Attack Scenario

Rather than generally searching for various types of threats, the starting point is to define a specific, narrowly focused threat that could be underway in the environment. In this step, the hunter should think through the overall techniques that could be used, the targets within the network that could be attacked, and the various vulnerabilities that can be exploited.

2. Formulate an Initial Hypothesis

In thinking through the goals of the attacker for each stage in the attack chain, threat hunters make a series of informed guesses about what tools and techniques the attacker might use and what evidence might be created by their activities. The hunt is then structured to look for the evidence that would be generated if indeed each sequential hypothesis is valid.

3. Identify and Gather Evidence to Investigate each Hypothesis

Your hunt team will need to assemble the data sources that will be analyzed within the hunt. To prove or disprove a hypothesis with a high degree of confidence, multiple forms of evidence are usually needed. Hunters will also need to document where data comes from to ensure that sources are both contextualized and consistent.

4. Leverage Analytics to Reveal Results

During this stage, evidence is correlated and subject to analytical and visualization techniques to uncover relationships within it. Threat hunters need a deep understanding of adversarial tradecraft as well as what’s normal within the environment to be successful here.

5. Report Results

It’s key to document the types of evidence collected, the nature of the analysis performed, and the logic behind the conclusions that are reached while the hunt is still in process. This enables the hunt team to communicate with management as well as incident responders.

Supporting a Formal Threat Hunting Framework

Security teams making their first foray into threat hunting might do so on an ad hoc basis, but hunts will be most effective with clear goals and a well-structured process. This makes it easier to repeat the investigative strategies that have given you the best results. It also facilitates measuring outcomes, presenting findings to incident responders, and incorporating what’s learned into the organization’s automated alerting infrastructure.

Though it’s possible to conduct a hunt according to a self-designed or threat intelligence vendor-supplied framework, the vast majority of security programs rely on the MITRE ATT&CK framework.

Image Source: The MITRE ATT&CK framework provides information on adversarial techniques, tactics, and common knowledge across the cyber kill chain. Enterprise SecOps and threat hunting teams can use this information to predict adversary behavior and guide threat hunting activities.

MITRE ATT&CK incorporates hundreds of known adversarial tactics and techniques into a globally accessible open-source knowledge base of cyber attack data.

Putting Threat Hunting into Action

The most important success factor in threat hunting is fast access to the right data, including long-term historical security data. With access to the right data sources, threat hunters can use analytics tools like Kibana to carry out their hunts by running queries and creating visualizations that draw out the important signals of a potential attack.

A critical step for organizations looking to improve their threat hunting capabilities is to build a security data lake that can operationalize security data at scale and support the massive data requirements of threat hunting analytics.

Chaos LakeDB helps merge the vast storage capabilities of data lakes with the accessibility of cloud databases, empowering threat hunting teams with the rich historical data and observability integrations they need to detect even the most elusive and persistent cyber threats. ChaosLake DB integrates directly with Amazon S3 to eliminate the need for complex extract, transform, load (ETL) processes and deliver award-winning cybersecurity analytics while ensuring enhanced cost efficiency and performance at scale. With ChaosLake DB, threat hunters can accelerate security insights, spend more time hunting for IoCs or malicious activity instead of worrying about the accessibility of data sets, and dramatically improve the overall security posture of their organizations.

Ready to learn more?

Download our free white paper Achieving the Security Data Lake to learn more about operationalizing security data and enhancing your threat hunting program with Chaos LakeDB.