High-Fidelity Threat Hunting Starts with Your Data Pipeline

SANS recently released its 2026 Threat Hunting Survey, and for the first time in the report's eleven-year history, respondents did not cite “lack of skilled staff” as the primary barrier to threat hunting success. The biggest challenge now? The timeless and complex problem of data. Half the surveyed organizations are struggling to give their hunters the appropriate data quality or volume to be effective.

Here at Zscaler, we've built and scaled threat hunting services across thousands of endpoints, identities, network transactions, and cloud resources — we've had to solve this data problem ourselves. The answer doesn’t come in the form of a specific tool. It comes when you treat your data pipeline as a first-class engineering discipline, giving it the same rigor you would to the skills and expertise of hunting itself. We'll walk through how we've solved specific data challenges, so your hunters can spend less time troubleshooting and more time hunting at scale.

Telemetry Firehoses and Blind Spots: Dialing in Visibility

Having enough data is necessary but not sufficient for effective hunting. Vendors have made strides in closing traditional visibility gaps—for example, EDR captures the command line telemetry that native Windows logging leaves disabled—but as the SANS survey highlights, blind spots in cloud and identity still plague teams. On the other end of the spectrum, high-volume, rich telemetry can be overwhelming to hunt through without sufficient filtering and enrichment.

Teams have to actively engineer solutions for both missing data and data overload. 

Filtering noise to get to signal
Let’s first look at the high-volume side of the spectrum, since at Zscaler we know it well, given our Zero Trust Exchange processes 750 billion transactions every day. Network proxy logs are rich but notoriously noisy, making too much volume a primary challenge. As we built our threat hunting service to analyze this data for Zscaler customers, we were confronted with a stark reality: you cannot dump all that telemetry into a SIEM, or costs will explode. Here, our threat hunting team has a unique advantage: infrastructure that does detection on our network data directly, but that solves just the first challenge. 

You also need the right data at the right time, and important details can be hidden in different places. You can’t just export CSVs and query them manually, or you won’t have the speed needed to catch threats early in the attack chain. To make this volume of data actionable, you need to programmatically enrich the relevant information with the exact context a hunter needs to understand activity quickly and ask further questions.

Consider a hunt involving a connection to GitHub. If you see telemetry showing an endpoint navigating to a GitHub repository or GitHub Pages, how do you know if it’s legitimate developer activity vs. an employee tricked into downloading a malicious script? It’s extremely difficult to tell if a repo is malicious or not, since threat actors can keep creating new ones or masking a repo’s purpose. Raw telemetry alone won't tell you, because the activity blends in perfectly as a trusted source, and you can’t rely on verifying against a list of known bad repos, because they’ll never be complete.

Rather than relying on allow or deny lists, which have to be maintained, we automatically pull deep repository-level context to enrich the raw URL. Because a standard URL doesn't show reputation, our backend automation programmatically scrapes the repository's metadata. Does the repo have a README file? How many contributors, watchers, forks, and stars does it have? By evaluating these signals, we can differentiate a major open-source library from a brand-new, zero-star repository stood up potentially by an adversary, turning thousands of noisy alerts into a tiny handful of suspicious anomalies. 

From there, we pull the surrounding five minutes of logs to build the narrative. In a recent campaign, looking at the context of other communications is what allowed our hunters to connect the dots. The raw telemetry showed a connection to a GitHub Page, but the enriched timeline revealed that immediately prior, the endpoint had visited a newly modified, Cloudflare-proxied domain. By automatically pairing the proxy's URL metadata with the endpoint's temporal log sequence, we were able to see the redirect chain in its entirety. Either of those communication steps, in isolation, would have looked like a routine developer connection – seeing them together revealed they were the final step in a malicious redirect sequence.

SANS-highlighted Blind Spots – How to Close those Gaps

The two most cited data blind spots in the SANS report are cloud and identity logs, infrastructures that often lack default logging configurations useful for security teams. Because our customers depend on us to hunt across those domains, we dedicate resources to engineering the data pipelines required to make them viable.

MFA fatigue is a good example of why that engineering work matters. Raw sign-in logs alone won't surface when an attacker is triggering endless MFA challenges, pressuring the user to approve access just to stop the barrage of alerts. To see what’s happening, you have to chain together a rapid succession of denied push events immediately followed by a successful session start. In Okta, that means correlating a spike of user.mfa.okta_verify.deny_push events with a subsequent user.session.start for the same user within a tight time window. That detection logic doesn't exist by default. It has to be deliberately built, and the underlying log source has to be flowing reliably before any of these detections will work.

The same principle applies on the cloud side. LLMjacking—where attackers abuse compromised cloud credentials to run inference workloads at the victim's expense—won't appear in standard alerting. Catching it requires querying GCP Cloud Audit Logs for sudden, high-volume spikes in API calls to services like aiplatform.googleapis.com originating from IPs outside your normal geographic baseline. Again, that logic has to be engineered, and the log source has to be configured to capture that data in the first place.

Setting the Standard: Normalizing Data Across Vendors

Once you temper noisy data sources and illuminate your blind spots, you run into the next major hurdle identified in the SANS report: "normalization and standards."

Even if your organization is actively consolidating its security stack around a platform vendor, all enterprises will have heterogeneous systems across the full IT landscape.  Our team lives this reality daily, since we hunt across multiple EDR platforms, cloud providers, and identity systems simultaneously.

The challenge of working with all this third-party data is that every vendor has its own nuances, APIs, and data structures. One EDR might log a critical field as process_command_line, while another simply calls it command_line. One vendor might heavily nest its JSON payloads, while another frequently updates its APIs, quietly unmapping critical fields. If your threat hunters have to manually translate their hypothesis into five different syntaxes for five different tools, the hunt fails to scale.

To illustrate how disparate telemetry can be, consider how the three major cloud providers log the exact same conceptual data. If a hunter wants to find "Who did what, from where?" manually, they have to map three entirely different schemas that may also have similar fields, as laid out in the following table.

To hunt efficiently, you have to decouple the hunter from the underlying vendor syntax. Our approach is to normalize messy, disparate data on the fly using a vendor-agnostic translation layer. We’ve built tools in-house that enable hunters to use a consistent query language that automatically translates their hunt into the native APIs of the respective EDR, identity, or cloud platform.

Furthermore, instead of forcing hunters to export and parse massive, flat CSV files, we load this normalized telemetry into structured, high-performance databases. This approach allows our team to run true relational log analysis using standard SQL. A hunter writes one consistent query, and the backend infrastructure handles the heavy lifting of mapping the fields, navigating the nested JSON, and returning unified results – regardless of which EDR, IDP, or cloud platform a customer is using.

Minding the Plumbing: Treating Pipeline Monitoring as a Core Hunting Function

No one gets into threat hunting to troubleshoot broken data pipelines. You want your analysts tracking down advanced adversaries, not spending hours figuring out that a query is returning zero results because a third-party vendor updated an API or table schemas overnight.

But as we scaled our own hunting operations, we quickly learned that normalizing your telemetry isn't a "set and forget" exercise. Because organizations rarely own all of their telemetry sources natively, pipelines are inherently volatile. Data formats evolve, and critical fields get unmapped without warning. You have to know your data flows well enough to understand exactly why a telemetry stream suddenly drops by 20 percent. Is it just a holiday weekend? Did a vendor roll out an undocumented change? Or is an adversary actively blocking telemetry at the firewall to blind your SOC?

If your threat hunters are the ones forced to pause their work and investigate telemetry drop-offs, they aren't hunting. We solved this bottleneck by decoupling the data maintenance from the actual hunt. By dedicating engineering resources specifically to monitoring connector health and pipeline integrity, we ensure data quality doesn't degrade. The best normalized queries in the world will quietly miss malicious activity if the underlying data stops flowing, so we built our operation around the reality that pipeline monitoring is just as critical as the hunt itself.

Seeing the Full Picture: Executing Cross-Domain Hunts at Scale

In threat hunting, domain silos are a serious challenge. For example, a single log showing a user browsing to a suspicious IP address doesn't mean much on its own. But if you can see that the user browsed to that IP, executed an unknown binary, and suddenly began authenticating to cloud resources from a new geographic location—that sequence suddenly becomes very interesting.

The problem is that if your environment is at all heterogenous, cross-domain correlation is very difficult by default. How do you efficiently correlate an Okta identity event with a Microsoft Defender for Endpoint alert, when the systems share no primary key? Historically, analysts have had to manually pull data from both systems, throw it into a table, and attempt to pivot across them. This manual approach doesn’t work if your goal is to catch fast-moving, sophisticated threats quickly. You want to reach a state where a hunter can execute one query that simultaneously hits endpoint, identity, cloud, and network telemetry—where a single query spans all domains and vendors. 

That’s why Zscaler is transitioning to using our Data Fabric for Security for threat hunting. The Data Fabric fuses first- and third-party telemetry into a single accessible layer – with deduplication, normalization, correlation, and enrichment – to build a security context graph. This correlated view will enable hunters to quickly trace an adversary's full path across network, endpoint, and cloud, regardless of the specific vendor data sources for any customer.

You Can Overcome Your Data Challenges

Solving your data problem will unlock your team's full capability, as even the most skilled hunters can't surface what the pipeline doesn't deliver. Data quality and the skills shortage—the two top challenges in the SANS survey—are actually less distinct problems than they seem. We’ll be talking more about tackling the skills issues in the next post in this series.

If you'd like to learn more about the top threats we're seeing so far in 2026 and how our hunting teams can operate as an extension of yours, tune into our upcoming webinar: 2026 Threat Trends: What 1,000s of MDR Investigations and Threat Hunts Reveal