Collection & Sourcing

Collection is the least glamorous part of intelligence and the one most often solved by purchase. The predictable result is a programme paying for several feeds with heavy overlap, none of which have changed a decision.

Collection should be driven by requirements, not availability. The question is never “what can we get” but “what decision are we trying to inform, and what would we need to observe to inform it”.

Evaluating a feed

Volume is the worst possible measure and the one most prominently advertised. Better questions, in roughly the order they disqualify things:

1. Overlap. How much does this add over what you already have? Measure it before renewal — take a month of indicators and compute the intersection. Feeds sold as differentiated are frequently the same upstream data reformatted.
2. Timeliness relative to your response. An indicator that arrives faster than you can act on it is worth no more than one that arrives at your response speed.
3. Actionability. Can it drive a detection, an enrichment, or a decision? An IP with no context is a lookup, not intelligence.
4. Attribution of the observation. Was this seen, or inferred? Reported once, or corroborated? This is source grading, applied to a supplier.
5. False positive cost. A feed that blocks a CDN range once will cost more than it ever returned.

Indicator feeds age badly — IPs and domains rotate fast. Behavioural content ages slowly. Weight accordingly: the durable value is usually in the reporting, not the IOC list.

Internal telemetry is the underrated source

The most relevant intelligence about your environment is generated by your environment, and it is usually the least mined. Nothing in a commercial feed is as pertinent as what already targeted you.

Sources worth treating as collection rather than exhaust:

• Blocked and failed activity — what was attempted and stopped is a statement about who is interested in you.
• Incident output — investigations produce indicators and behaviours with a provenance no vendor can match.
• Pentest and red team findings — reachability facts about your own estate.
• Authentication telemetry — identity is where most intrusions start, so it’s where the early signal is.

This has an obvious statistical trap: you can only observe what you instrumented, so internal telemetry systematically under-represents your blind spots. That is sampling bias, and it is the reason internal collection complements external reporting rather than replacing it.

OSINT tradecraft and its OPSEC

Open-source collection is genuinely powerful and carries operational risk that gets underestimated, because the activity feels passive when it isn’t.

Interacting with adversary infrastructure — resolving a domain, fetching a page, submitting a sample to a public sandbox — can notify the operator. Public malware sandboxes are searchable, so an uploaded sample can disclose that you are investigating, and sometimes disclose the victim through file contents or filenames.

Practical discipline: separate research infrastructure from corporate, know which of your tools are public before you use them, and prefer passive sources (passive DNS, historical records) over live interaction during an active intrusion. Attribution of the collector is a real risk, not a theoretical one.

Structured formats

STIX is a data model for threat intelligence; TAXII is the protocol for moving it. They are verbose, and that verbosity buys machine-readable relationships — this indicator belongs to this campaign, attributed to this actor, with this confidence — which flat lists cannot express.

MISP is the pragmatic middle ground most teams actually reach for: sharing communities, correlation across events, and a workable data model without the full STIX ceremony.

The honest position: structured formats earn their complexity when you are exchanging at scale or need relationships expressed. For a single team consuming a handful of feeds, they are frequently ceremony around a list of indicators, and a simpler pipeline serves better. Adopt them when the sharing requirement is real.

Deduplication and enrichment

Between overlapping feeds, dedupe on the observable and keep the union of context — an IP seen by three sources with three different attributions is more interesting than the same IP three times.

Enrichment (ASN, geolocation, passive DNS, prior sightings) is what turns an observable into something an analyst can judge, and it is the most obvious automation target in the whole pillar: high volume, well-defined, and low judgement per item. Keep the judgement — is this relevant to us — with a human.

Retirement

Indicators need expiry as much as ingestion. An IOC list that only grows becomes a false-positive generator and eventually gets ignored wholesale, which quietly removes the value of the good entries too. Set a decay policy by indicator type, and treat a feed with no expiry semantics as incomplete.

Where this connects

Collection feeds detection engineering most directly — the detection backlog is downstream of what you can observe. Source grading and confidence language from report writing are how collection quality gets carried into the product rather than lost at ingest. And forensics is the highest- provenance collection source you have, because it is the one where you know exactly what was seen and how.