Building an Effective SIEM: Detection Engineering

Plenty of organisations buy a Security Information and Event Management platform, point every log source at it, and declare themselves covered. Months later the SOC is drowning in alerts, most of them noise, and real intrusions slip through unnoticed. The platform was never the problem. A SIEM is a pipeline and a query engine; the value lives in the engineering that shapes raw logs into trustworthy detections. Treating that work as a deliberate practice — with a data pipeline, a schema, version-controlled rules, and a tuning loop — is what separates a SIEM that protects an organisation from one that merely bills for storage.

The log pipeline

Everything starts with getting the right data in reliably. Sources span the estate: endpoint detection agents, authentication and directory services, firewalls and proxies, cloud provider audit logs, DNS, and application logs. Two decisions dominate here. First, collection — how logs are shipped, buffered against outages, and delivered without loss. Second, and more strategic, coverage: rather than ingesting everything and paying to store noise, mature teams choose sources deliberately against the threats they need to detect. A useful anchor is the MITRE ATT&CK framework — asking, for each technique you care about, which data source would reveal it, and confirming that source is actually flowing.

Parsing and normalization

Raw logs arrive in a chaos of formats — syslog, JSON, key-value, vendor-specific blobs — each naming the same concept differently. One product calls it src_ip, another source.address, a third ClientIP. Parsing extracts the fields; normalization maps them onto a single common schema so that "source IP address" means one thing everywhere. This is the highest-leverage, least glamorous work in the whole pipeline.

Why a schema changes everything

Once every source populates a shared schema — the industry has converged on models like the Elastic Common Schema and the Open Cybersecurity Schema Framework — a detection can be written once and apply across every product that feeds that field. Without normalization, you write and maintain a separate rule for every vendor's log format, and analysts waste incidents translating field names in their heads. A good schema is what makes detections portable and investigations fast.

Every hour spent on normalization is repaid many times over in detections you only have to write once and investigations you no longer have to translate.

Correlation and detection rules

With clean, normalized data, you can write detections. These range from simple matches — a single log line that is inherently suspicious — to correlation rules that combine events across sources and time: a failed-login burst followed by a success followed by access to a sensitive share tells a story no single event can. Good detections are grounded in adversary behaviour rather than fragile indicators; a rule keyed to a specific malware hash dies the moment the attacker recompiles, while a rule keyed to the technique survives. Mapping your rule set to ATT&CK also exposes your blind spots: it makes visible which techniques you can detect and which you cannot.

Detection-as-code

The practice that has most transformed this discipline is treating detections like software. Rules live in version control, are written in a structured, portable format, go through peer review before deployment, and are tested against known-good and known-bad data so a change cannot silently break coverage. Detection-as-code brings the whole engineering toolkit — history, rollback, code review, CI testing — to security content. It turns a pile of undocumented rules that only one person understands into a maintainable, auditable asset the whole team can evolve with confidence.

Taming alert fatigue

The failure mode that quietly kills SIEM programmes is alert fatigue. When analysts face hundreds of low-quality alerts a day, they start ignoring them, and the one real detection buried in the flood gets closed without a look. The fix is not more alerts — it is fewer, better ones, driven by the feedback loop in Figure 1. Every alert a SOC analyst triages produces a verdict: true positive, false positive, or benign-true-positive. That verdict must flow back into the rules. A detection firing mostly false positives gets tuned — its logic tightened, known-good activity excluded, its threshold adjusted — or retired. This closed loop, run continuously, is what keeps signal high and the SOC's attention where it belongs.

Measuring what matters

To tune deliberately you have to measure. Track each rule's true-positive and false-positive rates, the volume it generates, and whether it has ever caught something real. A rule that has fired ten thousand times and never once mattered is not coverage — it is noise wearing a badge, and honest metrics give you the license to remove it.