Blog

Backtesting and retrohunts: testing a detection before it goes live

Run the rule against the past before you trust it with the future.

By Threat Hunting Tools staff · 17 September 2026 · 4 min read

What is a backtest?

A backtest runs a detection rule over data that has already been collected, as if the rule had been live during that period. The output is a list of everything the rule would have matched. The terms overlap: "backtest" is common in detection engineering, and "retrohunt" is often used for sending a hunt query or rule back through past telemetry. In both cases the point is the same: learn how a rule behaves before an analyst has to live with its alerts.

Two vendors in this guide describe the step in their own material. Mars Security's September 2026 press release states that each rule is backtested against 30 days of the customer's own data before it goes live, and that indicators are scored against false-positive history. Nebulock's detection engineering documentation states: "Every detection rule must pass a Retrohunt before deployment." Nebulock also describes immutable version history for rules, with compare and revert.

What does a backtest tell you?

Read the results in three passes.

Volume. How many matches would the rule have produced per day? A rule that would have fired hundreds of times a day on a quiet network will be muted within a week, whatever its logic.
Content. Look at what matched. Are the hits the behavior you meant, or something that shares its shape, such as an admin script, a backup job or a scanner?
Expected hits. If you know of real or simulated activity inside the window, did the rule find it? If you ran a test of the technique in a lab, this is where it should show up.

Each pass leads to a change: tighten a condition, add an exclusion with a written reason, or drop the rule. Then run the backtest again. The loop is fast because the data already exists.

What can a backtest not tell you?

Whether the rule would catch an attack that did not happen in the window. Zero matches on 30 days of data means the behavior did not appear, or the rule is wrong. It does not show which.
Whether the data will stay the same. A renamed field or a new log format can break a rule that passed its backtest. That is detection drift, and it needs monitoring after deployment.
Whether the window is long enough. Monthly jobs, quarterly processes and annual audits may never appear in a short window, and they are a common source of noise later.
Whether all the data was there. If one source has a shorter retention period than the others, the rule was only tested on part of the picture.

Where does the test fit in a detection workflow?

Backtesting sits between writing a rule and deploying it. In a detection-as-code workflow, it is one of the checks a change has to pass, along with a review by a second person. Vega's public pages describe detections that live in source control, where every change has a diff, an author and an approver. Nebulock describes the retrohunt as a gate: a rule that has not passed one does not deploy. Mars describes the backtest as part of turning intelligence into a detection, so a rule arrives with its test results. The scoring method explains how we weigh testing and version control under the detection-as-code criterion.

The order matters less than the rule that nothing ships untested. A useful habit is to store the backtest summary with the rule: the window, the number of matches, and the reason for each exclusion. When someone asks later why a rule ignores a particular service account, the answer is on record.

How long should the window be?

There is no single right figure. A longer window catches more periodic activity and more real history, but costs more to query and takes longer to review. Pick a window that covers at least one full cycle of the business processes the rule touches, and check that every data source the rule reads has retention covering that window. If the answer changes by source, note it in the backtest summary.

Does a backtest replace testing with simulated attacks?

No. The two answer different questions. A backtest asks how the rule behaves on your normal data. A simulated attack asks whether the rule fires when the behavior happens. Open resources such as the Sigma rule repository can help with the logic side, since Sigma rules are written in a generic format that converters such as pySigma translate into a platform's query language. Testing still has to happen on your data.

A short checklist

The rule has a stated purpose and an ATT&CK technique.
It has been run against a defined window of past data.
The match volume is written down and acceptable.
Every exclusion has a reason.
Known or simulated activity was found, or its absence is explained.
The rule and its test summary are versioned together.

For the step before this one, see from hunt to detection rule. For how the tools compare on testing and version control, see the ranking and the Mars Security and Nebulock reviews.

Sources

Editorial assessment · From vendor product pages, docs and announcements · Updated 2026-10-18