Desk research, including ours, can narrow a list. It cannot show how a tool behaves on your data. That is the job of a proof of concept. This post sets out a plan that makes the results comparable across tools.
Before it starts
- Map your data. List your sources, retention, access owners and query languages. Our lesson on mapping your data before a shortlist walks through it.
- Agree access. Grant read-only access, scoped to the sources the trial needs. Record the date and time each tool received access; you will need it later.
- Choose the hunts. Pick three to five hunts that matter to you. Include at least one that starts from a recent public advisory and at least one from your own hypothesis. Write each as a hypothesis with a data source and a window; see writing a hunt hypothesis.
- Write the success criteria. Decide in advance what counts as a pass for each hunt and for the trial as a whole. Criteria written after the results arrive tend to favor whichever tool looked best on the day.
- Fix the dates. A fixed window, such as a few weeks, keeps every vendor on the same clock.
While it runs
Run the same hunts on every tool, over the same data window, and record the same things.
- Time to first result. From access granted to the first completed hunt. Vendors make claims about this. Mars Security's press release says it deploys in hours; Artemis Security says connectors go live in under an hour, with real cases inside 48 hours. A trial is the place to see those times on your own setup.
- Evidence per finding. For every result a tool reports, check what it shows you: the queries, the matching records, the sources involved. A finding you cannot trace back to data is a lead, not a result.
- Negatives. When a hunt finds nothing, check whether the tool says what it searched and over which window. A silent empty result is not the same as a documented negative.
- Rule testing. If a hunt becomes a detection, watch how the tool tests it before deployment. Mars describes backtesting each rule on 30 days of the customer's data; Nebulock's docs say every rule must pass a retrohunt before deployment. Ask to see the test output, not only the pass.
- Version control and review. Check whether rule changes have a history, an author and a way to revert, and whether they fit the review process you already use.
- Analyst time. Log the hours your team spent per tool on setup, tuning and reviewing results. This is where claims about saved effort can be checked.
- Data platform load. Ask your platform owners what query volume each tool generated.
Keep a hunt log entry for every hunt on every tool. At the end, the log is your evidence.
Scoring the result
Use the same seven criteria as the shortlist, but score what you observed rather than what the vendor pages say:
- Hunt automation: did the hunts run without being started by hand?
- Working across your data: did the tool reach your top sources, and how?
- Analyst effort saved: how many hours did your team spend compared with doing the hunts manually?
- ATT&CK coverage evidence: could the tool show which techniques its hunts and rules address?
- Detection-as-code workflow: were rules tested, versioned and reviewable?
- Deployment: how long from access to first result?
- Pricing: did you receive a price on a basis you can compare?
Weight them the way you did for the shortlist. If you have not set weights yet, the lesson on setting your own weights and the calculator help. Keeping the same weights for desk research and trial makes the two comparable.
Closing it out
Write a one-page decision note: the hunts, the criteria, the scores, the evidence for each score and the open questions. Share the log with each vendor and ask them to correct anything factual before the decision. Note anything the trial could not test, such as behavior at full data volume, and decide how you will check it after purchase.
Common mistakes
- Letting each vendor choose its own demo hunts. The results will not compare.
- Running on a sample of data that is cleaner than production.
- Judging on the number of findings rather than the evidence behind them.
- Leaving access in place after the trial ends. Revoke it on the last day.
Sources
Related
Editorial assessment · From vendor product pages, docs and announcements · Updated 2026-10-18