Testing Whether Your Alerts Actually Fire

Everybody deploys detection. Almost nobody verifies it triggers.

The result is a dashboard full of rules, some of which stopped working months ago because a log source changed, a field was renamed, or an agent quietly stopped reporting. You find out during an incident, which is the worst possible time.

The idea

Atomic Red Team is a library of small, self-contained tests mapped to attack techniques. Each one performs a single, safe action that a real attack would perform, so you can check whether your monitoring notices.

Not an exploit framework — a set of deliberately boring actions with documented cleanup.

How a test cycle works

1. Pick a technique relevant to what you actually run — credential access, persistence, a suspicious child process
2. Read the test. Understand exactly what it does before running it
3. Run it in a controlled environment
4. Check whether your alert fired, and how long it took to be visible
5. Run the cleanup
6. Fix whatever did not fire

Step 4 is the entire point. Not "did the test run" but "did anyone find out."

What I have found doing this

Every time I have run this exercise with a client, something was broken. The pattern is consistent:

• An agent not installed on the newest server, which nobody noticed because absence produces no alerts
• A rule matching a field name that changed in a platform update
• Alerts firing correctly into a channel nobody had been watching since the person who set it up left
• A twenty-minute delay between event and alert, when the assumed response window was two minutes

None of those are exotic. All of them mean the detection was decorative.

The safety rules

Only on systems you are authorised to test. Written scope, always.

Never blind on production. A test environment first, and if production testing is genuinely required, agree it in advance with everyone who might respond to the alert.

Tell the responders — or deliberately do not, and tell their manager. An unannounced test that triggers a real incident response at 2am damages trust badly.

Always run the cleanup. Persistence tests that leave persistence behind are how a test becomes a finding in someone else's audit.

The value

Half a day of this produces a more accurate picture of your detection capability than any amount of reviewing rule configuration. It converts an assumption into a measurement.

Resources

• Repo: redcanaryco/atomic-red-team
• Technique reference: attack.mitre.org
• Video walkthroughs: YouTube: Atomic Red Team tutorial
• Write the rules first: Sigma