What challenges did the client face, and why were they critical?
The security team was already using its own methods, automation and tools to validate detection mechanisms. More complex scenarios remained part of offensive security exercises, while some controls were already automated.
The challenge was with tests that needed to be repeated after platform updates, configuration changes or tuning of SIEM and EDR rules. The team wanted to verify that the expected telemetry and alerts were still being generated correctly after such changes.
The organisation therefore needed to identify specific scenarios suitable for regular regression testing and run them through a single, controlled process. In this case, the objective was not to move the entire security validation model into one platform, but to bring greater structure to a selected part of the existing workflow.
Key challenges
- revalidating detections after changes to SIEM, EDR and environment configuration,
- verifying expected telemetry and alerts,
- identifying scenarios suitable for regression testing,
- reducing reliance on ad hoc activities for regular testing,
- retaining the existing working model for more complex exercises.
What did we implement, and how did it work in practice?
IOC Simulator was used to run scenarios that the organisation wanted to repeat regularly after changes to the environment. Tests were mapped to MITRE ATT&CK and executed on Windows endpoints in a controlled environment.
Unlike broad Breach and Attack Simulation (BAS) and continuous security validation platforms, which may include extensive scenario libraries, multiple testing areas, numerous integrations and additional reporting layers, IOC Simulator was used for a more narrowly defined task: repeatable validation of selected SIEM and EDR detection mechanisms.
This allowed the team to focus on scenarios relevant to its own environment and rerun them after configuration changes or rule tuning. There was no need to move the entire Purple Team process into a new tool, while more complex exercises could remain within the existing way of working.
After each simulation, the auto-restore mechanism rolled back changes created during the test, such as files, directories or registry keys, preparing the endpoint for the next run.
What were the results, and what changed in day-to-day operations?
IOC Simulator brought more structure to the part of the Purple Team process that required regular repetition of the same scenarios. After a system update, configuration change or rule adjustment, the team could rerun a previously defined test and verify whether the detection mechanisms still responded as expected.
For the selected regression scenarios, the time required to rerun and verify the tests was reduced by approximately 60%, compared with the previous process.
Repeatable execution made it easier to spot situations where a test had previously generated the expected telemetry or alert but produced a different result after a change. Such a difference could indicate a detection gap or detection drift and provide a starting point for further analysis and another test after the relevant adjustments had been made.
More extensive offensive security exercises remained part of the existing Purple Team process. IOC Simulator focused on the scenarios that needed to be repeated regularly and in a comparable way.
Results after implementation
- approximately 60% less time required to rerun and verify selected regression scenarios,
- a structured process for selected regression tests,
- the ability to revalidate detections after changes to the environment,
- a specialised approach instead of deploying a broad platform across multiple validation areas,
- easier identification of potential detection gaps and detection drift,
- the ability to retain the existing Purple Team workflow for more complex exercises.
See how IOC Simulator can complement your organisation’s detection validation process: IOC Simulator | Continuous Threat Validation & DORA Compliance