Security engagements are a hot mess

How can organizations make security engagements a less jarring experience?

Anyone that has ever been a part of a red team (or purple team) engagement will probably tell you the same thing - it's chaotic.

Maybe a particular tool fails to work as anticipated and the exercise has to "pause" so that it can be fixed. Or organizational processes begin to fail when they are exposed to the tailored threat. Or worse still, both teams enter "white card hell" where actions have to be hand waved in order for their objectives to be met.

Sound familiar?

But we cannot accept this as the status quo. Just like physical training, training your people the wrong way is negative training.

No APT is going to pause so you can tune your sensor.

No APT is going to wait while you adjust your organizational processes.

No APT is going accept your Plan of Action and Milestones (POA&M) explaining that the NEW tool will pick up their behavior, so they need to wait until you have it.

An APT is going to press through their campaign and your team needs to be prepared to engage the threat in real time.

In summary: 

Security engagements are extremely jarring, how can organizations make them less jarring experiences?

The root cause

What actually causes engagements to be such a pain for majority of organizations?

  1. Slow process change. Process change is difficult for small organizations and orders of magnitude more difficult for larger organizations. Running a security engagement involves many moving pieces and many unpredictable outcomes. Being slow to adapt and unsure what to do when an unpredictable outcome is reached can cause chaos.

  2. Technical limitations. Whether due to experience with a tool or using the wrong tools in general, security engagements can hit chaos quickly because it appears there is no way forward - whether on the offensive or defensive sides.

  3. Negative training. Going into a security engagement, if the rules are not well established around when to follow "normal" process and when to white-card (cheat) the process, it will cause chaos. Those participating in the engagement need to know what to do and when and it should center around realism, not negative training.

  4. High cost. Making changes can be extremely cost prohibitive, particularly for large organizations. So having an extremely expensive tool be discovered as wholly ineffective for protecting your environment, can be a scary wake up call and the instinct will be to try and use the tool regardless to justify the cost. Once you've dug yourself a hole, digging more will only get you in deeper.

Core Problems and Solutions

Let’s dive into what makes security engagements such jarring experiences and how organizations of any size can make them less jarring.

Approach security as Operations not Compliance

Let's get one thing straight: security is not compliance and compliance is not security.

Compliance is certainly part of the overall security posture for an organization but ultimately being PCI DSS compliant or achieving that ISO 27K accreditation does NOT mean you have a secure and resilient organization.

When approaching our resilient security posture we need to speak in terms of defense in depth, redundancy, and leveraging an "OODA" loop (Observe - Orient - Decide - Act) to handle incidents as they arise.

Let’s assume a penetration tester has made their way into our environment and compromised our crown jewels meeting their main objective. The tester reaches out to their point of contact, which just happens to be the SOC manager. It is at this point the SOC manager realizes that no alerts were created by the compromise. It’s all hands on deck as multiple SOC analysts investigate the host.

The analysts quickly realize the EDR agents are outdated and not properly communicating events outbound to our logging solution. Our organization is in big trouble now because a decision was made last quarter to turn off auditd and sysmon after the purchase of the new EDR solution and no other security logs are being ingested by the logging pipeline. Our organization has no backup logs and no contingency plan when EDR fails and has failed to properly monitor and respond to a simulated threat.

What not to do: 

Pause the engagement and fix the EDR. A real threat is not going to "stop" or "pause" so training your team this way is effectively negative training - you are reinforcing bad process and bad procedures while also creating an atmosphere of security being something that can be "on or off".

An engagement is designed to show you what happens when a threat actor attempts to breach your organization. Teams need to fight through tooling failures, process failures and organizational issues to respond to a threat. Think of it this way: if your organization doesn't live and breath the "OODA"-loop for security then you're not preparing for a real threat.

What to do:

Have a team investigate the root cause and fix the EDR solution while the red team is actively exploiting the misconfigured tool.

As the red team continues executing their engagement plan, have the blue team solve the problem at hand by answering the very real question: "How do we fight through a contested, degraded environment?"

In response to the above scenario, defenders should step through their OODA loop for tackle that problem: Observe what system is offline and what visibility was lost. Orient to how you could potentially detect threats via alternate detection mechanisms. Decide on which alternate detection strategy to run with. Act on that plan. Rinse and repeat.

Continuous process evaluation, not monolithic assessment

People, process, and technology change - often times at an extremely rapid pace. Particularly technology. Depending on the organization, different methodologies are applied to optimize how change is handled. Engineering teams lean on agile Software Development Life Cycles (SDLC) to drive iterative process of understanding requirements, planning, build, testing and deploying.

Operations (i.e. security teams) need to follow suit.

There is a reason that the military revolves around exercises; it helps uncover potential process failures while training the team on how to real world contingencies as they emerge.

Security teams have the advantage of time and knowledge of their environments but often fail to act on those strengths and find themselves “waiting” for the next attacker or penetration test to discover their weaknesses rather than proactively challenging their defenses. While red team exercises and penetration tests are effective at uncovering weaknesses, reliance on tests that are performed annually or quarterly don't help teams stay current on process or emergent threats.

What not to do: 

Run a couple red/purple/etc team security assessments per year without serious attention to what you hope to accomplish. Measuring this doesn't stop at what threat actor you would like to emulate.

What to do:

Put an emphasis on following your established security process, which should have room for the unexpected.

Transition to continuous testing and evaluation using automation tools. Run autonomous engagements daily using the TTPs from the last red team engagement. Once your team is able to reliably and robustly detect those techniques, bring in a red team to do a thorough assessment. Move the TTPs from that engagement to your automation suite, run those tests daily until your team can reliably detect those techniques.

Incremental progress, not large changes

Every time a new major compromise or hack occurs, cyber security companies line up to sell CISO's and CEO's a "promising new security product that will solve all of their problems." Today's flavor? Ransomware attacks. Corporate executives want an "easy button" solution, which means buying an expensive new product that gets partially installed and partially configured.

This is not a sustainable (let alone effective) approach to improving organizational security. But what is the right approach?

Lets look back at the SDLC for how an already established and effective process handles product deployment. Often times, a beta version will be pushed out to a small number of users across the enterprise (or customer base) so that the changes can be validated and confirmed effective.

In the military space, enlisted personnel and officers are encouraged to find process optimizations and test out new approaches to getting the mission accomplished. The goal is to do more with less while improving operations.

What not to do: 

Buy the next "hot new security product" or sink huge dollar amounts into a single "fix all" solution (because it doesn't exist). Do not lock your organization into security decisions from which there is no real way to pivot to other solutions.

What to do:

Use threat informed red and purple team engagements to perform a Crown Jewels analysis; understand what your organizations Centers of Gravity are (the assets which are absolutely critical to accomplishing your core business mission). Once you have an understanding of the threats relevant to your organization and the Crown Jewels they would target, begin to build a defense in depth strategy focused on defending those assets.

If you don't know what you need to protect, you won't be able to protect it.

Make a single, reversible change, and perform the threat informed engagement again to determine whether the change was effective at detecting and/or preventing the adversary's kill chain.

Start with process changes and tweaks to existing technology; do not immediately jump to buying a new product. Only make that purchase decision when you have data to inform the decision (i.e. you cannot detect a component of a kill chain due to lack of technological capacity).


Hopefully this has helped guide your thinking and provided insights into how to make security less of a jarring experience for your organization. The goal is to normalize the security engagement process and make operations a normal part of your day-to-day.

A team that is trained to to handle incidents properly as part of the "status quo" will be able to perform those same response actions in the face of a real threat. In other words, your security assessments don't need to be a hot mess. Instead, they can be an eyes-wide-open exercise to uncover your vulnerabilities. Organized chaos.