Get to know the similarities & differences of the leading open-source purple tools
This post will take a detailed look at the main open-source command-and-control platforms. We will zero in on Prelude Operator and MITRE Caldera, as the leading open-source platforms in this space. We will evaluate the pros and cons of each in their respective areas. As the designers of both systems, we feel uniquely qualified - and compelled - to write this post.
Every year, more open-source C2 tools are released on GitHub. There are 3 reasons for this. Going through tools on a feature-by-feature basis is an effort in futility as (by definition) open-source software can be modified and gain feature parity, if a developer were so motivated.
Instead, we prefer to look at tools through their immutable characteristics. These are the fundamental aspects and designs, which are baked into the intended use of the tool. Immutable characteristics cannot be changed through a simple feature change but would instead require a ground-up rewrite. This is what should drive your decision to use one tool over another.
Is there a purple team C2 you’d like us to do a similar review about? Reach out to us and tell us which one.
Let’s start with definitions.
Caldera is a MITRE command-and-control center, which allows you to launch autonomous adversary emulation exercises in a computer network. Operator is a C2 automation platform designed to make advanced (red team) security more accessible.
So which should you use? If you’re already using one, does it make sense to switch?
Our bottom lines up front:
David: Caldera was designed as a research project for automated decision-making whereas Operator was designed to incorporate this research and extend it to be operational. If you want to do research into decision-making, Caldera is for you, if you want to do security assessments, Operator is your choice.
Alex: The Federal Acquisition Regulation (FAR) limits FFRDC activities to prohibit them from competing with industry; their role is to solve hard problems and transition those solutions to Government and Industry. Caldera is a perfect example of an FFRDC pushing research to the point where it essentially created a Breach and Simulation (BAS) industry that can continue to move autonomous red teaming forward. Operator is our attempt to accomplish that objective while keeping the open-source/community spirit of Caldera alive.
Interested in diving a bit deeper? In the rest of this post, we'll walk through the architecture, use-cases & features, design & usability and the support of each platform.
We built Caldera as an HTTP server. It was designed this way purely to supply an easy transition between the research-focused first version of the framework - which was already written as a Python server - and the more usable second version. The decision was not, 'what is best for a platform like this?' but instead, 'what is easiest to get from point A to B?'.
David: I always regretted the complexity of the HTTP server I built into Caldera. While simple for someone with a programming background, I overestimated the skill set required to install and manage Caldera. In conducting several training workshops on the framework, I found we were losing the entire first day just getting it installed on people's machines!
The HTTP server was designed to be deployed on a laptop or single Linux server. It can run on Windows but is pretty shaky (i.e., we only had 1 developer out of the entire team using Windows). The server itself supports adding multiple users to the login... but this was a little misleading in retrospect: while you can add multiple users to the server, if any of them changes something (say a TTP or adversary profile) the change occurs globally for all users. In addition, all users have the same privileges, visibility and access. So in effect, Caldera is a single-user application passing itself as a multi-user one.
Alex: We weren't trying to be tricky, we just wanted a basic login page to prevent misuse. We always intended Caldera to be a single-user application.
Steps to deploy Caldera:
Install the Caldera HTTP server on your laptop or a VM in your network
Deploy 54ndc47 agents on machines you want to test and point them at your Caldera server
Login to the HTTP web portal, build & launch an adversary
We built Operator as a desktop application. It was such an important decision, we wrote an entire post on why we did this!
David: Remove any preconceived notions about SAAS: a desktop application is actually the same thing as a web server. The main difference is that a desktop app is integrated with your computer, allowing you to have a more personal feel. But in reality, it (in our case) is running a Chrome browser and can be thought of as a detached website.
Operator was designed to be installed on an individual laptop, desktop or server instance - of any operating system. You install it the same way as any other application: double-click the installation file. We wanted to move away from requiring users to understand Python, PIP and other programming nuances between operating systems in order to run the platform.
Alex: Even though Caldera was fairly simple to install (as technical tools go), we usually still had large numbers of students in training courses struggle to get it installed on their local system. As the platform added dependencies and complexity, this problem got worse. A pre-compiled, OS specific binary application solves that problem.
Similar to Caldera, Operator has a default Golang agent, called Pneuma. We liked how we designed 54ndc47 so much, we kept with the same language and general design. However, we felt like 54ndc47, over time, got too large and unwieldy so we opted to simplify and make Pneuma a streamlined version.
David: Pneuma is designed to be an example agent so you can design your own. For that reason, we believe in extreme simplicity to highlight clearly how you can write custom agents. In 2 years overseeing Caldera, we never had an open-source agent written for the framework. In just 2 months, we had one written for Operator. We attribute this to a phenomenal community but also the simplicity of the Prelude platform.
Steps to deploy Operator:
Install Operator on your laptop or a VM in your network
Deploy Pneuma agents on machines you want to test and point them at your Operator host
Open Operator to build & launch an adversary
The top-level deployment steps are nearly identical between Caldera and Operator, but actually installing and using the application is very different. The biggest difference is that one is an HTTP server and one is desktop application/server. As such, Operator is significantly easier to install and start up.
For either case, if you want to receive beacons from remote computers, you will either need to open a firewall port on the host or deploy a redirector which forwards traffic to it. The latter is highly recommended to run a safe test.
Use Cases & Features
As an adversary emulation framework, Caldera is designed to launch autonomous red team exercises against a given range of agents. It was birthed as a research project out of MITRE and has a strong bias toward research over real-world. As such, the framework is naturally missing many operational features, such as encryption (both in-transit and at rest), non-shell TTPs, modular implants, modern protocols, etc.
Alex: This is by design as an FFRDC is not allowed to compete with private sector nor build things that can be built by private sector; none of those features are novel or pushing research, which means we couldn't build them.
The basic gist is, Caldera wasn't designed for real world operations but instead as a research project to prove out autonomous decision making.
By contrast, Operator was built to allows users to incorporate the research of projects like Caldera and make them operational.
Operator covers all the same basic use-cases of autonomous red teaming that Caldera covers - and quite a bit more. We built in more protocols for agents (such as gRPC and two-way UDP), made the installation and usability easier, ensured the platform was built to handle large scale attacks and remain realistic, started using modular malware, non-shell execution implementations, and added configurable encryption options in-transit and at rest. And there’s a lot more to come, as Operator ships new features weekly.
Aside from autonomous red team as a use-case, Operator is also designed to:
Assist blue teams in a SOC environment determine which analytics are (and are not) working
Train red, blue and InfoSec teams on a continual basis, using an ongoing training program built into the application itself
In terms of features, Operator and Caldera contain the same general set of capabilities. However, Operator additionally contains some big improvements, including:
You do not need to manage plugins & procedures yourself. Operator automatically pulls in new plugins and updates them for you. It also watches repositories such as Community, Stockpile and Atomic Red Canary and notifies you of new procedures when they're merged.
An entire new Integrated Development Environment (IDE) for building attacks.
Built-in integration with Splunk, Slack, AWS, Google Cloud and other security products.
Security recommendations which give you actionable steps post-assessment.
Upgrading is seamless, with 1-click options built into the app.
Use-case & Features conclusion:
Caldera was designed as a research project whereas Operator is a commercial product designed as an operational platform. This distinction is evident in the features available in both systems. Think of Caldera as a subset of the Operator features and as Operator expands, the differences will continue to grow.
Usability & Design
We're not designers but we designed the Caldera UI and UX flows in-house. We think we did an OK job but we know it would have been better with a full-time, dedicated engineer with this skill.
Alex: This is not by accident. The goal of our project was to push research, not worry about UI/UX. Majority of the UI/UX tweaks were done on our personal time out of necessity.
The usability of Caldera is good but there are many major to minor flaws/bugs the team found along the way. Nothing wrong with this: there's a strong community of people who help fix bugs as they're noted but given the design of the app, these won't go away anytime soon.
By contrast, Operator is run by Prelude, a venture-funded, product-focused company. This has allowed us to incorporate time and resources into studying, designing and testing the UI and UX components. We think it shows when you take the product for a spin.
Everything in Operator is designed to lower the friction of using advanced security systems.
Usability & Design conclusion:
Because they come from different beginnings, Caldera and Operator have taken different directions related to UI/UX. Caldera is not focused on ease-of-use and creating a "product" whereas Operator is focused heavily on making the tool as accessible to as possible to any technology worker.
MITRE cannot support non-government customers or individuals who want to use the framework. It is intended to be a "use at your own discretion" type of tool. There is a key paragraph extracted from MITRE's public documentation explaining this fairly clearly:
“To ensure the highest levels of objectivity, the FAR limits FFRDC activities. For example, the FAR prohibits FFRDCs from manufacturing products, competing with industry, or working for commercial companies. These restrictions enable industry and government to confidently provide FFRDCs with sensitive or proprietary information without fear of improper use or disclosure.”
Even if we had wanted to push out Caldera to commercial organizations and provide new capabilities and support, we were legally not allowed to do so. We were however allowed to publish the code to the open-source world and fix bugs that were found from the open-source community.
This also meant we were not allowed to build a commercial community around the tool which made it hard to gather feedback from the community and incorporate it into the product.
As a private company, Prelude has set up several 24/7 support channels to help individuals in the community as well as customers using the platform. These include private Slack channels for any client and a community Discord server where anyone can interface with the team.
David: Being able to interface and support/grow a community of people around advanced red team tech like Operator is exactly what motivated me to expand from MITRE Caldera to building Operator.
Additionally, Prelude runs a 5-week free/open-source training program for anyone who wants to learn intro red teaming concepts or just how to use Operator. Send us a request at firstname.lastname@example.org if you're interested in joining.
As a product company, Prelude is dedicated to customer support at all levels. As a government contractor, MITRE is only able to support government agencies, if the funding is available.
Hopefully, this post has given you a good overview of Caldera and Operator, how they both came to be and what purposes each is trying to solve in the security space.
Here is the nutshell version:
Caldera is great for researching autonomous decision-making
Operator is great for running real-world security assessments from the red or blue perspective
Caldera is fundamentally limited by constraints that venture-funded, private companies do not have with respect to productionizing features and working with commercial entities.
As the builders of both Caldera and Operator, we hope this has given clarity on where and when you should use either tool. Reach out to chat anytime if you'd like us to analyze your particular situation for a personal assessment.