A history lesson
From Caldera to Operator and the milestones in between
As the designer/builder of the MITRE Caldera (v2) framework and the original author of Prelude Operator, I feel uniquely qualified to write this piece, which takes you on a journey of how the Caldera system came to be and how it sparked the open-source adversary emulation industry.
Coming onto the scene in 2015, the Caldera project was born through an idea in the MITRE Innovation Program (MIP), by most accounts kicking off the now explosive breach and simulation (BAS) industry.
MITRE, a non-profit operating several Federally Funded Research and Development Centers (FFRDC), is a government-funded organization that acts as a research arm of the U.S. military and various agencies. Every year MITRE opens up an idea market to employees, who can submit any and every idea in categories ranging from cyber security to health care all in the name of public interest. In most years, hundreds of research proposals are evaluated and only dozens are accepted, each given a 1-year runway with funding for a small team.
Most projects are intended to be future-looking, building technology that doesn’t yet exist but is predicted to be important 3, 5 or 10 years in the future.
Version #1: The prototype
In 2015’s idea market, Caldera was proposed and starting around 2016, the development of the plan took shape.
Caldera was originally designed to research machine learning in a niche sector known as automated planning. The idea was simple to understand yet complex in implementation: could MITRE develop a self-learning application that could mimic the way adversaries make decisions once inside a target computer system?
The program would need to balance stealth (act of not getting caught) with the effectiveness of each attack it executed. During this period, the ATT&CK matrix was also gaining prevalence, so Caldera was deemed a worthy counterpart for it to be developed against.
Automated planning, or A.I. planning as it is often referred to, for Caldera meant creating intensely complex pre-and-post condition logic that creates realized strategies (action sequences) for execution by remote access trojans (agents). The planning would take place in a non-deterministic way, meaning that each execution of the planning libraries could yield different results, as the learning could become stronger as the program became smarter.
This concept was an especially difficult one to grok at the time because a hacker usually has little or no knowledge of the system they infiltrated. As they make each move, the hacker learns about the system piece-by-piece. But each decision runs some level of risk of getting caught.
With an idea pitched and funding accepted, the original MITRE team was headed by Doug Miller, Ron Alford, Andy Applebaum, Henry Foster, Caleb Little and Blake Strom.
Version #2: Becoming usable
In October 2018, MITRE brought me on board to lead the Caldera project. I was just leaving a post at FireEye, where I served as the team leader of the secretive Nucleus project, which was a primary source of Mandiant threat intelligence. Previously, I had spent time between various defense contractors, U.S. government agencies and private security companies doing a combination of offensive security, engineering and big data. So developing the next iteration of Caldera was a perfect fit.
For my first month on the Caldera project, I evaluated the existing Caldera research and prototype system, which was developed for Windows and limited in scope with only 22 tactics, techniques and procedures (TTPs) available to execute. After picking through the Python code and planner math to understand the beauty and complexity of it all, I set to work to operationalize the system.
Between November 2018 and April 2019, I built Caldera version 2 in place, with a small team of engineers including Alex Manners (hey, this guy should sound familiar!), Henry Foster and Caleb Little, on top of the existing Python 3 asyncio code base. Among many design decisions, we chose to keep the HTTP(S) command-and-control architecture, meaning RATs would communicate over this protocol to the C2.
Another big decision was to build the framework as a base system and break extra functionality into a new concept called plugins. To retain backwards compatibility with v1, we bucketed all the v1 code into an “adversary” plugin and allowed the user to leverage that from within the new code base.
Within the plugins, we created a collective base of available functionality:
Stockpile was intended to hold all open-source TTPs, of which we created 80+ to start.
Sandcat was a GoLang agent we wrote which could be compiled to run on Windows, Linux and MacOS system.
Manx was another GoLang agent written as a reverse-shell alternative to Meterpreter (Metasploit).
Mock was a completely simulated environment for practice.
Response was a blue-version of the normally red-focused Caldera to allow autonomous blue agent capabilities, such as detection training.
GameBoard was written as a red-vs-blue chessboard.
And the list goes on. By the end of v2 support, the team had written and supported more than 15 open-source plugins.
I unveiled Caldera v2 as a speaker at the BSides Baltimore security conference in April 2019, and uploaded the code open-source shortly after through the handle privateducky.
Early in my career, I worked for the U.S. military as a cyber operator. The job took me around the world, often dropping me in interesting locations to help a greater cause. When in the states, my team was intentionally stationed (geographically) apart. I worked from the midwest, with other team members dialing in from the northeast, D.C. and the west coast.
This was the one job I had that mirrored a spy movie, to some extent: we rotated phone numbers every 3 months and ran through a number of hacker “handles” as we trashed them after each contract.
Toward the end of my stint, I was getting ready to use a new handle, privateducky. The handle never got used however so I adopted it as a personal handle when I left my job behind. This handle has since followed me through several jobs and non-sensitive assignments. It now serves as my official public handle.
Over the lifetime of active Caldera v2 development, I also wrote the Training plugin, which turned into (probably) the most influential of all the plugins.
I initially wrote Training while on a flight back from a Colorado red-team engagement for the Air Force. The Caldera team was getting so popular at MITRE that I needed a way to determine who had the right skill set to join the team and who wasn't yet ready to join such a fast-paced project. I wrote Training to be a real-time certification program which would teach Caldera to a user and reward them with a certificate code proving their knowledge. This was intended to be a living certification program which would get updated as new functionality was added to Caldera.
Following the release of this prototype certificate program, we started getting flooded with certificate codes from the Caldera user base online. Aside from my own personal use-case of determining who was (and wasn’t) skilled enough to join our internal team, people online were hungry to learn more about computer hacking in a realistic, easy-to-use environment.
A new direction: Prelude Operator
Over the summer of 2020, I co-founded Prelude Research Inc. (with CEO Spencer Thompson) whose reason for existence was educating the population in ways that would bring real social and financial change to the individual level.
The Caldera training plugin had since sparked a significant amount of traffic, as people from all over were seeking Caldera certification as a way to self-conduct continued education. In conjunction with the Prelude team, I determined to double-down on our combined vision and meld the concepts of ongoing training with the unique education vantage point that made Prelude special. With cybersecurity being one of the fastest growing industries, there was a real opportunity to build an all-encompassing tool which packaged the best of security with the new concept of real-time certification for impactful education.
The project, dubbed Operator, was to be different, unhinged in many ways from the prior work I'd done in the space.
In the past, the decisions I made had to fit requirements set by MITRE. Three big requirements I needed to work around were that:
We could not release code using encryption and other "high-end" security measures.
We could not build a solution which included Software As A Service (SaaS).
We could not build a product or support commercial entities or it would violate the FAR.
The latter was the big one, as this restricted my ability to create a multi-user account or have any persistence for a user in cloud databases. Essentially, this meant that Caldera could not be made for production use.
After joining Prelude, head engineer Lewis Zimmerman and myself set out to build a cutting edge C2 desktop application.
This was a ground-up new inspiration. Instead of using Python, we wrote the app in NodeJS on top of the ElectronJS framework. ElectronJS was chosen because, since its inception, many great and scalable desktop applications (such as Slack) have been built on top of it, proving its ability to handle a professional-grade desktop environment.
This new desktop application was to include all the main functionality I designed in Caldera but be easier to install, use and understand. We made special effort to keep the plugin architecture but make it feel more embedded. We also determined that machine-learning (planning) is key to the future of autonomous red teaming systems, so we emphasized its role in Operator.
The most integral part of the desktop application was the training, however. Instead of being a plugin, like we did in Caldera, Operator training was baked into the core of the platform.
What lies ahead
Over my career I’ve noticed that staying educated in cybersecurity is difficult. It's not the learning as much as learning what you need to learn. Because cybersecurity as a topic ranges across so many different areas - from red teaming and blue teaming to malware analysis and threat intelligence - it becomes a daunting task to stay educated.
Operator is designed to bridge the gap. It is a unique tool that is intended to be your daily driver as a security professional but also the place where you train and grow your skill set.
As we move Operator into the future, we're doing so with a singular goal: making red team technology more accessible. And the best way to do that? Create the simplest tool to use, make it free and bundle tons of training resources right into it.