Adversarial A(i)dvantage PT.1
The Need For More Adversarial AI Security Capabilities & Tools
The rise of machine learning and deep learning (ML/DL) capabilities has dramatically increased while the barrier to entry for introducing these capabilities into a software stack has decreased. A small tech firm today can offer capabilities that benefit from ML/DL applications without necessarily needing a data scientist on staff and it's safe to assume they might not have a security engineer on staff either. Much of the ML/DL applications used today were originally designed without considering potential adversarial threats. Even at a large company it shouldn't be assumed that data scientists are fully aware of the potential security risks to ML/DL applications and I believe that for most security professionals there hasn't been a significant issue yet within ML/DL to put these systems fully on their radar. This leaves a huge opportunity for hackers to do a considerable amount of damage before reactionary counter measures are taken within the tech industry. I generally find, especially within the security industry, that it's 95% reactive and 5% proactive with most proactive capabilities existing within the more wealthy businesses.
In the geopolitical space we have two major nation states (the United States and China) that are significantly contributing to the advancement of Artificial Intelligence. The technical philosophy of these two counties is quite a bit different and leaving one of those countries in particular, in a tough situation. China has a significant amount of data that can be harvested, so their focus is more about how to get models to produce the end product. This means China can get to the end result faster and potentially leap frog features initially. The United States - because it has vastly less potential in data gathering - has been forced to spend its research energy on how to extract the greatest amount of training from as little amount of data as possible, in the quickest amount of time. The United States has also been pushed to perform research in things like data privacy and has been going through technical growing pains where learning/inference capabilities have been questioned within certain scenarios (like facial recognition) which China does not have to deal with. This puts the United States in a tough situation because China can spend its energy getting to the end result and use - or even potentially steal - any significant new fundamental capabilities from the United States to make greater advancement.Â
While most small businesses won't be operating on the cutting edge, all of us play on the same battlefield of the internet. The tech and security industry needs to start aggressively developing both proactive and reactive security capabilities for the wave of ML/DL attacks that are likely happening now and will likely increase as more significant advancement occurs and more companies rely on training models.
Thinking as an attacker, the end goal of attacking ML/DL is a lot different than many of the popular attacks today. For ransomware the attacker’s goal is to deny the user of their data and make them pay to get their data back. A clear cut transaction. An attacker may use denial of service to stop for a period of time business operations or try to use a vulnerability within a system to steal sensitive internal or customer data. For ML/DL applications, the end goal is not as cut and dry, nor as noticeable to the target.Â
When thinking about application based Adversarial AI Attacks there are several known categories:Â
Evasion Attacks
The most popular attack scenario, manipulating inputs to a trained model with the goal of producing incorrect output.Â
Data Poisoning Attacks
At the training phase, contaminating the training data in a particular way produces incorrect output at inference time. (This is primarily what the B1-66ER adversary seeks to achieve)
Exploratory Attacks Â
These attacks are a bit less pinpointed. The attacker is not directly trying to contaminate or manipulate the data but more trying to understand the model itself or extract knowledge based on the training. Exploratory attacks not only fall under covert capabilities but also privacy related data extraction like membership inference attacks.
While some of these Adversarial AI application attacks may get noticed by an EDR by way of getting to the intended target machine, once the attacker is on the target machine very little is out there to protect a company from application based Adversarial AI attacks. A company could take measures to air-gap sensitive training environments, but that is not necessarily a catch all solution nor does it solve the underlying issue and some companies may not know how to do this or do it effectively. An issue I have witnessed in the field is that there needs to be an example that produces a real world adversarial attack with repeatability that is available to a mass audience. Adversarial AI attacks are being produced to develop security controls and tools, but much of it is being done in Academia foundational research where prototypes are developed, papers are published, and a small sample set of ideas move on and the majority goes dormant when funding is allocated elsewhere.
The tech industry needs developed capabilities like simulated Adversarial AI attacks on the network and also have various proven developed adversarial countermeasures like adversarial retraining. At Prelude we are developing B1-66ER, an adversarial chain which intends to provide security professionals a way to simulate an end-to-end Adversarial AI attack. In this adversarial chain, you will perform not only the training, but also the data poisoning to contaminate the training model. Once B1-66ER is completed, you will have the ability to test your network against Adversarial Attacks and be able to test developed countermeasures. This will hopefully result in the ability to use developed adversaries like B1-66ER as a test bed to further develop proactive and reactive Adversarial AI capabilities. Â